Assessment method and device for language model, equipment, medium and program product

Abstract

The invention relates to the technical field of computers, and discloses an evaluation method and device for a language model, equipment, a medium and a program product. Processing the test question by using a target language model to obtain a first answer; obtaining assertion description corresponding to the test question, wherein the assertion description comprises a first reply key point and first description information corresponding to the first reply key point; and based on the first answer, the assertion description and an evaluation standard, evaluating the first answer to obtain a first evaluation result of the target language model. According to the method, more refined evaluation analysis can be carried out on the first answer, and the accuracy of the evaluation result is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an evaluation method, apparatus, device, medium, and program product for a language model. Background Art

[0002] Language models, especially those for text-based language models, are generally given model questions and the model gives corresponding responses. Since the accuracy of the model's response directly affects the user's experience of using the product, it is necessary to pay attention to the accuracy of the language model's response. Therefore, it is necessary to propose an evaluation method for language models to evaluate the output of the language model in order to guide the subsequent iterative update of the language model. Summary of the invention

[0003] In view of this, the present disclosure provides a method, apparatus, device, medium and program product for evaluating a language model to solve the problem of evaluating a language model.

[0004] In a first aspect, the present disclosure provides a method for evaluating a language model, the method comprising:

[0005] Get test questions;

[0006] Processing the test question using the target language model to obtain a first answer;

[0007] Obtaining an assertion description corresponding to the test question, the assertion description including a first answer point and first description information corresponding to the first answer point;

[0008] Based on the first answer, the assertion description and the evaluation criteria, the first answer is evaluated to obtain a first evaluation result of the target language model.

[0009] In a second aspect, the present disclosure provides an evaluation device for a language model, the device comprising:

[0010] A question acquisition module is used to obtain test questions;

[0011] A question processing module, used to process the test question using a target language model to obtain a first answer;

[0012] An assertion acquisition module, used to acquire an assertion description corresponding to the test question, wherein the assertion description includes a first answer point and first description information corresponding to the first answer point;

[0013] An answer evaluation module is used to evaluate the first answer based on the first answer, the assertion description and the evaluation criteria to obtain a first evaluation result of the target language model.

[0014] In a third aspect, the present disclosure provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the evaluation method for a language model of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0015] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the evaluation method for a language model according to the first aspect or any corresponding embodiment thereof.

[0016] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, where the computer instructions are used to enable a computer to execute the evaluation method for a language model according to the first aspect or any corresponding embodiment thereof.

[0017] The evaluation method for the language model provided by the embodiment of the present disclosure uses the target language model to process the test question after obtaining the test question, and obtains the first answer given by the target language model to the test question. After obtaining the test question, the assertion description corresponding to the test question is obtained, and the assertion description includes the first answer point and the first description information corresponding to the first answer point. The first answer is then evaluated in combination with the first answer, the assertion description and the evaluation criteria, and then the first evaluation result of the target language model is obtained. In the above processing, since the test question is analyzed, the first description information corresponding to the answer point is obtained. The answer point represents the key points that need to be given when answering the test question, and the first description information represents the description information for confirming the first answer, thereby providing certain guidance for the evaluation of the first answer, which can ensure that a more refined evaluation and analysis can be performed on the first answer, thereby improving the accuracy of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a flowchart of a method for evaluating a language model according to an embodiment of the present disclosure;

[0020] Figure 2 is a flowchart of another evaluation method for a language model according to an embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram of a process for generating a first reply key point according to an embodiment of the present disclosure;

[0022] Figure 4 is a schematic diagram of a generation process of assertion information according to an embodiment of the present disclosure;

[0023] Figure 5 is a flowchart of another evaluation method for a language model according to an embodiment of the present disclosure;

[0024] Figure 6 is a schematic diagram of an evaluation system for a language model according to an embodiment of the present disclosure;

[0025] Figure 7 is a structural block diagram of an evaluation device for a language model according to an embodiment of the present disclosure;

[0026] Figure 8 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0028] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0029] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0030] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] In related technologies, language model evaluation is generally done manually. Specifically, fixed evaluation questions are prepared in advance, and then test points and scoring criteria are manually written for the evaluation questions, and then the responses output by the language model are manually judged and scored.

[0034] However, the evaluation questions and test points in this evaluation method are relatively fixed, and they are updated slowly because they need to be maintained manually. In addition, in the evaluation process, subjective bias is easily added during manual evaluation, and different people have different understandings of the evaluation criteria, which leads to unstable implementation in the actual evaluation process.

[0035] In other related technologies, the evaluation of language models uses automated evaluation, that is, by writing prompt instructions, the evaluation model automatically evaluates the response of the language model. However, the prompt instructions are fixed in the above process, making it difficult to make targeted evaluations for different test questions and difficult to adapt to changing problem scenarios. In addition, the automated evaluation in related technologies is generally used for the evaluation of test questions with reference answers. This evaluation method is difficult to apply to the evaluation of open, subjective or questions without fixed reference answers.

[0036] Based on this, the embodiment of the present disclosure provides an evaluation method for a language model, which analyzes the test question to obtain the corresponding assertion description, and then evaluates the first answer given by the target language model to the test question and the evaluation criteria, thereby obtaining the evaluation result of the target language model. This method can ensure that a more refined evaluation analysis can be performed on the first answer, thereby improving the accuracy of the evaluation result.

[0037] According to an embodiment of the present disclosure, an embodiment of an evaluation method for a language model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] In this embodiment, a method for evaluating a language model is provided, which can be used in a server. Figure 1is a flowchart of a method for evaluating a language model according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0039] Step S101, obtaining test questions.

[0040] The test questions may come from a test data set, which may include multiple test questions. The types of test questions may also be diverse, for example, some are explanation questions, some are query questions, some are prediction questions, etc. Different types of test questions may be in the same test data set, or may be distinguished by using different test data sets, which is not limited here.

[0041] When the target language model needs to be evaluated, test questions can be extracted from the test data set to obtain test questions for input into the target language model.

[0042] Step S102: Process the test question using the target language model to obtain a first answer.

[0043] It should be noted that the target language model used for evaluation can be the language model itself, or an interactive product based on the language model, such as a language assistant product, etc. There is no limitation on the form of the target language model, which can be determined according to actual needs.

[0044] After the test question is obtained in the above step S101, the test question is input into the target language model, and the target language model is used to process the test question to obtain a corresponding first answer, wherein the first answer is the answer given by the target language model to the test question.

[0045] The evaluation of the target language model is the evaluation of the answer given by the target language model. In other words, the accuracy of the first answer indicates the accuracy of the target language model. Therefore, the first answer needs to be evaluated later.

[0046] Step S103, obtaining the assertion description corresponding to the test question.

[0047] The assertion description includes the first reply point and the first description information corresponding to the first reply point.

[0048] For the evaluation of the answer given by the target language model, it is necessary to know what key points of the answer to the test question are needed, that is, what content the answer to the test question needs to include. Furthermore, in addition to what content is included, it is also necessary to verify whether these contents are included during the evaluation process, that is, the corresponding first description information can be obtained based on the first key point of the answer, which is used to provide corresponding prompts for subsequent evaluation.

[0049] Specifically, the assertion description can be obtained by semantically analyzing the test question, searching in the database based on the result of the semantic analysis, and obtaining the corresponding answer points, that is, the first answer points. For different answer points, there is corresponding first description information, and accordingly, the first description information corresponding to the answer point can be obtained.

[0050] The assertion description may also be obtained by relying on the generation of an assertion model. The input of the assertion model includes a test question, and the output includes an assertion description corresponding to the test question. The assertion description includes the first answer key points for the test question and its corresponding first description information.

[0051] Step S104, based on the first answer, the assertion description and the evaluation criteria, the first answer is evaluated to obtain a first evaluation result of the target language model.

[0052] The evaluation criteria is used to give a criterion and evaluate according to the criterion to ensure the consistency of the evaluation criteria. The first answer is the answer given by the target language model to the test question, and the assertion description is the first answer point given by the analysis of the test question and its corresponding first description information.

[0053] The first answer may be evaluated by combining the evaluation model to obtain an evaluation result of the first answer. For example, the first answer, the assertion description, and the evaluation criteria are input into the evaluation model, and the evaluation model outputs an evaluation result (which may be a score) for the first answer. Alternatively, the evaluation model may be trained based on the evaluation criteria so that the evaluation model can learn information about the evaluation criteria. After the training is completed, the input of the evaluation model may include the first answer and the assertion description.

[0054] For the test questions input into the target language model, the corresponding evaluation results are obtained after the test questions are input. Furthermore, the first evaluation result of the target language model can be obtained by combining the evaluation results of all the test questions.

[0055] In some optional implementations, the first description information in the assertion description can be used to perform semantic analysis on the first answer in turn to determine whether the first answer contains the content in the first description information. After obtaining the analysis results corresponding to all the first description information, they are merged to obtain the evaluation result of the first answer. Furthermore, the first evaluation result of the target language model can be obtained by combining the evaluation results of all the test questions.

[0056] The evaluation method for the language model provided in this embodiment analyzes the test question and obtains the first description information corresponding to the answer points. The answer points represent the key points that need to be given when answering the test question, and the first description information represents the description information for confirming the first answer, thereby providing certain guidance for the evaluation of the first answer, ensuring that a more refined evaluation and analysis can be performed on the first answer, and improving the accuracy of the evaluation results. In addition, since the evaluation method does not rely on reference answers, it can be applied to test scenarios without reference answers, subjectivity, and openness, thereby improving the evaluation scope of the language model.

[0057] In this embodiment, a method for evaluating a language model is provided, which can be used in a server. Figure 2 is a flowchart of a method for evaluating a language model according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:

[0058] Step S201, obtain test questions. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0059] Step S202: Use the target language model to process the test question and obtain a first answer. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0060] Step S203: Obtain the assertion description corresponding to the test question.

[0061] The assertion description includes the first reply point and the first description information corresponding to the first reply point.

[0062] Specifically, the above step S203 includes:

[0063] Step S2031, obtaining the first answer key points of the test question.

[0064] When analyzing the test questions, you can first analyze the key points of the answer, and then generate the assertion description based on this. The key point analysis of the answer can be obtained through the key point model of the answer, or through database matching, etc. There is no limitation on the generation method of the first key point of the answer, which can be set according to actual needs.

[0065] In some optional implementations, the above step S2031 includes:

[0066] Step a1, obtaining the first question type of the test question.

[0067] Step a2, based on the first question type and the key point analysis model, obtain the first answer key point.

[0068] The generation of answer points can be based on the question type of the test question. Since different question types correspond to different questioning scenarios, analyzing the answer points in combination with the question type can ensure that the first answer points obtained can better match the questioning scenario.

[0069] The method for obtaining the question type for the test question may be implemented through a classification model. That is, the test question is input into the classification model to obtain the first question type corresponding to the test question. The first question type includes but is not limited to explanation type, evaluation type, comparison type, recommendation type, prediction type, solution type, query type, and problem solving type, etc.

[0070] For the same test question, the corresponding first question type may be one or more, and there is no limit to the number. The first question type of the test question may be distinguished by an identifier, for example, using identifiers 0-7 to represent the above 8 types respectively, and for each test question, the 8 numbers 0-7 are used as identifiers of the first question type corresponding to the test question. Of course, other methods may also be used to represent and distinguish the first question type, and there is no limit to them.

[0071] After obtaining the first question type, a demand analysis is performed in combination with a key point analysis model to obtain the first answer key point. The input of the key point analysis model includes the first question type or the identifier of the first question type, and the output includes the first answer key point.

[0072] The first test question is analyzed for type to obtain the corresponding first question type, thereby ensuring that the requirements of the first test question can be understood and evaluated from the user's perspective to dynamically adapt to complex problem scenarios.

[0073] In some optional implementations, the above step a2 includes:

[0074] Step a21, based on the first question type and the first evaluation indicator, query the first knowledge base to obtain the first analysis rule, the first knowledge base is used to characterize the mapping relationship between the question type, the evaluation indicator and the key analysis rule, and the first analysis rule is used to characterize the prompts for the key points of the first answer.

[0075] Step a22, obtaining first prompt information based on the fusion of the first analysis rule and the first prompt template.

[0076] Step a23, input the first prompt information into the key point analysis model to obtain the first answer key point.

[0077] When analyzing the needs, the first evaluation indicator can be used for processing. The first evaluation indicator is used to characterize the dimensions that the evaluation is concerned about, including but not limited to satisfaction of needs, authenticity and accuracy, novelty and timeliness, rich content, simplicity and refinement, and ease of reading. For each test question, the first evaluation indicator can be fixed or adjusted according to the different adaptability of the test question.

[0078] like Figure 3 As shown, the first answer key point is generated based on the key point analysis model, and the input of the key point analysis model includes the first prompt information. The first prompt information can also be understood as dynamic prompt information, where "dynamic" represents that the prompt information input into the key point analysis model changes with the test question, and is not fixed.

[0079] Specifically, the first prompt information is obtained based on the question type of the test question and the first evaluation indicator. The question type and the first evaluation indicator are used to perform rule query in the first knowledge base to obtain the first analysis rule. For example, the question type and the first evaluation indicator constitute a query keyword, and the query keyword is used to perform semantic matching in the first knowledge base to obtain the first analysis rule representing the matching result.

[0080] The first knowledge base represents the mapping relationship between question types, evaluation indicators and key analysis rules. When matching using keywords in the first knowledge base, the matching result may be multiple analysis rules or one analysis rule, and there is no limit on the number of them.

[0081] For the key point analysis model, it is processed based on the prompt information. And the prompt information is generated based on the first prompt template. The first prompt template includes fixed information input into the key point analysis model, and the change information related to the test question can be represented in the form of variables. After determining the variable value corresponding to the test question, it is merged with the first prompt template to obtain the first prompt information corresponding to the test question. Among them, in this embodiment, the variable value corresponding to the test question can be understood as the first analysis rule queried. In addition, the fusion of the first analysis rule and the first prompt template can be considered as filling the first prompt template with the first analysis rule to obtain the first prompt information.

[0082] The first knowledge base is queried through the first question type and the first evaluation index to obtain the first analysis rule and form the first prompt information. The first prompt information is set to adapt to the uniqueness of each test question, and it can be ensured that the first answer points generated based on the first prompt information can be applied to the evaluation of test questions without fixed reference answers, thereby improving the accuracy of the evaluation results.

[0083] In some optional implementations, the above step a2 further includes: matching the first question type in the first evaluation indicator to obtain a first assessment indicator, and the first answer points also include the first assessment indicator.

[0084] Due to the diversity of test questions, some are simple and some are complex, and not all test questions need to examine all evaluation indicators. Therefore, by dynamically analyzing and determining the first evaluation indicator corresponding to each first test question and including it in the first reply points, the evaluation efficiency can be improved. Specifically, as described above, the first evaluation indicator includes multiple evaluation indicators. After determining the first question type of the test question, the correspondence between the question type and the examination dimension can be used to determine the first examination indicator corresponding to the first question type. Alternatively, it is also possible to combine the examination dimension analysis model, with the input including the first question type and the first evaluation indicator, and the output including the first examination indicator. Furthermore, the obtained first examination indicator is included in the first reply points.

[0085] In some optional embodiments, the first answer points include point fields and descriptions corresponding to the point fields, the type field includes at least one of primary requirement, secondary requirement, necessary points and richness points, and the first question type.

[0086] The first reply key points can be represented by key point fields and their corresponding description forms. For the test questions, the corresponding main requirements, secondary requirements, necessary key points and richness key points, etc. are given. The output of the key point analysis model for the first reply key points can be output based on the reply key point model. The reply key point model includes key point fields and their field placeholders. After obtaining the content of each reply key point through the reply key point model, the content of the reply key point is filled into the placeholder of the corresponding key point field to obtain the first reply key point.

[0087] Furthermore, the first answer key also includes a first question type of test question, which is used for generating subsequent assertions.

[0088] The main points of the first reply are represented by describing the correspondence between fields, which can clearly represent the form of the main points of the first reply.

[0089] Step S2032: Generate an assertion based on the first reply key points and at least one first evaluation indicator to obtain an assertion description.

[0090] The first reply key points are obtained based on the above step S2031, and the at least one first evaluation indicator can be a full set of evaluation indicators, that is, all preset evaluation indicators. The first reply key points and the at least one first evaluation indicator are used as inputs of the assertion generation model, and the assertion information is obtained after being processed by the assertion generation model.

[0091] As described above, the assertion information includes the first description information corresponding to the first reply point. It should be noted that the first description information may not only include the question corresponding to the first reply point, but also include reply sub-points for the corresponding first reply point, and provide the corresponding first description information for the reply sub-points.

[0092] That is, the assertion description not only provides the corresponding first description information for the first reply point, but also provides reply sub-points for each first reply point, and then provides the corresponding first description information for the reply sub-points. Furthermore, the assertion description may also include assertion information other than the first reply point, and may also have corresponding first description information for additional assertion information, and so on.

[0093] In some optional implementations, the above step S2032 includes:

[0094] Step b1, based on the test question, the first answer key points and the first evaluation indicator, query the second knowledge base to obtain the second analysis rule, the second knowledge base is used to characterize the mapping relationship between the test question, the answer key points and the evaluation indicator and the assertion analysis rule, and the second analysis rule is used to characterize the prompts for the assertion description.

[0095] Step b2: Process the second analysis rule using the assertion generation model to obtain assertion information.

[0096] The second knowledge base is similar to the first knowledge base, and both represent mapping relationships, wherein the second knowledge base represents the mapping relationship between test questions, answer points, and evaluation indicators and assertion analysis rules.

[0097] like Figure 4 As shown, the test question, the first answer key points and the first evaluation index are used to query the second knowledge base to obtain the second analysis rule. The second analysis rule includes a prompt for generating assertion information. It should be noted that the second analysis rule matched can be one or more, and the specific number is not limited and can be set according to actual needs.

[0098] The second analysis rule obtained by querying the second knowledge base is input into the assertion generation model to obtain assertion information, which corresponds to the test question.

[0099] By setting up the second knowledge base, matching is performed according to the first test question, the first answer key points and the first evaluation indicator, and assertion information can be dynamically generated to adapt to problem scenarios of different test questions.

[0100] In some optional embodiments, the assertion description includes an assertion field and first description information corresponding to the assertion field, the assertion field includes at least one of a primary requirement, a secondary requirement, a necessary point, and a richness point, and a first question type.

[0101] The representation form of the assertion description is similar to that of the first reply points, and is also represented by fields and their corresponding contents. The assertion fields include at least one of the main requirements, secondary requirements, necessary points, and richness points. The assertion fields can be the same as the key points fields in the first reply points, or more than the key points fields, etc.

[0102] Step S204: Based on the first answer, the assertion description and the evaluation criteria, the first answer is evaluated to obtain a first evaluation result of the target language model. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0103] The evaluation method for the language model provided in this embodiment analyzes the key points of the answer to the first test question, that is, performs a demand analysis on the first test question, so as to adapt to complex problem scenarios. On this basis, assertions are generated, that is, the key points of the answer are further refined and converted into clear instructions and requirements that are more suitable for the language model to understand, so as to facilitate subsequent evaluation. Furthermore, by generating assertion descriptions through the key points of the first answer, it is possible to evaluate test questions that are open, subjective, or have no fixed answers, and it is possible to improve the accuracy of the evaluation, reduce illusions and inconsistent scenes during the evaluation process, and make the evaluation more robust.

[0104] In this embodiment, a method for evaluating a language model is provided, which can be used in a server. Figure 5 is a flowchart of a method for evaluating a language model according to an embodiment of the present disclosure, such as Figure 5 As shown, the process includes the following steps:

[0105] Step S501, obtain test questions. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0106] Step S502: Use the target language model to process the test question and obtain a first answer. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0107] Step S503: Obtain the assertion description corresponding to the test question.

[0108] The assertion description includes the first reply point and the first description information corresponding to the first reply point. Figure 2Step S203 of the illustrated embodiment will not be described in detail here.

[0109] Step S504: Based on the first answer, the assertion description and the evaluation criteria, the first answer is evaluated to obtain a first evaluation result of the target language model.

[0110] Specifically, the above step S504 includes:

[0111] Step S5041, process the evaluation criteria to obtain a scoring standard that can be processed by the evaluation model.

[0112] The evaluation criteria may be evaluation criteria described in natural language, which cannot be directly processed by the evaluation model. The evaluation criteria are processed to convert them into scoring criteria that can be processed by the evaluation model.

[0113] Step S5042: input the assertion description, the first answer, and the scoring criteria into the evaluation model to obtain a first evaluation result of the target language model.

[0114] The input of the evaluation model includes the assertion description, the first answer and the scoring criteria, and the output includes the evaluation result of the first answer. Multiple corresponding evaluation results are obtained through multiple test questions, and the multiple evaluation results are merged to obtain the first evaluation result of the target language model. For example, the evaluation result corresponding to each test question is a scoring value, and the average of the multiple scoring values ​​is calculated, and the obtained average score is used as the first evaluation result of the target language model.

[0115] In some optional implementations, if a reference answer exists for the test question, the reference answer is also input into the evaluation model to obtain an evaluation result. For test questions that have a reference answer, the reference answer is also used as an input to the evaluation model to evaluate the first answer with the reference answer.

[0116] For example, Figure 6 As shown, for the test set Query, the test questions included therein are sequentially input into the language model to output answers, which are used as an input of the evaluation model. In addition, for the test questions, they are analyzed to obtain the first answer points, and the assertion description is obtained based on the first answer points combined with the evaluation indicators. Then, the scoring standards after the conversion of the evaluation criteria are combined and input into the evaluation model to obtain the corresponding evaluation results. Furthermore, if the test questions include reference answers, the reference answers are also input into the evaluation model to obtain the first evaluation result of the language model.

[0117] The evaluation method for the language model provided in this embodiment processes the evaluation criteria into a scoring criteria that can be processed by the evaluation model, and uses the evaluation model to evaluate the first answer, thereby obtaining an accurate evaluation result.

[0118] As a specific application example of the embodiment of the present disclosure, if the test question given is: What are the methods and rules for simplifying Chinese characters? Do you think they are completely reasonable? Please give your reasons.

[0119] Input it into the language model and get the answer given by the language model.

[0120] The key points of the test question are analyzed, and the generated first answer key points can be described in JSON language, that is, can be expressed in the following form:

[0121]

[0122]

[0123] Combine the first answer point and the first evaluation indicator above to generate the assertion description. The assertion description can be described in JSON language, that is, it can be expressed in the following form:

[0124]

[0125]

[0126]

[0127]

[0128] After obtaining the assertion information, the assertion information, the first answer and the scoring criteria are input into the evaluation model to obtain a scoring result for the first answer. After obtaining multiple scoring results for multiple test questions, these scoring results are merged to obtain a first evaluation result of the target language model.

[0129] In this embodiment, an evaluation device for a language model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0130] This embodiment provides an evaluation device for a language model, such as Figure 7 As shown, including:

[0131] The question acquisition module 701 is used to acquire test questions.

[0132] The question processing module 702 is used to process the test question using the target language model to obtain a first answer.

[0133] The assertion acquisition module 703 is used to obtain the assertion description corresponding to the test question, where the assertion description includes the first answer point and the first description information corresponding to the first answer point.

[0134] The answer evaluation module 704 is used to evaluate the first answer based on the first answer, the assertion description and the evaluation criteria to obtain a first evaluation result of the target language model.

[0135] In some optional implementations, the assertion acquisition module 703 includes:

[0136] Key point analysis unit, used to obtain the key points of the first answer to the test questions.

[0137] The assertion generation unit is used to generate an assertion based on the first reply key point and at least one first evaluation indicator to obtain an assertion description.

[0138] In some optional embodiments, the key point analysis unit includes:

[0139] The type analysis subunit is used to obtain a first question type of the test question.

[0140] The answer key sub-unit is used to obtain the first answer key based on the first question type and the key point analysis model.

[0141] In some optional implementations, the reply key points subunit includes:

[0142] The first query sub-unit is used to query the first knowledge base based on the first question type and the first evaluation indicator to obtain the first analysis rule. The first knowledge base is used to characterize the mapping relationship between the question type, the evaluation indicator and the key analysis rule. The first analysis rule is used to characterize the prompt for the key point of the first answer.

[0143] The fusion subunit is used to obtain first prompt information based on the fusion of the first analysis rule and the first prompt template.

[0144] The analysis model processing subunit is used to input the first prompt information into the key point analysis model to obtain the first reply key point.

[0145] In some optional implementations, the reply key points subunit further includes:

[0146] The first matching subunit is used to match the first evaluation indicator based on the first question type to obtain the first assessment indicator. The first answer points also include the first assessment indicator.

[0147] In some optional embodiments, the first answer points include point fields and descriptions corresponding to the point fields, the type field includes at least one of primary requirements, secondary requirements, necessary points and richness points, and the first question type.

[0148] In some optional implementations, the assertion generation unit includes:

[0149] The second query sub-unit is used to query the second knowledge base based on the test question, the first answer points and at least one first evaluation indicator to obtain a second analysis rule, the second knowledge base is used to characterize the mapping relationship between the test question, the answer points and the evaluation indicators and the assertion analysis rules, and the second analysis rules are used to characterize the prompts for the assertion description.

[0150] The assertion model processing subunit is used to process the second analysis rule using the assertion generation model to obtain assertion information.

[0151] In some optional embodiments, the assertion description includes an assertion field and first description information corresponding to the assertion field, the assertion field includes at least one of a primary requirement, a secondary requirement, a necessary point, and a richness point, and a first question type.

[0152] In some optional implementations, the answer evaluation module 704 includes:

[0153] The standard processing unit is used to process the evaluation standard to obtain a scoring standard that can be processed by the evaluation model.

[0154] The evaluation unit is used to input the assertion description, the first answer and the scoring standard into the evaluation model to obtain the first evaluation result of the model.

[0155] In some optional implementations, if a reference answer exists for the test question, the reference answer is also input into the evaluation model to obtain a first evaluation result.

[0156] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0157] The evaluation device for the language model in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0158] The present disclosure also provides an electronic device having the above Figure 7 An evaluation apparatus for a language model is shown.

[0159] See also Figure 8 , Figure 8 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present disclosure, such as Figure 8 As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0160] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0161] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0162] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0163] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0164] The electronic device further comprises a communication interface 30 for the electronic device to communicate with other devices or a communication network.

[0165] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0166] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0167] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for evaluating a language model, characterized in that: The method comprises: Get test questions; Processing the test question using the target language model to obtain a first answer; Obtaining an assertion description corresponding to the test question, the assertion description including a first answer point and first description information corresponding to the first answer point; Based on the first answer, the assertion description and the evaluation criteria, the first answer is evaluated to obtain a first evaluation result of the target language model.

2. The method according to claim 1, characterized in that: The obtaining of the assertion description corresponding to the test problem includes: Obtaining the first answer key point of the test question; An assertion is generated based on the first reply key points and at least one first evaluation indicator to obtain the assertion description.

3. The method according to claim 2, characterized in that The obtaining of the first answer key points of the test question includes: Obtaining a first question type of the test question; Based on the first question type and the key point analysis model, the first answer key points are obtained.

4. The method according to claim 3, characterized in that The obtaining of the first answer key points based on the first question type and the key point analysis model includes: Based on the first question type and the first evaluation indicator, query a first knowledge base to obtain a first analysis rule, wherein the first knowledge base is used to characterize a mapping relationship between question types, evaluation indicators and key analysis rules, and the first analysis rule is used to characterize a prompt for the key points of the first answer; Based on the fusion of the first analysis rule and the first prompt template, obtaining first prompt information; The first prompt information is input into the key point analysis model to obtain the first reply key point.

5. The method according to claim 3, characterized in that: The obtaining the first answer key points based on the first question type and the key point analysis model further includes: Based on the matching of the first question type in the first evaluation indicator, a first assessment indicator is obtained, and the first answer points also include the first assessment indicator.

6. The method according to claim 2, characterized in that The first answer key includes a key field and a description corresponding to the key field, and the type field includes at least one of a primary requirement, a secondary requirement, a necessary key point, and a richness key point, as well as the first question type.

7. The method according to claim 2, characterized in that: The generating of the assertion based on the first reply key point and at least one first evaluation indicator to obtain the assertion description includes: Based on the test question, the first answer key points and at least one of the first evaluation indicators, query a second knowledge base to obtain a second analysis rule, wherein the second knowledge base is used to characterize the mapping relationship between the test question, the answer key points and the evaluation indicators and the assertion analysis rule, and the second analysis rule is used to characterize the prompt for the assertion description; The second analysis rule is processed using the assertion generation model to obtain the assertion information.

8. The method according to claim 7, characterized in that The assertion description includes an assertion field and first description information corresponding to the assertion field, wherein the assertion field includes at least one of a primary requirement, a secondary requirement, a necessary point, and a richness point, and the first question type.

9. The method according to any one of claims 1 to 8, characterized in that The step of evaluating the first answer based on the first answer, the assertion description, and the evaluation criteria to obtain a first evaluation result of the target language model includes: Processing the evaluation criteria to obtain scoring criteria that can be processed by the evaluation model; The assertion description, the first answer, and the scoring standard are input into an evaluation model to obtain a first evaluation result of the target language model.

10. The method according to claim 9, characterized in that If a reference answer exists for the test question, the reference answer is also input into the evaluation model to obtain the first evaluation result.

11. An evaluation device for a language model, characterized in that: The device comprises: A question acquisition module is used to obtain test questions; A question processing module, used to process the test question using a target language model to obtain a first answer; An assertion acquisition module, used to acquire an assertion description corresponding to the test question, wherein the assertion description includes a first answer point and first description information corresponding to the first answer point; An answer evaluation module is used to evaluate the first answer based on the first answer, the assertion description and the evaluation criteria to obtain a first evaluation result of the target language model.

12. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the evaluation method for the language model according to any one of claims 1 to 10 by executing the computer instructions.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the evaluation method for a language model according to any one of claims 1 to 10.

14. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the evaluation method for a language model according to any one of claims 1 to 10.

Images