Evaluation method, device, equipment, medium and program product for language model

Abstract

The present disclosure relates to the technical field of computers, and discloses an evaluation method and device for a language model, equipment, a medium and a program product. The method comprises: obtaining a test question; processing the test question by using a target language model to obtain a first answer; obtaining an assertion description corresponding to the test question, the assertion description comprising a first reply gist and first description information corresponding to the first reply gist; and evaluating the first answer based on the first answer, the assertion description and an evaluation standard to obtain a first evaluation result of the target language model. The method can ensure that the first answer can be evaluated and analyzed more finely, and the accuracy of the evaluation result is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically to methods, apparatus, devices, media, and program products for evaluating language models. Background Technology

[0002] Language models, especially text-to-text language models, typically provide a response to a given question. Since the accuracy of the model's response directly impacts the user experience, it's crucial to focus on the accuracy of language model responses. Therefore, an evaluation method for language models is needed to assess their output, guiding subsequent iterative updates and other improvements. Summary of the Invention

[0003] In view of this, the present disclosure provides a method, apparatus, device, medium and program product for evaluating language models to solve the problem of evaluating language models.

[0004] In a first aspect, this disclosure provides an evaluation method for language models, the method comprising:

[0005] Get test issues;

[0006] The test question is processed using the target language model to obtain the first answer;

[0007] Obtain the assertion description corresponding to the test question, wherein the assertion description includes a first response point and a first description information corresponding to the first response point;

[0008] Based on the first answer, the assertion description, and the evaluation criteria, the first answer is evaluated to obtain the first evaluation result of the target language model.

[0009] Secondly, this disclosure provides an evaluation apparatus for a language model, the apparatus comprising:

[0010] The issue retrieval module is used to retrieve test issues;

[0011] The problem processing module is used to process the test problem using the target language model to obtain a first answer;

[0012] The assertion acquisition module is used to acquire the assertion description corresponding to the test question, wherein the assertion description includes a first response point and a first description information corresponding to the first response point;

[0013] The answer evaluation module is used to evaluate the first answer based on the first answer, the assertion description, and the evaluation criteria to obtain the first evaluation result of the target language model.

[0014] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the language model evaluation method of the first aspect or any corresponding embodiment described above.

[0015] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the language model evaluation method of the first aspect or any corresponding embodiment described above.

[0016] Fifthly, this disclosure provides a computer program product, including computer instructions for causing a computer to execute the language model evaluation method of the first aspect or any corresponding embodiment described above.

[0017] The language model evaluation method provided in this disclosure, after obtaining a test question, processes it using a target language model to obtain a first answer from the target language model for the test question. After obtaining the test question, an assertion description corresponding to the test question is obtained, which includes first response points and corresponding first descriptive information. The first answer, assertion description, and evaluation criteria are then combined to evaluate the first answer, thereby obtaining a first evaluation result for the target language model. In the above processing, because the test question is analyzed, and the first descriptive information corresponding to the response points is obtained, the response points represent the key points that need to be given when answering the test question, and the first descriptive information represents the descriptive information confirming the first answer. This provides guidance for the evaluation of the first answer, ensuring a more refined evaluation analysis and improving the accuracy of the evaluation results. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of an evaluation method for a language model according to an embodiment of the present disclosure;

[0020] Figure 2 This is a flowchart illustrating another method for evaluating a language model according to an embodiment of this disclosure;

[0021] Figure 3 This is a schematic diagram of the generation process of the first response points according to an embodiment of the present disclosure;

[0022] Figure 4 This is a schematic diagram of the process for generating assertion information according to an embodiment of the present disclosure;

[0023] Figure 5 This is a flowchart illustrating another method for evaluating a language model according to an embodiment of the present disclosure;

[0024] Figure 6 This is a schematic diagram of an evaluation system for a language model according to an embodiment of the present disclosure;

[0025] Figure 7 This is a structural block diagram of an evaluation apparatus for a language model according to an embodiment of the present disclosure;

[0026] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0029] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0030] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0032] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0033] In related technologies, the evaluation of language models generally adopts a manual evaluation method. Specifically, fixed evaluation questions are prepared in advance, and then test points and scoring criteria are manually written for the evaluation questions. Finally, the language model's output response is judged and scored by humans.

[0034] However, this assessment method uses relatively fixed evaluation questions and test points, and updates are slow due to the need for manual maintenance. Furthermore, the assessment process is prone to subjective bias due to human intervention, and different individuals may have varying understandings of the assessment criteria, leading to instability in its actual implementation.

[0035] In other related technologies, language model evaluation employs automated evaluation, where prompt instructions are written to automate the evaluation of the language model's responses. However, the fixed prompt instructions in this process make it difficult to provide targeted evaluations for different test questions and adapt to varying question scenarios. Furthermore, automated evaluation in these technologies is generally used for test questions with reference answers, making it difficult to apply to open-ended, subjective, or questions without fixed reference answers.

[0036] Based on this, this disclosure provides an evaluation method for a language model. The method involves analyzing a test question to obtain corresponding assertion descriptions, then combining these with the first answer provided by the target language model for the test question and the evaluation criteria to evaluate the first answer, thereby obtaining the evaluation result of the target language model. This method ensures a more refined evaluation analysis of the first answer, improving the accuracy of the evaluation results.

[0037] According to embodiments of this disclosure, an embodiment of an evaluation method for a language model is provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] This embodiment provides an evaluation method for language models, which can be used in servers. Figure 1This is a flowchart of a language model evaluation method according to an embodiment of the present disclosure, such as... Figure 1 As shown, the process includes the following steps:

[0039] Step S101: Obtain the test problem.

[0040] Test questions can come from a test dataset, which may contain multiple test questions of various types, such as explanation, query, and prediction. Different types of test questions can be from the same test dataset or distinguished using different test datasets; no restrictions are placed on this.

[0041] When it is necessary to evaluate the target language model, test questions can be extracted from the test dataset to obtain test questions for input into the target language model.

[0042] Step S102: Process the test question using the target language model to obtain the first answer.

[0043] It should be noted that the target language model used for evaluation can be the language model itself, or an interactive product based on the language model, such as a language assistant product. There are no restrictions on the form of the target language model; it can be determined based on actual needs.

[0044] After obtaining the test question in step S101 above, the test question is input into the target language model, which processes it to obtain the corresponding first answer. The first answer is the response given by the target language model to the test question.

[0045] The evaluation of the target language model is essentially an evaluation of the answer provided by the target language model. In other words, the accuracy of the first answer represents the accuracy of the target language model. Therefore, subsequent evaluations of the first answer are necessary.

[0046] Step S103: Obtain the assertion description corresponding to the test question.

[0047] The assertion description includes the first response key points and the first description information corresponding to the first response key points.

[0048] Evaluating the answers provided by the target language model requires understanding the key points of the response to the test question; that is, what content the answer to the test question should include. Furthermore, in addition to defining what content to include, the evaluation process also needs to verify whether this content is included. In other words, the initial descriptive information can be obtained based on the first key points of the response to provide appropriate prompts for subsequent evaluations.

[0049] Specifically, the assertion description can be obtained by performing semantic analysis on the test question, and then retrieving the results from the database to obtain the corresponding response points, i.e., the first response points. For different response points, there are corresponding first descriptive information, and thus, the first descriptive information corresponding to the response points can be obtained.

[0050] Assertion descriptions can also be obtained by relying on assertion models. The input to this assertion model includes the test question, and the output includes the assertion description corresponding to the test question. The assertion description includes the first key points of the response to the test question and its corresponding first descriptive information.

[0051] Step S104: Based on the first answer, assertion description, and evaluation criteria, evaluate the first answer to obtain the first evaluation result of the target language model.

[0052] Evaluation criteria are used to evaluate based on a given standard, ensuring the consistency of the evaluation criteria. The first answer is the response given by the target language model to the test question; the assertion description is the first response points and their corresponding first descriptive information given by the analysis of the test question.

[0053] Evaluating the first answer can be achieved by combining an evaluation model with the first answer to obtain an evaluation result. For example, the first answer, assertion description, and evaluation criteria can be input into the evaluation model, and the model can output an evaluation result (which could be a score) for the first answer. Alternatively, the evaluation model can be trained based on the evaluation criteria, enabling it to learn the information of the evaluation criteria. After training, the input to the evaluation model can include only the first answer and the assertion description.

[0054] For the test questions input into the target language model, the corresponding evaluation results are obtained after inputting the test questions. Furthermore, by combining the evaluation results of all test questions, the first evaluation result of the target language model can be obtained.

[0055] In some optional implementations, the first descriptive information in the assertion description can be used to sequentially perform semantic analysis on the first answer to determine whether the first answer contains the content in the first descriptive information. After obtaining the analysis results corresponding to all the first descriptive information, they are fused to obtain the evaluation result of the first answer. Furthermore, combining the evaluation results of all test questions yields the first evaluation result of the target language model.

[0056] The language model evaluation method provided in this embodiment analyzes the test question to obtain first descriptive information corresponding to the key points of the response. These key points characterize the essential information needed to answer the test question, while the first descriptive information confirms the first answer. This provides guidance for evaluating the first answer, ensuring a more refined evaluation and improving the accuracy of the evaluation results. Furthermore, since this evaluation method does not rely on a reference answer, it is applicable to test scenarios with no reference answer, subjectivity, and open-ended questions, thus broadening the scope of language model evaluation.

[0057] This embodiment provides an evaluation method for language models, which can be used in servers. Figure 2 This is a flowchart of a language model evaluation method according to an embodiment of the present disclosure, such as... Figure 2 As shown, the process includes the following steps:

[0058] Step S201: Obtain the test issue. See details below. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0059] Step S202: Process the test question using the target language model to obtain the first answer. See details... Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0060] Step S203: Obtain the assertion description corresponding to the test question.

[0061] The assertion description includes the first response key points and the first description information corresponding to the first response key points.

[0062] Specifically, step S203 includes:

[0063] Step S2031: Obtain the first key points of the test question.

[0064] When analyzing test questions, you can first analyze the key points of the responses, and then generate assertion descriptions based on these points. The key points of the responses can be obtained through a response point model, database matching, etc. There are no restrictions on how the first set of response points is generated; the specific method can be set according to actual needs.

[0065] In some optional implementations, step S2031 above includes:

[0066] Step a1: Obtain the first question type of the test question.

[0067] Step a2: Based on the first question type and the key point analysis model, the first answer key points are obtained.

[0068] The generation of response points can be based on the question type of the test question. Since different question types correspond to different question scenarios, analyzing the response points in conjunction with the question type can ensure that the resulting initial response points better match the question scenario.

[0069] The method for obtaining the question type of a test question can be through a classification model. That is, the test question is input into the classification model to obtain the first question type corresponding to the test question. This first question type includes, but is not limited to, explanation, evaluation, comparison, recommendation, prediction, solution, query, and problem-solving, etc.

[0070] For the same test question, there can be one or more corresponding first question types, and there is no limit to the number. The first question type of a test question can be distinguished by an identifier. For example, the identifiers 0-7 can be used to represent the eight types mentioned above, and for each test question, these eight numbers can be used as the identifier of the first question type. Of course, other methods can also be used to distinguish the first question type, and there is no limitation on them.

[0071] After identifying the first question type, a requirements analysis is performed using the key point analysis model to obtain the first response key points. The input to the key point analysis model includes the first question type or its identifier, and the output includes the first response key points.

[0072] Type analysis is performed on the first test question to obtain the corresponding first question type, thereby ensuring that the requirements of the first test question can be understood and evaluated from the user's perspective, so as to dynamically adapt to complex problem scenarios.

[0073] In some alternative implementations, step a2 above includes:

[0074] Step a21: Based on the first question type and the first evaluation index, query the first knowledge base to obtain the first analysis rule. The first knowledge base is used to represent the mapping relationship between the question type, the evaluation index and the key point analysis rule. The first analysis rule is used to represent the prompts for the first answer key points.

[0075] Step a22: Based on the fusion of the first analysis rule and the first prompt template, the first prompt information is obtained.

[0076] Step a23: Input the first prompt information into the key point analysis model to obtain the first response key point.

[0077] During requirements analysis, a primary evaluation metric can be incorporated. This metric characterizes the dimensions of interest in the evaluation, including but not limited to requirement fulfillment, accuracy, novelty, timeliness, richness of content, conciseness, and ease of reading. For each test question, the primary evaluation metric can be fixed or adjusted based on the specific needs of the test question.

[0078] like Figure 3 As shown, the first response point is generated based on the key point analysis model, and the input of the key point analysis model includes the first prompt information. This first prompt information can also be understood as dynamic prompt information; "dynamic" means that the prompt information input into the key point analysis model changes with the test question, and is not fixed.

[0079] Specifically, the first prompt information is derived based on the question type and the first evaluation metric of the test question. A rule query is then performed in the first knowledge base using the question type and the first evaluation metric to obtain the first analysis rule. For example, the question type and the first evaluation metric form the query keywords, which are then used to perform semantic matching in the first knowledge base to obtain the first analysis rule representing the matching result.

[0080] The first knowledge base represents the mapping relationship between question types, evaluation indicators, and key analysis rules. When matching using keywords in the first knowledge base, the matching result may be multiple analysis rules or a single analysis rule; there is no limit to the number.

[0081] The key point analysis model processes information based on prompts, which are generated from a first prompt template. This first prompt template includes fixed information input into the key point analysis model, while variable information related to the test question can be represented by variables. After determining the variable values ​​corresponding to the test question, these values ​​are fused with the first prompt template to obtain the first prompt information for the test question. In this embodiment, the variable values ​​corresponding to the test question can be understood as the first analysis rule retrieved. Furthermore, the fusion of the first analysis rule and the first prompt template can be considered as using the first analysis rule to populate the first prompt template, thereby obtaining the first prompt information.

[0082] By querying the first knowledge base using the first question type and the first evaluation metric, the first analysis rule is obtained, generating the first prompt information. The setting of this first prompt information is designed to adapt to the uniqueness of each test question, ensuring that the first response points generated based on the first prompt information can be applied to the evaluation of test questions without fixed reference answers, thus improving the accuracy of the evaluation results.

[0083] In some optional implementations, step a2 above further includes: matching the first question type with the first evaluation index to obtain the first assessment index, and the first response points also include the first assessment index.

[0084] Due to the diversity of test questions, ranging from simple to complex, not all test questions require consideration of all evaluation metrics. Therefore, dynamically analyzing and determining the primary evaluation metric for each primary test question and including it in the primary response points can improve evaluation efficiency. Specifically, as described above, the primary evaluation metric includes multiple evaluation metrics. After determining the primary question type of the test questions, the correspondence between question types and evaluation dimensions can be used to determine the primary evaluation metric corresponding to the primary question type. Alternatively, an evaluation dimension analysis model can be used, with input including the primary question type and the primary evaluation metric, and output including the primary evaluation metric. Furthermore, the obtained primary evaluation metric is included in the primary response points.

[0085] In some alternative implementations, the first response points include a point field and a description corresponding to the point field, the type field includes at least one of primary requirement, secondary requirement, necessary point and richness point, and a first question type.

[0086] The first response key points can be represented using key point fields and their corresponding descriptive forms. For the test question, this includes the corresponding primary requirements, secondary requirements, necessary key points, and richness key points, etc. The output of the key point analysis model for the first response key points can be based on the response key point model. The response key point model includes key point fields and their placeholders. After obtaining the content of each response key point through the response key point model, the content of the response key points is filled into the placeholders of the corresponding key point fields, thus obtaining the first response key points.

[0087] Furthermore, the first response points also include a first question type with test questions, used for the generation of subsequent assertions.

[0088] By characterizing the key points of the first response through the description of the fields and their corresponding forms, the form of the key points of the first response can be clearly represented.

[0089] Step S2032: Based on the key points of the first response and at least one first evaluation index, an assertion is generated to obtain an assertion description.

[0090] The first response key point is obtained based on step S2031 above. At least one first evaluation indicator can be all evaluation indicators, that is, all pre-set evaluation indicators. The first response key point and at least one first evaluation indicator are used as input to the assertion generation model. After processing by the assertion generation model, assertion information is obtained.

[0091] As described above, the assertion information includes the first descriptive information corresponding to the first response point. It should be noted that the first descriptive information may not only include the question corresponding to the first response point, but may also include response sub-points for the corresponding first response point, and provide corresponding first descriptive information for the response sub-points.

[0092] That is, the assertion description not only provides corresponding first descriptive information for the first response point, but also provides response sub-points for each first response point, and then provides corresponding first descriptive information for each response sub-point. Furthermore, the assertion description can also include assertion information other than the first response point, and can also have corresponding first descriptive information for additional assertion information, and so on.

[0093] In some optional implementations, step S2032 above includes:

[0094] Step b1: Based on the test question, the first response points, and the first evaluation metric, query the second knowledge base to obtain the second analysis rule. The second knowledge base is used to represent the mapping relationship between the test question, the response points, and the evaluation metric and the assertion analysis rule. The second analysis rule is used to represent the hints for the assertion description.

[0095] Step b2: The second analysis rule is processed using the assertion generation model to obtain assertion information.

[0096] Similar to the first knowledge base, the second knowledge base represents mapping relationships. Specifically, the second database represents the mapping relationship between test questions, response points, evaluation metrics, and assertion analysis rules.

[0097] like Figure 4 As shown, the second analysis rule is obtained by querying the second knowledge base using the test question, the key points of the first response, and the first evaluation metric. The second analysis rule includes prompts for generating assertion information. It should be noted that the number of matched second analysis rules can be one or more; there is no limit to the specific number, and it can be set according to actual needs.

[0098] The second analysis rule obtained from the second knowledge base is input into the assertion generation model to obtain assertion information, which corresponds to the test question.

[0099] By setting up a second knowledge base, and matching it with the first test question, the first answer key points, and the first evaluation index, assertion information can be dynamically generated to adapt to different test question scenarios.

[0100] In some optional implementations, the assertion description includes an assertion field and first descriptive information corresponding to the assertion field. The assertion field includes at least one of primary requirements, secondary requirements, essential points, and richness points, as well as a first problem type.

[0101] The representation of assertions is similar to that of the first response points, also represented by fields and their corresponding content. Assertion fields include at least one of the following: primary requirements, secondary requirements, essential points, and richness points. Assertion fields can be the same as the key points in the first response points, or they can be more than the key points, etc.

[0102] Step S204: Based on the first answer, assertion description, and evaluation criteria, evaluate the first answer to obtain the first evaluation result of the target language model. See details... Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0103] The language model evaluation method provided in this embodiment analyzes the key points of the response to the first test question, that is, it performs a requirements analysis for the first test question, thereby adapting to complex problem scenarios. Based on this, assertions are generated, that is, the key points of the response are further refined and transformed into clear instructions and requirements more suitable for the language model's understanding, facilitating subsequent evaluation. Furthermore, by generating assertion descriptions from the first response points, it is possible to evaluate open-ended, subjective, or non-fixed-answer test questions, improving evaluation accuracy, reducing illusions and inconsistencies in the evaluation process, and making the evaluation more robust.

[0104] This embodiment provides an evaluation method for language models, which can be used in servers. Figure 5 This is a flowchart of a language model evaluation method according to an embodiment of the present disclosure, such as... Figure 5 As shown, the process includes the following steps:

[0105] Step S501: Obtain the test issue. See details below. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0106] Step S502: Process the test question using the target language model to obtain the first answer. See details... Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0107] Step S503: Obtain the assertion description corresponding to the test question.

[0108] The assertion description includes the key points of the first response and the corresponding descriptive information. See details... Figure 2Step S203 of the illustrated embodiment will not be described again here.

[0109] Step S504: Based on the first answer, assertion description, and evaluation criteria, evaluate the first answer to obtain the first evaluation result of the target language model.

[0110] Specifically, step S504 includes:

[0111] Step S5041: Process the evaluation criteria to obtain the scoring criteria that the evaluation model can process.

[0112] The evaluation criteria can be described using natural language, which the evaluation model cannot process directly. By processing the evaluation criteria, they can be converted into scoring criteria that the evaluation model can handle.

[0113] Step S5042: Input the assertion description, first answer, and scoring criteria into the evaluation model to obtain the first evaluation result of the target language model.

[0114] The evaluation model's input includes assertion descriptions, a first answer, and scoring criteria. The output includes the evaluation result of the first answer. Multiple evaluation results are obtained through multiple test questions, and these results are then fused to obtain the first evaluation result of the target language model. For example, the evaluation result for each test question is a score; the average of these scores is calculated, and the resulting average score is used as the first evaluation result of the target language model.

[0115] In some optional implementations, if a test question has a reference answer, that reference answer is also input into the evaluation model to obtain the evaluation result. For test questions with reference answers, these reference answers are also used as input to the evaluation model to evaluate the first answer.

[0116] For example, such as Figure 6 As shown, for the test set Query, the test questions included are sequentially input into the language model to output answers, which are then used as input to the evaluation model. Furthermore, for each test question, analysis yields the first response key points. Based on these first response key points and evaluation metrics, an assertion description is obtained. This description, combined with the scoring criteria transformed from the evaluation standards, is then input into the evaluation model to obtain the corresponding evaluation result. Further, if the test question includes a reference answer, this reference answer is also input into the evaluation model to obtain the first evaluation result of the language model.

[0117] The language model evaluation method provided in this embodiment processes the evaluation criteria into scoring criteria that the evaluation model can handle, and uses the evaluation model to evaluate the first answer, thereby obtaining accurate evaluation results.

[0118] As a specific application embodiment of this disclosure, if the test question is: What are the methods and rules for simplifying Chinese characters? Do you think they are completely reasonable? Please state your reasons.

[0119] Input it into the language model to get the answer given by the language model.

[0120] The key points of the test question are analyzed, and the generated first response can be described using JSON language, that is, it can be represented in the following form:

[0121]

[0122]

[0123] Based on the key points of the first response and the first evaluation metric above, the assertion description is generated. The assertion description can be written in JSON language, that is, it can be represented in the following form:

[0124]

[0125]

[0126]

[0127]

[0128] After obtaining the assertion information, the assertion information, the first answer, and the scoring criteria are input into the evaluation model to obtain a score for the first answer. After obtaining multiple scores for multiple test questions, these scores are fused to obtain the first evaluation result of the target language model.

[0129] This embodiment also provides an evaluation device for a language model, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0130] This embodiment provides an evaluation device for language models, such as... Figure 7 As shown, it includes:

[0131] The issue acquisition module 701 is used to acquire test issues.

[0132] Problem processing module 702 is used to process the test problem using the target language model to obtain the first answer.

[0133] The assertion acquisition module 703 is used to acquire the assertion description corresponding to the test question. The assertion description includes the first response point and the first description information corresponding to the first response point.

[0134] The answer evaluation module 704 is used to evaluate the first answer based on the first answer, the assertion description, and the evaluation criteria to obtain the first evaluation result of the target language model.

[0135] In some alternative implementations, the assertion acquisition module 703 includes:

[0136] The key point analysis unit is used to obtain the first key points of the answer to the test question.

[0137] An assertion generation unit is used to generate assertions based on the first response key points and at least one first evaluation index to obtain an assertion description.

[0138] In some optional implementations, the key point analysis unit includes:

[0139] The type analysis subunit is used to obtain the first question type of the test question.

[0140] The response key points sub-unit is used to obtain the first response key points based on the first question type and the key point analysis model.

[0141] In some alternative implementations, the response point subunit includes:

[0142] The first query subunit is used to query the first knowledge base based on the first question type and the first evaluation index to obtain the first analysis rule. The first knowledge base is used to represent the mapping relationship between the question type, the evaluation index and the key point analysis rule. The first analysis rule is used to represent the prompts for the first answer key points.

[0143] The fusion subunit is used to obtain the first prompt information by fusing the first analysis rule and the first prompt template.

[0144] The analysis model processing subunit is used to input the first prompt information into the key point analysis model to obtain the first response key point.

[0145] In some optional implementations, the response point subunit further includes:

[0146] The first matching subunit is used to match the first question type with the first evaluation index to obtain the first examination index. The first answer points also include the first examination index.

[0147] In some alternative implementations, the first response points include a point field and a description corresponding to the point field, the type field includes at least one of primary requirement, secondary requirement, necessary point and richness point, and a first question type.

[0148] In some optional implementations, the assertion generation unit includes:

[0149] The second query subunit is used to query the second knowledge base based on the test question, the first response points, and at least one first evaluation metric to obtain the second analysis rule. The second knowledge base is used to represent the mapping relationship between the test question, the response points, the evaluation metric, and the assertion analysis rule. The second analysis rule is used to represent the hints for the assertion description.

[0150] The assertion model processing subunit is used to process the second analysis rule using the assertion generation model to obtain assertion information.

[0151] In some optional implementations, the assertion description includes an assertion field and first descriptive information corresponding to the assertion field. The assertion field includes at least one of primary requirements, secondary requirements, essential points, and richness points, as well as a first problem type.

[0152] In some alternative implementations, the answer evaluation module 704 includes:

[0153] The standard processing unit is used to process the evaluation criteria to obtain the scoring criteria that the evaluation model can process.

[0154] The evaluation unit is used to input the assertion description, the first answer, and the scoring criteria into the evaluation model to obtain the first evaluation result of the model.

[0155] In some optional implementations, if a reference answer exists for the test question, the reference answer is also input into the evaluation model to obtain the first evaluation result.

[0156] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0157] In this embodiment, the language model evaluation device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0158] This disclosure also provides an electronic device having the above-described features. Figure 7 The device shown is an evaluation apparatus for a language model.

[0159] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of this disclosure, such as... Figure 8 As shown, the electronic device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0160] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0161] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0162] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0163] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0164] The electronic device also includes a communication interface 30 for communicating with other devices or communication networks.

[0165] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded over a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0166] A portion of this disclosure can be applied to computer program products, such as computer program instructions, which, when executed by a computer, can invoke or provide methods and / or technical solutions according to this disclosure through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0167] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for evaluating a language model, the method comprising: The method comprises: acquiring a test question; processing the test question by using a target language model to obtain a first answer; acquiring an assertion description corresponding to the test question, the assertion description comprising a first reply gist and first description information corresponding to the first reply gist; based on the first answer, the assertion description and an evaluation standard, evaluating the first answer to obtain a first evaluation result of the target language model; wherein the acquiring the assertion description corresponding to the test question comprises: acquiring the first reply gist of the test question; generating an assertion based on the first reply gist and at least one first evaluation index to obtain the assertion description; the generating an assertion based on the first reply gist and at least one first evaluation index to obtain the assertion description comprises: based on the test question, the first reply gist and at least one first evaluation index, querying a second knowledge base to obtain a second analysis rule, the second knowledge base being used to represent a mapping relationship between a test question, a reply gist and an evaluation index and an assertion analysis rule, the second analysis rule being used to represent a prompt for the assertion description; processing the second analysis rule by using an assertion generation model to obtain the assertion description.

2. The method of claim 1, wherein, the acquiring the first reply gist of the test question comprises: acquiring a first question type of the test question; based on the first question type and a gist analysis model, obtaining the first reply gist.

3. The method of claim 2, wherein, the obtaining the first reply gist based on the first question type and the gist analysis model comprises: based on the first question type and a first evaluation index, querying a first knowledge base to obtain a first analysis rule, the first knowledge base being used to represent a mapping relationship between a question type, an evaluation index and a gist analysis rule, the first analysis rule being used to represent a prompt for the first reply gist; based on fusion of the first analysis rule and a first prompt template, obtaining first prompt information; inputting the first prompt information into the gist analysis model to obtain the first reply gist.

4. The method of claim 2, wherein, the obtaining the first reply gist based on the first question type and the gist analysis model further comprises: based on the first question type, matching in the first evaluation index to obtain a first investigation index, the first reply gist further comprising the first investigation index.

5. The method of claim 1, wherein, the first reply gist comprises a gist field and first description information corresponding to the gist field, the gist field comprising at least one of a main requirement, a secondary requirement, a necessary gist and a richness gist, and a first question type of the test question.

6. The method of claim 1, wherein, the assertion description comprises an assertion field and first description information corresponding to the assertion field, the assertion field comprising at least one of a main requirement, a secondary requirement, a necessary gist and a richness gist, and a first question type of the test question.

7. The method according to any one of claims 1 to 6, characterized in that, the evaluating the first answer based on the first answer, the assertion description and an evaluation standard to obtain a first evaluation result of the target language model comprises: The evaluation criteria are processed to obtain scoring criteria that can be processed by the evaluation model; The assertion description, the first answer, and the scoring criteria are input into the evaluation model to obtain a first evaluation result of the target language model.

8. The method of claim 7, wherein, If the test question has a reference answer, the reference answer is also input into the evaluation model to obtain the first evaluation result.

9. An evaluation device for a language model, characterized in that The device comprises: a question acquisition module configured to acquire a test question; a question processing module configured to process the test question using a target language model to obtain a first answer; an assertion acquisition module configured to acquire an assertion description corresponding to the test question, the assertion description comprising a first answer gist and first description information corresponding to the first answer gist; an answer evaluation module configured to evaluate the first answer based on the first answer, the assertion description, and evaluation criteria to obtain a first evaluation result of the target language model; The assertion acquisition module comprises: a gist analysis unit configured to acquire the first answer gist of the test question; an assertion generation unit configured to generate an assertion based on the first answer gist and at least one first evaluation index to obtain the assertion description; The assertion generation unit comprises: a second query subunit configured to query a second knowledge base based on the test question, the first answer gist, and at least one first evaluation index to obtain a second analysis rule, the second knowledge base being configured to represent a mapping relationship between test questions, answer gists, and evaluation indices and assertion analysis rules, the second analysis rule being configured to represent a prompt for the assertion description; an assertion model processing subunit configured to process the second analysis rule using an assertion generation model to obtain the assertion description.

10. An electronic device, comprising: comprise: a memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the evaluation method for a language model according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the evaluation method for a language model according to any one of claims 1 to 8.

12. A computer program product, characterised in that, comprise computer instructions, and the computer instructions are used to make a computer execute the evaluation method for a language model according to any one of claims 1 to 8.

Images