Evaluation method, device and equipment of dialogue language model, medium and program product

By integrating the answer analysis rules into the automatic generation mechanism of Prompt, the problem of low accuracy in the evaluation of dialogue language models in the existing technology is solved, and more efficient and consistent evaluation results are achieved, reducing manual intervention.

CN118940856APending Publication Date: 2024-11-12CHINA MOBILE COMM LTD RES INST +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410967627.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The evaluation method of the existing dialogue language model has reduced the accuracy of the evaluation due to the difference between Prompt and answer analysis rules, and cannot adapt to the output results of different models for effective analysis.

Method used

By integrating answer analysis rules into Prompt's automatic generation mechanism, a suitable Prompt is generated to evaluate the conversational language model, and dynamically adjust Prompt using historical evaluation results and evaluation sets to ensure that the output results comply with the answer analysis rules.

Benefits of technology

It improves the accuracy and automation of dialogue language model evaluation, reduces manual intervention, and significantly improves evaluation efficiency and consistency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940856A_ABST
    Figure CN118940856A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses an evaluation method and device of a dialogue language model, equipment, a medium and a program product, and the method comprises the steps that Prompt is generated according to configured data, and the configured data comprises answer analysis rules of the dialogue language model; and based on the Prompt and a pre-obtained evaluation set, evaluating a to-be-evaluated conversational language model, the evaluation set being data used for evaluating the conversational language model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence, and in particular, relates to a method, device, equipment, medium and program product for evaluating a conversational language model. Background Art

[0002] Model evaluation is a key link in the field of machine learning and artificial intelligence, and it is used to measure the performance of the model on a specific task. In related technologies, the evaluation method of a conversational language model includes: inputting a data set and a prompt word (Prompt) matching the data set into the conversational language model to be evaluated, parsing the output result of the conversational language model according to the answer parsing rules of the conversational language model, and obtaining the evaluation result of the conversational language model.

[0003] In the related art, during the evaluation process of a conversational language model, specific prompts and specific answer parsing rules are used. Due to the differences between different conversational language models, when using specific prompts to evaluate some conversational language models, the output results of the conversational language models cannot be parsed according to the specific answer parsing rules, thereby reducing the evaluation accuracy of the conversational language model. Summary of the invention

[0004] Embodiments of the present application provide a method, apparatus, device, medium, and program product for evaluating a conversational language model.

[0005] The present application embodiment provides a method for evaluating a conversational language model, the method comprising:

[0006] Generate a Prompt according to the configured data, wherein the configured data includes an answer parsing rule of a conversational language model;

[0007] The conversational language model to be evaluated is evaluated based on the Prompt and a pre-acquired evaluation set, where the evaluation set is data used to evaluate the conversational language model.

[0008] In some embodiments, the configured data also includes historical evaluation results of the conversational language model and Prompts corresponding to the historical evaluation results, wherein the historical evaluation results represent format matching scores between historical output results of the conversational language model and the answer parsing rules.

[0009] It can be seen that when the configured data includes historical evaluation results of a conversational language model and the Prompt corresponding to the historical evaluation results, before the conversational language model to be evaluated is evaluated, it is possible to understand the historical evaluation results used in the evaluation process of the conversational language model and the Prompt corresponding to the historical evaluation results, which helps to generate a Prompt suitable for the current evaluation based on the historical evaluation results and the Prompt corresponding to the historical evaluation results.

[0010] In some embodiments, the historical evaluation results of the conversational language model include at least a previous evaluation score of the conversational language model, and the previous evaluation score indicates a format matching score between a previous output result of the conversational language model and the answer parsing rule; after evaluating the conversational language model to be evaluated based on the Prompt and a pre-acquired evaluation set, the method further includes: in response to a current evaluation score of the conversational language model being higher than a previous evaluation score of the conversational language model, updating the historical evaluation results and the Prompt corresponding to the historical evaluation results in the configured data according to the current evaluation score of the conversational language model and the Prompt corresponding to the current evaluation score; the current evaluation score indicates a format matching score between a current output result of the conversational language model and the answer parsing rule.

[0011] It can be seen that if the current evaluation score of the conversational language model is higher than the previous evaluation score of the conversational language model, then after updating the historical evaluation results and the Prompt corresponding to the historical evaluation results in the configured data according to the current evaluation score of the conversational language model and the Prompt corresponding to the current evaluation score, the updated historical evaluation results saved in the configured data can reflect a higher format matching score between the output results of the conversational language model and the answer parsing rules, thereby helping to generate a Prompt suitable for evaluating the conversational language model according to the updated historical evaluation results and the Prompt corresponding to the updated historical evaluation results.

[0012] In some embodiments, before evaluating the conversational language model to be evaluated, the method further includes: randomly extracting the evaluation set from a pre-acquired data set; after evaluating the conversational language model to be evaluated based on the Prompt and the pre-acquired evaluation set, the method further includes: obtaining a current evaluation score of the conversational language model to be evaluated; the current evaluation score represents a format matching score between the current output result of the conversational language model and the answer parsing rule; when the conversational language model to be evaluated meets a preset evaluation end condition, evaluating the conversational language model to be evaluated according to the Prompt corresponding to the current evaluation score and the data set to obtain a final evaluation result of the conversational language model to be evaluated; the final evaluation result represents a correctness score of the output result of the conversational language model.

[0013] It can be seen that since the evaluation set is data randomly extracted from the data set, when the conversational language model to be evaluated meets the preset evaluation end conditions, the correctness of the output results of the conversational language model can be more accurately determined by using the full data set to evaluate the conversational language model.

[0014] In some embodiments, the method further includes: when the conversational language model to be evaluated does not meet a preset evaluation termination condition, repeatedly executing the steps of generating the Prompt, randomly extracting the evaluation set, and evaluating the conversational language model to be evaluated based on the Prompt and the evaluation set.

[0015] It can be seen that by repeatedly executing the steps of generating a prompt, randomly extracting an evaluation set, and evaluating the conversational language model to be evaluated based on the prompt and the evaluation set, it is beneficial to generate a prompt that is more suitable for evaluating the conversational language model.

[0016] In some embodiments, the preset evaluation end condition includes at least one of the following: the current evaluation score reaches a preset target value, the current evaluation score is less than or equal to the previous evaluation score of the conversational language model, and the number of evaluations of the conversational language model to be evaluated reaches a preset number of times.

[0017] In some embodiments, the evaluation set includes input text and annotation information, where the annotation information includes expected answer information corresponding to the input text or a label of the input text.

[0018] The present application also provides a conversational language model evaluation device, the device comprising:

[0019] A generation module, configured to generate a prompt according to configured data, wherein the configured data includes an answer parsing rule of a conversational language model;

[0020] A processing module is used to evaluate the conversational language model to be evaluated based on the Prompt and a pre-acquired evaluation set, where the evaluation set is data used to evaluate the conversational language model.

[0021] An embodiment of the present application also provides an electronic device, comprising a processor and a memory for storing a computer program that can be run on the processor; wherein the processor is used to run the computer program to execute any one of the above-mentioned conversational language model evaluation methods.

[0022] An embodiment of the present application also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned methods for evaluating the conversational language model is implemented.

[0023] An embodiment of the present application also provides a computer program product, including a computer program, which implements any of the above-mentioned methods for evaluating a conversational language model when executed by a processor.

[0024] It can be seen that the embodiment of the present application can integrate the answer parsing rules of the conversational language model into the automatic generation mechanism of Prompt, so that when the conversational language model to be evaluated is evaluated according to Prompt, the output result of the conversational language model can be made suitable for evaluation according to the answer parsing rules, thereby improving the evaluation accuracy of the conversational language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of an evaluation method for a conversational language model in the related art;

[0026] Figure 2 A flowchart of a method for evaluating a conversational language model according to an embodiment of the present application;

[0027] Figure 3 A flowchart for implementing conversational language model evaluation in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the structure of an evaluation device for a conversational language model according to an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] Figure 1A flowchart of an evaluation method for a conversational language model in related art, such as Figure 1 As shown, in the evaluation process of the conversational language model, first, a data set and a Prompt matching the data set can be obtained; in some embodiments, a data set for a certain evaluation capability can be prepared, generally one or more question-answer pair files containing certain aspects of the model's knowledge. For example, a Chinese Massive Multitask Language Understanding (CMMLU) data set can be prepared to test the model's Chinese knowledge capability. The data set covers 67 subject files from basic subjects to advanced professional levels. After preparing the data set, the Prompt can be manually adjusted according to the form, content, and evaluation framework requirements of the data set to ensure that it meets the requirements of the evaluation framework of the conversational language model. The implementation method of adjusting the Prompt can include modifying the text and format of the Prompt. The best Prompt can be found by manually adjusting the Prompt. For example, the official evaluation script of the CMMLU data set may use the following Prompt: "The following is a single-choice question about {subject}. Please directly give the correct answer option."

[0031] After obtaining the data set and the prompt that matches the data set, the data set and the prompt may be combined to form input content of the conversational language model to be evaluated; the conversational language model to be evaluated is called to process the input content to generate an output result.

[0032] In the related art, answer parsing rules can be formulated based on the output results of the prompt and the conversational language model to be evaluated, and the output results can be scored using the answer parsing rules. Exemplarily, in the related art, the method for scoring the output results can be a regular expression matching method, and the use of the regular expression matching method can ensure the accuracy and consistency of the scoring.

[0033] In the related art, the design of Prompt relies on the experience and subjective judgment of the evaluator and requires constant manual modification attempts, which may not fully cover all possible situations. That is, the manually designed Prompt may not be suitable for application in the evaluation scenario of the conversational language model, resulting in reduced evaluation accuracy of the conversational language model.

[0034] Furthermore, even if a prompt suitable for the evaluation scenario of the conversational language model can be found, due to the differences between different conversational language models, when using a specific prompt to evaluate some conversational language models, the output results of the conversational language model cannot be parsed according to specific answer parsing rules, thereby further reducing the evaluation accuracy of the conversational language model. In order to improve the evaluation accuracy of the conversational language model, it is necessary to modify and improve the answer parsing rules for different conversational language models during the evaluation process of the conversational language model.

[0035] In view of the technical problems of the related art, the technical solution of the embodiment of the present application is proposed. The embodiment of the present application can improve the automation and consistency of the conversational language model evaluation and reduce the dependence on manual intervention. Through a new automation method, the answer parsing rules are applied to the prompt automatic generation mechanism to solve the problem of manually adjusting the prompt and constantly modifying the answer parsing rules in the related art.

[0036] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings and examples. It should be understood that the embodiments provided herein are only used to explain the embodiments of the present application and are not intended to limit the embodiments of the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions recorded in the embodiments of the present application can be implemented in any combination.

[0037] It should be noted that, in the embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or device including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other related elements (such as steps in a method or units in a device, for example, a unit may be a part of a circuit, a part of a processor, a part of a program or software, etc.) in the method or device including the element.

[0038] The evaluation method for a conversational language model provided in the embodiment of the present application includes a series of steps, but the evaluation method for a conversational language model provided in the embodiment of the present application is not limited to the recorded steps. Similarly, the evaluation device for a conversational language model provided in the embodiment of the present application includes a series of modules, but the device provided in the embodiment of the present application is not limited to including the modules explicitly recorded, and may also include modules required to obtain relevant information or perform processing based on information.

[0039] Figure 2 Flow chart of a method for evaluating a conversational language model according to an embodiment of the present application, such as Figure 2 As shown, the process may include:

[0040] Step 201: Generate a prompt word Prompt according to the configured data, wherein the configured data includes answer parsing rules of the conversational language model.

[0041] In the embodiment of the present application, the answer parsing rules are information in a pre-prepared task description language; in actual scenarios, it is crucial to define a clear task description language, and by defining the task description language, the objectives and conditions of the evaluation can be accurately conveyed.

[0042] Step 202: Based on the Prompt and a pre-acquired evaluation set, the conversational language model to be evaluated is evaluated, where the evaluation set is data used to evaluate the conversational language model.

[0043] In some embodiments, the evaluation set may be randomly extracted from the pre-acquired data set. Exemplarily, the evaluation set may include input text and annotation information, wherein the annotation information includes the expected answer information corresponding to the input text or the label of the input text.

[0044] After generating Prompt and obtaining the evaluation set, the Prompt and the evaluation set can be input into the conversational language model to be evaluated, and then the output result of the conversational language model to be evaluated can be evaluated using the answer parsing rule to obtain the current evaluation score of the conversational language model. Here, the current evaluation score represents the format matching score of the current output result of the conversational language model and the answer parsing rule; illustratively, the answer parsing rule can be a rule for matching the output result of the conversational language model using a regular expression. In an exemplary scenario, the current output result of the conversational language model is "the correct answer is A", and the answer parsing rule is a rule for matching the format of the current output result of the conversational language model using a configured regular expression. When the configured regular expression includes "the correct answer is", it can be considered that the format of the current output result of the conversational language model is correct, and the current evaluation score of the conversational language model is a higher score; conversely, when the configured regular expression does not include "the correct answer is", it can be considered that the format of the current output result of the conversational language model is incorrect, and the current evaluation score of the conversational language model is a lower score.

[0045] In practical applications, steps 201 to 202 can be implemented based on a processor, and the processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0046] It can be seen that the embodiment of the present application can integrate the answer parsing rules of the conversational language model into the automatic generation mechanism of Prompt, so that when the conversational language model to be evaluated is evaluated according to Prompt, the output result of the conversational language model can be made suitable for evaluation according to the answer parsing rules, thereby improving the evaluation accuracy of the conversational language model.

[0047] Furthermore, by adopting the technical solution of the embodiment of the present application, the answer parsing rules of the conversational language model can be integrated into the automatic generation mechanism of Prompt, so that when the conversational language model to be evaluated is evaluated according to Prompt, there is no need to adjust the answer parsing rules according to the output results of the conversational language model. At the same time, the format of the output results of the conversational language model can also be made to meet expectations. In this way, the degree of automation and consistency of the conversational language model evaluation can be effectively improved, and the reliance on manual intervention can be reduced, thereby significantly improving the efficiency and accuracy of the conversational language model evaluation.

[0048] In some embodiments of the present application, the configured data also includes historical evaluation results of the conversational language model and the Prompt corresponding to the historical evaluation results; the historical evaluation results represent the format matching scores of the historical output results of the conversational language model and the answer parsing rules.

[0049] Here, the historical evaluation results may include at least the last evaluation score of the conversational language model, and the meaning of the last evaluation score may be understood with reference to the meaning of the current evaluation score.

[0050] The Prompt corresponding to the historical evaluation result may be the Prompt used to obtain the historical evaluation result. It is understandable that before the conversational language model to be evaluated is evaluated for the first time, there is no historical evaluation result and the Prompt corresponding to the historical evaluation result. When the historical evaluation result may include at least multiple evaluation scores of the conversational language model, obtaining the historical evaluation results of the conversational language model and the Prompt corresponding to the historical evaluation results may help understand the Prompt corresponding to the highest evaluation score in the evaluation process of the conversational language model.

[0051] It can be seen that when the configured data includes historical evaluation results of a conversational language model and the Prompt corresponding to the historical evaluation results, before the conversational language model to be evaluated is evaluated, it is possible to understand the historical evaluation results used in the evaluation process of the conversational language model and the Prompt corresponding to the historical evaluation results, which helps to generate a Prompt suitable for the current evaluation based on the historical evaluation results and the Prompt corresponding to the historical evaluation results.

[0052] In some embodiments of the present application, after evaluating the conversational language model to be evaluated based on the Prompt and a pre-acquired evaluation set, the method further includes: in response to a current evaluation score of the conversational language model being higher than a previous evaluation score of the conversational language model, updating the historical evaluation results and the Prompt corresponding to the historical evaluation results in the configured data according to the current evaluation score of the conversational language model and the Prompt corresponding to the current evaluation score.

[0053] It can be seen that if the current evaluation score of the conversational language model is higher than the previous evaluation score of the conversational language model, then after updating the historical evaluation results and the Prompt corresponding to the historical evaluation results in the configured data according to the current evaluation score of the conversational language model and the Prompt corresponding to the current evaluation score, the updated historical evaluation results saved in the configured data can reflect a higher format matching score between the output results of the conversational language model and the answer parsing rules, thereby helping to generate a Prompt suitable for evaluating the conversational language model according to the updated historical evaluation results and the Prompt corresponding to the updated historical evaluation results.

[0054] In some embodiments of the present application, before evaluating the conversational language model to be evaluated, the method further includes: randomly extracting the evaluation set from a pre-acquired data set;

[0055] After evaluating the conversational language model to be evaluated based on the Prompt and the pre-acquired evaluation set, the method further includes: obtaining a current evaluation score of the conversational language model to be evaluated; when the conversational language model to be evaluated meets a preset evaluation end condition, evaluating the conversational language model to be evaluated according to the Prompt corresponding to the current evaluation score and the data set to obtain a final evaluation result of the conversational language model to be evaluated, wherein the final evaluation result represents a correctness score of an output result of the conversational language model.

[0056] Exemplarily, the preset evaluation end condition includes at least one of the following: the current evaluation score reaches a preset target value, the current evaluation score is less than or equal to the previous evaluation score of the conversational language model, and the number of evaluations of the conversational language model to be evaluated reaches a preset number of times.

[0057] Here, the target value and the preset number of times can be preset according to actual conditions. When the current evaluation score is less than or equal to the previous evaluation score of the conversational language model, it can be considered that a Prompt that is better used to evaluate the conversational language model cannot be generated.

[0058] It can be seen that since the evaluation set is data randomly extracted from the data set, when the conversational language model to be evaluated meets the preset evaluation end conditions, the correctness of the output results of the conversational language model can be more accurately determined by using the full data set to evaluate the conversational language model.

[0059] In some embodiments of the present application, the method further includes: when the conversational language model to be evaluated does not meet a preset evaluation end condition, repeatedly executing the steps of generating a Prompt, randomly extracting an evaluation set, and evaluating the conversational language model to be evaluated based on the Prompt and the evaluation set.

[0060] It can be seen that by repeatedly executing the steps of generating a prompt, randomly extracting an evaluation set, and evaluating the conversational language model to be evaluated based on the prompt and the evaluation set, it is beneficial to generate a prompt that is more suitable for evaluating the conversational language model, thereby improving the evaluation accuracy of the conversational language model.

[0061] Figure 3 This is a flowchart for implementing conversational language model evaluation in an embodiment of the present application, refer to Figure 3 In the process of implementing the conversational language model evaluation, the configuration data is first prepared, and then the prompt can be automatically generated according to the configuration data. For example, the configuration data can be processed using an open source large language model to obtain the prompt.

[0062] An evaluation set is randomly extracted from the data set, and the conversational language model to be evaluated is evaluated according to the evaluation set and Prompt; illustratively, the evaluation set and Prompt can be input into the conversational language model to be evaluated, and the conversational language model is used to evaluate the evaluation set and Prompt to obtain the output result of the conversational language model; then the output result of the conversational language model to be evaluated is evaluated according to the answer parsing rule to obtain the current evaluation score of the conversational language model; if the current evaluation score of the conversational language model is higher than the previous evaluation score of the conversational language model, the historical evaluation result is updated according to the current evaluation score of the conversational language model and the Prompt corresponding to the current evaluation score.

[0063] If the conversational language model does not meet the evaluation end condition, the steps of preparing the configured data, generating a prompt, randomly extracting an evaluation set, and evaluating the conversational language model to be evaluated based on the prompt and the evaluation set are repeated until the conversational language model meets the evaluation end condition.

[0064] When the conversational language model meets the evaluation termination condition, the conversational language model to be evaluated is fully evaluated according to the prompt corresponding to the current evaluation score and the data set to obtain a final evaluation result of the conversational language model. In some embodiments, the step of performing the full evaluation can be repeated multiple times, so that multiple evaluations of the same conversational language model can be automatically achieved, which is conducive to effectively improving the accuracy and fairness of the conversational language model evaluation.

[0065] In the related art, a specific model evaluation method uses a specific prompt and a specific answer parsing rule, and cannot dynamically adjust the prompt according to the degree of match between the context and the answer parsing rule; if you want to perform a multi-dimensional model evaluation, you need to adjust the answer parsing rule according to the prompt of the new evaluation dimension, and the universality of the model evaluation framework cannot be achieved. The embodiment of the present application introduces a mechanism for automatically identifying a large model data set and generating a prompt according to the answer parsing rule, automatically completing the optimization of the prompt, and realizing a universal model evaluation method, which greatly reduces the probability of misjudgment of the evaluation result and significantly improves the accuracy of the model evaluation scoring link.

[0066] In the technical solution of the embodiment of the present application, the answer parsing rules can be used as the key input factor of the prompt automatic generation mechanism to guide the output format of the conversational language model to comply with the provisions of the answer parsing rules. Since the generation process of the prompt in the evaluation process of the conversational language model has integrated the result matching rules, no matter how the conversational language model to be evaluated changes, the answer parsing rules can perfectly adapt to the output form of the model without any modification, which ensures to a certain extent that the model answer will not be misjudged due to the matching rules. In addition, the technical solution of the embodiment of the present application does not require manual intervention, and can automatically generate prompts according to the data set and the answer parsing rules, automatically optimize the prompts, and finally obtain the optimal evaluation results of the conversational language model.

[0067] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0068] Figure 4 Schematic diagram of the structure of the evaluation device of the conversational language model of the embodiment of the present application. Figure 4 As shown, the device comprises:

[0069] A generation module 401, configured to generate a prompt according to configured data, wherein the configured data includes an answer parsing rule of a conversational language model;

[0070] Processing module 402 is used to evaluate the conversational language model to be evaluated based on the Prompt and a pre-acquired evaluation set, where the evaluation set is data used to evaluate the conversational language model.

[0071] In some embodiments, the configured data also includes historical evaluation results of the conversational language model and Prompts corresponding to the historical evaluation results; the historical evaluation results represent format matching scores between historical output results of the conversational language model and the answer parsing rules.

[0072] In some embodiments, the historical evaluation results of the conversational language model include at least a last evaluation score of the conversational language model; the last evaluation score represents a format matching score between a last output result of the conversational language model and the answer parsing rule;

[0073] The processing module 402 is further configured to, after evaluating the conversational language model to be evaluated based on the Prompt and a pre-acquired evaluation set, update the historical evaluation results and the Prompt corresponding to the historical evaluation results in the configured data in response to a current evaluation score of the conversational language model being higher than a previous evaluation score of the conversational language model, according to the current evaluation score of the conversational language model and the Prompt corresponding to the current evaluation score; the current evaluation score represents a format matching score between the current output result of the conversational language model and the answer parsing rule.

[0074] In some embodiments, the processing module 402 is further configured to randomly extract the evaluation set from the pre-acquired data set before evaluating the conversational language model to be evaluated;

[0075] The processing module 402 is further configured to obtain a current evaluation score of the conversational language model to be evaluated after evaluating the conversational language model to be evaluated based on the Prompt and the pre-acquired evaluation set; and when the conversational language model to be evaluated meets a preset evaluation end condition, evaluate the conversational language model to be evaluated according to the Prompt corresponding to the current evaluation score and the data set to obtain a final evaluation result of the conversational language model to be evaluated, wherein the final evaluation result represents a correctness score of an output result of the conversational language model.

[0076] In some embodiments, the processing module 402 is further used to repeatedly execute the steps of generating the Prompt, randomly extracting the evaluation set, and evaluating the conversational language model to be evaluated based on the Prompt and the evaluation set when the conversational language model to be evaluated does not meet a preset evaluation end condition.

[0077] In some embodiments, the preset evaluation end condition includes at least one of the following: the current evaluation score reaches a preset target value, the current evaluation score is less than or equal to the previous evaluation score of the conversational language model, and the number of evaluations of the conversational language model to be evaluated reaches a preset number of times.

[0078] In some embodiments, the evaluation set includes input text and annotation information, and the annotation information includes expected answer information corresponding to the input text or a label of the input text.

[0079] In practical applications, the generating module 401 and the processing module 402 may be implemented based on a processor.

[0080] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0081] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a terminal, a server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0082] Correspondingly, an embodiment of the present application further provides a computer program product, which includes computer executable instructions, and the computer executable instructions are used to implement any conversational language model evaluation method provided in the embodiment of the present application.

[0083] Accordingly, an embodiment of the present application further provides a computer storage medium, on which computer executable instructions are stored, and the computer executable instructions are used to implement any one of the conversational language model evaluation methods provided in the above embodiments.

[0084] An embodiment of the present application also provides an electronic device. Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the electronic device 50 may include:

[0085] Memory 501, used for storing executable instructions;

[0086] The processor 502 is configured to implement any one of the above-mentioned conversational language model evaluation methods when executing the executable instructions stored in the memory 501.

[0087] The processor 502 may be at least one of an ASIC, a DSP, a DSPD, a PLD, a FPGA, a CPU, a controller, a microcontroller, and a microprocessor.

[0088] The above-mentioned computer-readable storage medium and memory 502 can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various terminals including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0089] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0090] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0091] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0092] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0093] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0095] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for evaluating a conversational language model, characterized in that: The method comprises: Generate a prompt word Prompt according to the configured data, wherein the configured data includes an answer parsing rule of a conversational language model; The conversational language model to be evaluated is evaluated based on the Prompt and a pre-acquired evaluation set, where the evaluation set is data used to evaluate the conversational language model.

2. The method according to claim 1, characterized in that The configured data also includes historical evaluation results of the conversational language model and Prompts corresponding to the historical evaluation results, wherein the historical evaluation results represent format matching scores between historical output results of the conversational language model and the answer parsing rules.

3. The method according to claim 2, characterized in that The historical evaluation results of the conversational language model at least include a last evaluation score of the conversational language model, where the last evaluation score indicates a format matching score between a last output result of the conversational language model and the answer parsing rule; After evaluating the conversational language model to be evaluated based on the Prompt and a pre-acquired evaluation set, the method further includes: in response to a current evaluation score of the conversational language model being higher than a previous evaluation score of the conversational language model, updating the historical evaluation results and the Prompt corresponding to the historical evaluation results in the configured data according to the current evaluation score of the conversational language model and the Prompt corresponding to the current evaluation score; the current evaluation score represents a format matching score between the current output result of the conversational language model and the answer parsing rule.

4. The method according to claim 1, characterized in that Before evaluating the conversational language model to be evaluated, the method further includes: randomly extracting the evaluation set from a pre-acquired data set; After evaluating the conversational language model to be evaluated based on the Prompt and the pre-acquired evaluation set, the method further includes: Obtaining a current evaluation score of the conversational language model to be evaluated; the current evaluation score represents a format matching score between the current output result of the conversational language model and the answer parsing rule; When the conversational language model to be evaluated meets a preset evaluation end condition, the conversational language model to be evaluated is evaluated according to the Prompt corresponding to the current evaluation score and the data set to obtain a final evaluation result of the conversational language model to be evaluated, where the final evaluation result represents a correctness score of an output result of the conversational language model.

5. The method according to claim 4, characterized in that The method further includes: when the conversational language model to be evaluated does not meet a preset evaluation termination condition, repeatedly executing the steps of generating the Prompt, randomly extracting the evaluation set, and evaluating the conversational language model to be evaluated based on the Prompt and the evaluation set.

6. The method according to claim 4 or 5, characterized in that: The preset evaluation end condition includes at least one of the following: the current evaluation score reaches a preset target value, the current evaluation score is less than or equal to the previous evaluation score of the conversational language model, and the number of evaluations of the conversational language model to be evaluated reaches a preset number of times.

7. The method according to any one of claims 1 to 5, characterized in that: The evaluation set includes input text and annotation information, where the annotation information includes expected answer information corresponding to the input text or a label of the input text.

8. A conversational language model evaluation device, characterized in that: The device comprises: A generation module, used to generate a prompt word Prompt according to the configured data, wherein the configured data includes an answer parsing rule of a conversational language model; A processing module is used to evaluate the conversational language model to be evaluated based on the Prompt and a pre-acquired evaluation set, where the evaluation set is data used to evaluate the conversational language model.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory for storing a computer program that can be run on the processor; wherein, The processor is used to run the computer program to execute the conversational language model evaluation method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating a conversational language model according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the conversational language model evaluation method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Output optimization method of large language model

    CN117573846A

  • Evaluation method and device for multi-modal large language model

    CN117951275A