Model performance evaluation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a model performance evaluation method and device, equipment and a storage medium, and the method comprises the steps: generating a business evaluation question and answer library based on a business question document and a preset question generation model; generating a to-be-evaluated answer of a target question in a business evaluation question and answer library through the to-be-evaluated model, wherein the business evaluation question and answer library comprises the target answer corresponding to the target question; scoring the to-be-evaluated answer through the target answer to obtain a target scoring result; and performing performance evaluation on the to-be-evaluated model based on the target scoring result. The to-be-evaluated answer of the target question in the business evaluation question and answer library is generated through the to-be-evaluated model, and the to-be-evaluated answer is scored through the target answer corresponding to the target question, so that performance evaluation is performed on the to-be-evaluated model based on the scoring result; therefore, the technical problems that in the prior art, performance evaluation is conducted on the language model based on a manual evaluation mode, time and labor are wasted, and expansibility is poor are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model performance evaluation method, device, equipment and storage medium. Background Art

[0002] With the rapid development of computer technology, language models have also received widespread attention and use. Language models are an important part of speech recognition systems and are mainly used in the language understanding stage of speech recognition and the process. Among them, the development and iteration of language models need to be quickly evaluated and verified. However, the existing manual evaluation method is time-consuming and labor-intensive and has poor scalability, making it impossible to perform performance evaluation and verification on language models.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a model performance evaluation method, device, equipment and storage medium, aiming to solve the technical problems in the prior art of evaluating the performance of language models based on manual evaluation, which is time-consuming, labor-intensive and has poor scalability.

[0005] To achieve the above object, the present invention provides a model performance evaluation method, which comprises the following steps:

[0006] Generate a business evaluation question and answer library based on business problem documents and preset question generation models;

[0007] Generate an answer to be evaluated for a target question in the business evaluation question and answer library through the model to be evaluated, wherein the business evaluation question and answer library includes a target answer corresponding to the target question;

[0008] Scoring the answer to be evaluated by using the target answer to obtain a target scoring result;

[0009] The performance of the model to be evaluated is evaluated based on the target scoring result.

[0010] Optionally, the step of generating the answer to be evaluated of the target question in the business evaluation question and answer library by using the model to be evaluated includes:

[0011] Constructing a question prompt by using the target question in the business evaluation question and answer library and the target answer corresponding to the target question;

[0012] An answer to the target question to be evaluated is generated based on the question prompt and the model to be evaluated.

[0013] Optionally, the step of generating an answer to be evaluated for the target question based on the question prompt and the model to be evaluated includes:

[0014] Generate a preset number of answers to be evaluated for the target question based on the question prompt and the model to be evaluated;

[0015] The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result comprises:

[0016] The target answer is used to score each answer to be evaluated a predicted number of times, and a target scoring result is obtained based on each scoring result.

[0017] Optionally, the step of generating a business evaluation question and answer library based on the business question document and the preset question generation model includes:

[0018] Generate several business problems through business problem documents and preset tasks;

[0019] Generate a target business question based on each business question and a preset question generation model, and obtain a business question answer corresponding to the target business question;

[0020] A business evaluation question and answer library is generated according to the target business question and the answer to the business question.

[0021] Optionally, the step of generating a target business question based on each business question and a preset question generation model, and obtaining a business question answer corresponding to the target business question includes:

[0022] Generate a target business problem based on each business problem and a preset problem generation model, and generate a business problem library based on the target business problem;

[0023] The business question library is traversed based on a preset language model integration tool to obtain business question answers corresponding to the target business question in the business question library.

[0024] Optionally, before the step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result, the step further includes:

[0025] Obtaining a question type corresponding to the target question, and determining a scoring strategy corresponding to the target question according to the question type;

[0026] The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result includes:

[0027] The answer to be evaluated is scored using the target answer and the scoring strategy to obtain a target scoring result.

[0028] Optionally, the question type includes: an objective question; the step of obtaining the question type corresponding to the target question and determining the scoring strategy corresponding to the target question according to the question type includes:

[0029] Obtaining the topic type corresponding to the target topic;

[0030] If the question type is the objective question, the scoring strategy corresponding to the target question is determined as the accurate scoring strategy;

[0031] The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result includes:

[0032] The answer to be evaluated is scored using the target answer and the accurate scoring strategy to obtain a target scoring result.

[0033] In addition, to achieve the above-mentioned purpose, the present invention also proposes a model performance evaluation device, which includes: a memory, a processor, and a model performance evaluation program stored in the memory and executable on the processor, and the model performance evaluation program is configured to implement the steps of the model performance evaluation method described above.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a model performance evaluation program is stored, and when the model performance evaluation program is executed by a processor, the steps of the model performance evaluation method described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present invention also proposes a model performance evaluation device, which includes: a question and answer library generation module, an answer generation module, a scoring module and a performance evaluation module;

[0036] The question and answer library generation module is used to generate a business evaluation question and answer library based on the business question document and the preset question generation model;

[0037] The answer generation module is used to generate an answer to be evaluated for a target question in the business evaluation question and answer library through a model to be evaluated, wherein the business evaluation question and answer library includes a target answer corresponding to the target question;

[0038] The scoring module is used to score the answer to be evaluated by using the target answer to obtain a target scoring result;

[0039] The performance evaluation module is used to perform performance evaluation on the model to be evaluated based on the target scoring result.

[0040] The present invention discloses a method for generating a business evaluation question and answer library based on a business problem document and a preset problem generation model; generating an answer to be evaluated for a target problem in the business evaluation question and answer library through the model to be evaluated, wherein the business evaluation question and answer library includes a target answer corresponding to the target problem; scoring the answer to be evaluated through the target answer to obtain a target scoring result; and performing performance evaluation on the model to be evaluated based on the target scoring result. Since the present invention generates an answer to be evaluated for a target problem in the business evaluation question and answer library through the model to be evaluated, and scores the answer to be evaluated through the target answer corresponding to the target problem, so as to perform performance evaluation on the model to be evaluated based on the scoring result, the technical problem that the performance evaluation of the language model based on manual evaluation in the prior art is time-consuming, labor-intensive and has poor scalability is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the structure of a model performance evaluation device for a hardware operating environment involved in an embodiment of the present invention;

[0042] Figure 2 It is a flow chart of the first embodiment of the model performance evaluation method of the present invention;

[0043] Figure 3 It is a flow chart of the second embodiment of the model performance evaluation method of the present invention;

[0044] Figure 4 It is a flow chart of the third embodiment of the model performance evaluation method of the present invention;

[0045] Figure 5 It is a structural block diagram of the first embodiment of the model performance evaluation device of the present invention.

[0046] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0048] Reference Figure 1 , Figure 1 It is a schematic diagram of the structure of a model performance evaluation device for a hardware operating environment involved in an embodiment of the present invention.

[0049] like Figure 1As shown, the model performance evaluation device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0050] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the model performance evaluation device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0051] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a model performance evaluation program.

[0052] exist Figure 1 In the model performance evaluation device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the model performance evaluation device of the present invention can be set in the model performance evaluation device, and the model performance evaluation device calls the model performance evaluation program stored in the memory 1005 through the processor 1001, and executes the model performance evaluation method provided by the embodiment of the present invention.

[0053] The present invention provides a method for evaluating model performance. Figure 2 , Figure 2 Schematic diagram of the flow chart of the first embodiment of the model performance evaluation method of the present invention.

[0054] In this embodiment, the model performance evaluation method includes the following steps:

[0055] Step S10: Generate a business evaluation question and answer library based on the business problem document and the preset question generation model.

[0056] It should be noted that the execution subject of the method of this embodiment can be a model performance evaluation device that evaluates the performance of the language model, or other model performance evaluation systems that can achieve the same or similar functions and include the model performance evaluation device. Here, the model performance evaluation method provided in this embodiment and the following embodiments is specifically described using a model performance evaluation system (hereinafter referred to as the system).

[0057] It should be understood that the above-mentioned business problem document may be a document composed of user content logs accumulated in some specific business scenarios and accumulated document-type business content.

[0058] It can be understood that the above-mentioned preset question generation model can be a model for generating business questions for users, such as GPT4, and this embodiment does not limit this.

[0059] It should be noted that the above-mentioned business evaluation question and answer library can be a question and answer library consisting of business questions generated by a preset question generation model, and answers to the business questions.

[0060] In the specific implementation, we can first build a business problem document through the user content logs accumulated in some specific business scenarios and the accumulated document-type business content, and determine the business problems based on the business problem document. Then, we can connect the generated business problems to the preset problem generation model GPT4, so that similar business problems can be generated through GPT4. At this time, we can determine the answers to these business problems, and generate a business evaluation question and answer library based on these business problems and the answers to the business problems.

[0061] Step S20: Generate an answer to be evaluated for a target question in the business evaluation question and answer library through the model to be evaluated, wherein the business evaluation question and answer library includes a target answer corresponding to the target question.

[0062] It should be understood that the above-mentioned model to be evaluated may be a fine-tuning model for performance evaluation in this embodiment.

[0063] It is understandable that the above target questions may be all questions stored in the business evaluation question and answer library. Correspondingly, the target answers may be the answers corresponding to the target questions stored in the business evaluation question and answer library.

[0064] It should be noted that the above-mentioned answer to be evaluated can be the answer corresponding to the target question generated by the model to be evaluated. In practical applications, this embodiment can effectively solve the problem that the manual evaluation method is laborious and has poor scalability by combining business-specific log data and using the model to be evaluated to expand questions and generate answers.

[0065] It should be understood that the step S20 may specifically include: constructing a question prompt through the target question in the business evaluation question and answer library and the target answer corresponding to the target question; generating the answer to be evaluated of the target question based on the question prompt and the model to be evaluated.

[0066] It is understandable that the above question prompt may be a prompt, wherein the prompt is some clues and hints given to the pre-trained model so that the raised question (downstream task) can be better understood.

[0067] Furthermore, in order to comprehensively test the sensitivity of the model to be evaluated to different prompts, the step of constructing question prompts through the target questions in the business evaluation question and answer library and the target answers corresponding to the target questions may include: constructing several different question prompts through the target questions in the business evaluation question and answer library and the target answers corresponding to the target questions.

[0068] It should be understood that using multiple prompts for model evaluation can reduce the impact of a specific prompt on the results to a certain extent, and improve the reliability and stability of the results. When a single prompt is used for model evaluation, some models may be more adaptable to the prompt, resulting in instability in the evaluation results. This embodiment can use multiple different prompts for testing, so that the model can be better evaluated.

[0069] In a specific implementation, the system can traverse the target questions and the target answers corresponding to the target questions in the business evaluation question and answer library to construct a question prompt, set the corresponding parameters of the model request, and send it to the fine-tuning model to be evaluated for result generation, so as to obtain the answer to be evaluated corresponding to each question in the business evaluation question and answer library. In addition, for the same question, multiple prompts can be constructed at the same time to improve the stability of the evaluation results, wherein this embodiment does not limit the specific number of constructed prompts.

[0070] Step S30: Scoring the answer to be evaluated by using the target answer to obtain a target scoring result.

[0071] Further, in order to improve the certainty of the answers to be evaluated generated by the model to be evaluated, the step of generating the answers to be evaluated for the target question based on the question prompt and the model to be evaluated may specifically include: generating a preset number of answers to be evaluated for the target question based on the question prompt and the model to be evaluated; correspondingly, the step S30 may specifically include: scoring each of the answers to be evaluated a predicted number of times through the target answer, and obtaining the target scoring result based on each scoring result.

[0072] It should be noted that, since the answers to be evaluated generated by the model to be evaluated are uncertain, in this embodiment, the model to be evaluated can generate answers to be evaluated corresponding to the target question multiple times, and accordingly a preset number of answers to be evaluated for the target question can be obtained, where the preset number can be 3 times, and this embodiment does not limit this.

[0073] Step S40: performing performance evaluation on the model to be evaluated based on the target scoring result.

[0074] In a specific implementation, this embodiment can also use a preset scoring model to score the answer to be evaluated multiple times, and summarize the multiple scoring results, average the scores of all answers under the same question, and finally output the target scoring result of the target question, so that the specific score of the model to be evaluated can be determined according to the target scoring result, so as to evaluate the performance of the model to be evaluated through the score. This embodiment can effectively reduce the volatility of the scoring model by setting different prompts for the same question and scoring multiple times using a preset scoring model, and widen the gap between different versions of the model in predicting the score, so that the overall score can be more credible.

[0075] Furthermore, in order to solve the problem of the gap between traditional academic assessment and actual business, after step S40, the method further includes: forming a target test question based on the target question, the target answer and different question prompts corresponding to the target question.

[0076] It should be understood that in this embodiment, multiple prompts are constructed for the same question at the same time, and correspond to the same target answer, thereby forming a test question. This allows corresponding test questions to be generated based on specific business, so as to solve the problem that there may still be a certain gap between traditional academic evaluation and actual business, and to provide more comprehensive and guiding optimization conclusions.

[0077] The present embodiment discloses generating a business evaluation question and answer library based on a business problem document and a preset question generation model; generating an answer to be evaluated for a target question in the business evaluation question and answer library through the model to be evaluated, and the business evaluation question and answer library includes a target answer corresponding to the target question; scoring the answer to be evaluated through the target answer to obtain a target scoring result; and performing performance evaluation on the model to be evaluated based on the target scoring result; since the present embodiment generates an answer to be evaluated for a target question in the business evaluation question and answer library through the model to be evaluated, and scores the answer to be evaluated through the target answer corresponding to the target question, so as to perform performance evaluation on the model to be evaluated based on the scoring result, the technical problem that the performance evaluation of the language model based on manual evaluation in the prior art is time-consuming, labor-intensive and has poor scalability is solved.

[0078] refer to Figure 3 , Figure 3 Schematic diagram of the flow chart of the second embodiment of the model performance evaluation method of the present invention.

[0079] Based on the above first embodiment, in this embodiment, step S10 includes:

[0080] Step S101: Generate a number of business problems through business problem documents and preset tasks.

[0081] It should be noted that, in this embodiment, a plurality of business problems may be determined through a pre-constructed business problem document and according to different task forms or themes, and this embodiment does not limit the specific number of business problems.

[0082] Step S102: Generate a target business question based on each business question and a preset question generation model, and obtain a business question answer corresponding to the target business question.

[0083] It should be understood that after generating a number of business problems, the generated business problems can be connected to a preset problem generation model, and similar target business problems can be generated through the preset problem generation model.

[0084] It should be noted that the step S102 may specifically include: generating a target business problem based on each business problem and a preset problem generation model, and generating a business problem library based on the target business problem; traversing the business problem library based on a preset language model integration tool to obtain the business problem answer corresponding to the target business problem in the business problem library.

[0085] It should be understood that the above-mentioned business problem library may be a space for storing all target business problems.

[0086] It is understandable that the above-mentioned preset language model integration tool can be a framework for building a language model application, such as a langchain tool, or other tools having the same or similar functions as the langchain tool, and this embodiment is not limited to this.

[0087] Furthermore, the steps of generating target business problems based on each business problem and a preset problem generation model, and generating a business problem library based on the target business problem may specifically include: iteratively generating target business problems based on each business problem and a preset problem generation model, wherein the evaluation index value between the target business problem and the historical problem library does not exceed a preset evaluation index threshold; when the number of problems of the target business problem exceeds a preset problem number threshold, generating a business problem library based on all generated target business problems.

[0088] It should be noted that the above evaluation index value may be a value used to characterize the degree of match between the target business question newly generated by the preset question generation model and the old question library, for example, a rouge value. The rouge value may be calculated by comparing the summary or answer generated by the model with the reference answer to obtain a corresponding score.

[0089] It should be understood that the above-mentioned preset question number threshold value may be the maximum number of required business questions. In practical applications, this embodiment may measure the matching degree between the generated target business questions and the historical question library by the rouge value. Specifically, the maximum rouge value between the newly generated questions and the old question library in the preset question generation model may be set to 0.3, that is, the rouge value between the newly generated questions and the old question library does not exceed 0.3, so as to obtain the above-mentioned target business questions, and when the number of target business questions reaches the maximum number of required business questions, a business question library is generated based on all target business questions.

[0090] It can be understood that before generating answers to questions, this embodiment can clean the business questions in the business question library, and after cleaning, it can use the langchain tool to access the preset question generation model GPT4, and traverse the business question library, and use the results generated by the model as the business question answers corresponding to the target business questions in the business question library.

[0091] Step S103: Generate a business evaluation question and answer library according to the target business question and the answer to the business question.

[0092] In the specific implementation, we can first determine multiple business problems based on pre-built business problem documents and different task forms or topics, and then connect them to the preset problem generation model, and iterate to generate target business problems similar to the business problems, where the rouge value between the newly generated target business problem and the old problem library does not exceed the preset evaluation index threshold of 0.3. When the number of generated target business problems meets the maximum number of required business problems, a business problem library is formed based on all generated target business problems. After the business problem library is formed, all business problems in the business problem library can be cleaned, and the langchain tool can be used to connect to GPT4, traverse the business problem library, use the results generated by the model as the answer to the business problem, and combine the target business problem and the answer corresponding to the target business problem to generate a business evaluation question and answer library. In the subsequent model performance evaluation, the answer to be evaluated can be scored directly according to the answer corresponding to the question in the business evaluation question and answer library, which effectively improves the efficiency of model performance evaluation.

[0093] This embodiment generates several business problems in the form of business problem documents and preset tasks, generates target business problems based on the business problem and preset problem generation model, and then generates a business evaluation question and answer library based on the target business problems and the business problem answers corresponding to the target business problems, thereby effectively improving the efficiency of model performance evaluation.

[0094] refer to Figure 4 , Figure 4 Schematic diagram of the flow chart of the third embodiment of the model performance evaluation method of the present invention.

[0095] Based on the above embodiments, in order to accurately score the answer to be evaluated, in this embodiment, before step S30, the method further includes:

[0096] Step S300: Obtain the question type corresponding to the target question, and determine the scoring strategy corresponding to the target question according to the question type.

[0097] It should be noted that the above-mentioned question types may include subjective questions and objective questions.

[0098] It should be understood that the above scoring strategy can be a strategy for scoring the target question. Since the target question includes two types, subjective questions and objective questions, and different question types correspond to different scoring strategies, this embodiment needs to first determine the question type corresponding to the target question, and score the target question according to the scoring strategy corresponding to the question type.

[0099] Accordingly, the step S30 includes: Step S30': scoring the answer to be evaluated by using the target answer and the scoring strategy to obtain a target scoring result.

[0100] It is understandable that when the target question is an objective question, the scoring strategy corresponding to the target question can be determined as an accurate scoring strategy, and the answer to be evaluated is scored by the target answer and the accurate scoring strategy to obtain the target scoring result. Among them, the accurate scoring strategy can be a strategy for scoring the answer to be evaluated strictly according to the calculation method of the target answer corresponding to the target question.

[0101] It should be noted that when the target question is a subjective question, the scoring strategy corresponding to the target question is determined as the scoring strategy of the scorer, and the answer to be evaluated is scored by the target answer and the scoring strategy of the scorer to obtain the target scoring result. Among them, the scoring strategy of the scorer can be a strategy of scoring by spelling the scoring prompt and widening the scoring range to allow the scorer (such as GPT4) to score. Among them, this embodiment can repeat the scoring 5 times to improve the accuracy of the scoring.

[0102] In a specific implementation, when scoring the answer to be evaluated, you can first determine the type of question corresponding to the target answer, and determine the scoring strategy corresponding to the target question based on the question type. Specifically, if the target question is an objective question, score it strictly according to the calculation method of the target answer corresponding to the target question; if the target question is a subjective question, you can spell the scoring prompt, expand the scoring range, and repeat the scoring multiple times to obtain the scoring result. This embodiment determines the corresponding scoring strategy according to the question type of the target question, and scores the answer to be evaluated according to the corresponding scoring strategy, so that the accuracy of scoring can be improved, thereby improving the accuracy of model performance evaluation.

[0103] This embodiment determines the scoring strategy corresponding to the target question by the question type corresponding to the target question, and scores the answer to be evaluated by the target answer and the scoring strategy to obtain the target scoring result. If the target question is an objective question, the scoring strategy corresponding to the target question is determined as the accurate scoring strategy, and the answer to be evaluated is scored by the target answer and the accurate scoring strategy to obtain the target scoring result; if the target question is a subjective question, the scoring strategy corresponding to the target question is determined as the scoring device scoring strategy, and the answer to be evaluated is scored by the target answer and the scoring device scoring strategy to obtain the target scoring result, thereby improving the accuracy of the model performance evaluation.

[0104] In addition, an embodiment of the present invention further proposes a storage medium, on which a model performance evaluation program is stored. When the model performance evaluation program is executed by a processor, the steps of the model performance evaluation method described above are implemented.

[0105] Reference Figure 5 , Figure 5 It is a structural block diagram of the first embodiment of the model performance evaluation device of the present invention.

[0106] like Figure 5 As shown, the model performance evaluation device proposed in the embodiment of the present invention includes: a question and answer library generation module 501, an answer generation module 502, a scoring module 503 and a performance evaluation module 504;

[0107] The question and answer library generation module 501 is used to generate a business evaluation question and answer library based on the business question document and the preset question generation model;

[0108] The answer generation module 502 is used to generate an answer to be evaluated for a target question in the business evaluation question and answer library through a model to be evaluated, wherein the business evaluation question and answer library includes a target answer corresponding to the target question;

[0109] The scoring module 503 is used to score the answer to be evaluated by using the target answer to obtain a target scoring result;

[0110] The performance evaluation module 504 is used to perform performance evaluation on the model to be evaluated based on the target scoring result.

[0111] Furthermore, the answer generation module 502 is also used to construct a question prompt through the target question in the business evaluation question and answer library and the target answer corresponding to the target question; and generate the answer to be evaluated of the target question based on the question prompt and the model to be evaluated.

[0112] Furthermore, the answer generation module 502 is further configured to generate a preset number of answers to be evaluated for the target question based on the question prompt and the model to be evaluated;

[0113] The scoring module 503 is further used to score each of the answers to be evaluated a predicted number of times using the target answer, and obtain a target scoring result based on each scoring result.

[0114] The model performance evaluation device of the present embodiment discloses generating a business evaluation question and answer library based on a business problem document and a preset question generation model; generating an answer to be evaluated for a target question in the business evaluation question and answer library through the model to be evaluated, and the business evaluation question and answer library includes a target answer corresponding to the target question; scoring the answer to be evaluated through the target answer to obtain a target scoring result; and performing performance evaluation on the model to be evaluated based on the target scoring result; since the present embodiment generates the answer to be evaluated for the target question in the business evaluation question and answer library through the model to be evaluated, and scores the answer to be evaluated through the target answer corresponding to the target question, so as to perform performance evaluation on the model to be evaluated based on the scoring result, the technical problem that the performance evaluation of the language model based on manual evaluation in the prior art is time-consuming, labor-intensive and has poor scalability is solved.

[0115] Based on the above-mentioned first embodiment of the model performance evaluation device of the present invention, a second embodiment of the model performance evaluation device of the present invention is proposed.

[0116] In this embodiment, the question and answer library generation module 501 is also used to generate a number of business problems in the form of business problem documents and preset tasks; generate target business problems based on each business problem and preset problem generation model, and obtain business problem answers corresponding to the target business problems; generate a business evaluation question and answer library based on the target business problems and the business problem answers.

[0117] Furthermore, the question and answer library generation module 501 is also used to generate target business questions based on various business questions and preset question generation models, and to generate a business question library based on the target business questions; and to traverse the business question library based on a preset language model integration tool to obtain business question answers corresponding to the target business questions in the business question library.

[0118] This embodiment generates several business problems in the form of business problem documents and preset tasks, generates target business problems based on the business problem and preset problem generation model, and then generates a business evaluation question and answer library based on the target business problems and the business problem answers corresponding to the target business problems, thereby effectively improving the efficiency of model performance evaluation.

[0119] Based on the above-mentioned device embodiments, a third embodiment of the model performance evaluation device of the present invention is proposed.

[0120] In this embodiment, the scoring module 503 is also used to obtain the question type corresponding to the target question, and determine the scoring strategy corresponding to the target question according to the question type; score the answer to be evaluated according to the target answer and the scoring strategy to obtain the target scoring result.

[0121] Furthermore, the scoring module 503 is also used to obtain the question type corresponding to the target question; if the question type is the objective question, the scoring strategy corresponding to the target question is determined as the accurate scoring strategy; the answer to be evaluated is scored using the target answer and the accurate scoring strategy to obtain the target scoring result.

[0122] This embodiment determines the scoring strategy corresponding to the target question by the question type corresponding to the target question, and scores the answer to be evaluated by the target answer and the scoring strategy to obtain the target scoring result. If the target question is an objective question, the scoring strategy corresponding to the target question is determined as the accurate scoring strategy, and the answer to be evaluated is scored by the target answer and the accurate scoring strategy to obtain the target scoring result; if the target question is a subjective question, the scoring strategy corresponding to the target question is determined as the scoring device scoring strategy, and the answer to be evaluated is scored by the target answer and the scoring device scoring strategy to obtain the target scoring result, thereby improving the accuracy of the model performance evaluation.

[0123] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0124] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0126] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

[0127] The present invention discloses A1, a model performance evaluation method, the model performance evaluation method comprising:

[0128] Generate a business evaluation question and answer library based on business problem documents and preset question generation models;

[0129] Generate an answer to be evaluated for a target question in the business evaluation question and answer library through the model to be evaluated, wherein the business evaluation question and answer library includes a target answer corresponding to the target question;

[0130] Scoring the answer to be evaluated by using the target answer to obtain a target scoring result;

[0131] The performance of the model to be evaluated is evaluated based on the target scoring result.

[0132] A2. The model performance evaluation method as described in A1, wherein the step of generating the answer to be evaluated for the target question in the business evaluation question and answer library through the model to be evaluated comprises:

[0133] Constructing a question prompt by using the target question in the business evaluation question and answer library and the target answer corresponding to the target question;

[0134] An answer to the target question to be evaluated is generated based on the question prompt and the model to be evaluated.

[0135] A3. The model performance evaluation method as described in A2, wherein the step of generating the answer to be evaluated for the target question based on the question prompt and the model to be evaluated comprises:

[0136] Generate a preset number of answers to be evaluated for the target question based on the question prompt and the model to be evaluated;

[0137] The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result comprises:

[0138] The target answer is used to score each answer to be evaluated a predicted number of times, and a target scoring result is obtained based on each scoring result.

[0139] A4. In the model performance evaluation method described in A1, the step of generating a business evaluation question and answer library based on the business problem document and the preset question generation model comprises:

[0140] Generate several business problems through business problem documents and preset tasks;

[0141] Generate a target business question based on each business question and a preset question generation model, and obtain a business question answer corresponding to the target business question;

[0142] A business evaluation question and answer library is generated according to the target business question and the answer to the business question.

[0143] A5. The model performance evaluation method as described in A4, wherein the step of generating a target business problem based on each business problem and a preset problem generation model and obtaining a business problem answer corresponding to the target business problem comprises:

[0144] Generate a target business problem based on each business problem and a preset problem generation model, and generate a business problem library based on the target business problem;

[0145] The business question library is traversed based on a preset language model integration tool to obtain business question answers corresponding to the target business question in the business question library.

[0146] A6. The model performance evaluation method as described in A1, before the step of scoring the answer to be evaluated by the target answer to obtain the target scoring result, further includes:

[0147] Obtaining a question type corresponding to the target question, and determining a scoring strategy corresponding to the target question according to the question type;

[0148] The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result includes:

[0149] The answer to be evaluated is scored using the target answer and the scoring strategy to obtain a target scoring result.

[0150] A7. The model performance evaluation method as described in A6, wherein the question type includes: objective questions; the step of obtaining the question type corresponding to the target question and determining the scoring strategy corresponding to the target question according to the question type includes:

[0151] Obtaining the topic type corresponding to the target topic;

[0152] If the question type is the objective question, the scoring strategy corresponding to the target question is determined as the accurate scoring strategy;

[0153] The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result includes:

[0154] The answer to be evaluated is scored using the target answer and the accurate scoring strategy to obtain a target scoring result.

[0155] A8. The model performance evaluation method as described in A7, wherein the question type further includes: a subjective question; and after the step of obtaining the question type corresponding to the target question, the method further includes:

[0156] If the question type is the subjective question, the scoring strategy corresponding to the target question is determined as the scoring strategy of the scorer;

[0157] The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result includes:

[0158] The answer to be evaluated is scored using the target answer and the scoring strategy of the scorer to obtain a target scoring result.

[0159] A9. The model performance evaluation method as described in A5, wherein the step of generating a target business problem based on each business problem and a preset problem generation model, and generating a business problem library based on the target business problem, comprises:

[0160] Iteratively generate a target business problem based on each business problem and a preset problem generation model, wherein the evaluation index value between the target business problem and the historical problem library does not exceed a preset evaluation index threshold;

[0161] When the number of the target business problems exceeds a preset problem number threshold, a business problem library is generated based on all generated target business problems.

[0162] A10. In the model performance evaluation method described in A2, the step of constructing a question prompt using the target question in the business evaluation question and answer library and the target answer corresponding to the target question comprises:

[0163] A number of different question prompts are constructed by using the target questions in the business evaluation question-answer library and the target answers corresponding to the target questions.

[0164] A11. The model performance evaluation method as described in A10, after the step of performing performance evaluation on the model to be evaluated based on the target scoring result, further comprising:

[0165] A target test question is formed based on the target question, the target answer and different question prompts corresponding to the target question.

[0166] The present invention also discloses B12, a model performance evaluation device, which includes: a memory, a processor, and a model performance evaluation program stored in the memory and executable on the processor, wherein the model performance evaluation program is configured to implement the steps of the model performance evaluation method described above.

[0167] The present invention also discloses C13, a storage medium, on which a model performance evaluation program is stored, and when the model performance evaluation program is executed by a processor, the steps of the model performance evaluation method described above are implemented.

[0168] The present invention also discloses D14, a model performance evaluation device, the model performance evaluation device comprising: a question and answer library generation module, an answer generation module, a scoring module and a performance evaluation module;

[0169] The question and answer library generation module is used to generate a business evaluation question and answer library based on the business question document and the preset question generation model;

[0170] The answer generation module is used to generate an answer to be evaluated for a target question in the business evaluation question and answer library through a model to be evaluated, wherein the business evaluation question and answer library includes a target answer corresponding to the target question;

[0171] The scoring module is used to score the answer to be evaluated by using the target answer to obtain a target scoring result;

[0172] The performance evaluation module is used to perform performance evaluation on the model to be evaluated based on the target scoring result.

[0173] D15, the model performance evaluation device as described in D14, wherein the answer generation module is further used to construct a question prompt through the target question in the business evaluation question and answer library and the target answer corresponding to the target question;

[0174] The answer generation module is further used to generate an answer to the target question to be evaluated based on the question prompt and the model to be evaluated.

[0175] D16. The model performance evaluation device as described in D15, wherein the answer generation module is further used to generate a preset number of answers to be evaluated for the target question based on the question prompt and the model to be evaluated;

[0176] The scoring module is further used to score each of the answers to be evaluated a predicted number of times through the target answer, and obtain a target scoring result based on each scoring result.

[0177] D17. The model performance evaluation device as described in D14, wherein the question and answer library generation module is further used to generate a plurality of business questions in the form of business question documents and preset tasks;

[0178] The question-answer library generation module is further used to generate target business questions based on each business question and a preset question generation model, and obtain business question answers corresponding to the target business questions;

[0179] The question-and-answer library generation module is also used to generate a business evaluation question-and-answer library based on the target business question and the answer to the business question.

[0180] D18. The model performance evaluation device as described in D17, wherein the question-answer library generation module is further used to generate a target business question based on each business question and a preset question generation model, and to generate a business question library based on the target business question;

[0181] The question and answer library generation module is also used to traverse the business question library based on a preset language model integration tool to obtain business question answers corresponding to the target business questions in the business question library.

[0182] D19. In the model performance evaluation device as described in D14, the scoring module is further used to obtain the question type corresponding to the target question, and determine the scoring strategy corresponding to the target question according to the question type;

[0183] The scoring module is further used to score the answer to be evaluated according to the target answer and the scoring strategy to obtain a target scoring result.

[0184] D20, the model performance evaluation device as described in D19, wherein the question type includes: objective questions; the scoring module is further used to obtain the question type corresponding to the target question;

[0185] The scoring module is further configured to determine the scoring strategy corresponding to the target question as an accurate scoring strategy if the question type is the objective question;

[0186] The scoring module is further used to score the answer to be evaluated using the target answer and the accurate scoring strategy to obtain a target scoring result.

Claims

1. A model performance evaluation method, characterized in that: The model performance evaluation method includes: Generate a business evaluation question and answer library based on business problem documents and preset question generation models; Generate an answer to be evaluated for a target question in the business evaluation question and answer library through the model to be evaluated, wherein the business evaluation question and answer library includes a target answer corresponding to the target question; Scoring the answer to be evaluated by using the target answer to obtain a target scoring result; The performance of the model to be evaluated is evaluated based on the target scoring result.

2. The model performance evaluation method according to claim 1, characterized in that: The step of generating the answer to be evaluated for the target question in the business evaluation question and answer library through the model to be evaluated includes: Constructing a question prompt by using the target question in the business evaluation question and answer library and the target answer corresponding to the target question; An answer to the target question to be evaluated is generated based on the question prompt and the model to be evaluated.

3. The model performance evaluation method according to claim 2, characterized in that: The step of generating the answer to be evaluated of the target question based on the question prompt and the model to be evaluated includes: Generate a preset number of answers to be evaluated for the target question based on the question prompt and the model to be evaluated; The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result comprises: The target answer is used to score each answer to be evaluated a predicted number of times, and a target scoring result is obtained based on each scoring result.

4. The model performance evaluation method according to claim 1, characterized in that: The step of generating a business evaluation question and answer library based on the business question document and the preset question generation model includes: Generate several business problems through business problem documents and preset tasks; Generate a target business question based on each business question and a preset question generation model, and obtain a business question answer corresponding to the target business question; A business evaluation question and answer library is generated according to the target business question and the answer to the business question.

5. The model performance evaluation method according to claim 4, characterized in that: The step of generating a target business question based on each business question and a preset question generation model, and obtaining a business question answer corresponding to the target business question, includes: Generate a target business problem based on each business problem and a preset problem generation model, and generate a business problem library based on the target business problem; The business question library is traversed based on a preset language model integration tool to obtain business question answers corresponding to the target business question in the business question library.

6. The model performance evaluation method according to claim 1, characterized in that: Before the step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result, the method further includes: Obtaining a question type corresponding to the target question, and determining a scoring strategy corresponding to the target question according to the question type; The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result includes: The answer to be evaluated is scored using the target answer and the scoring strategy to obtain a target scoring result.

7. The model performance evaluation method according to claim 6, characterized in that: The question type includes: objective questions; the step of obtaining the question type corresponding to the target question and determining the scoring strategy corresponding to the target question according to the question type includes: Obtaining the topic type corresponding to the target topic; If the question type is the objective question, the scoring strategy corresponding to the target question is determined as the accurate scoring strategy; The step of scoring the answer to be evaluated by using the target answer to obtain a target scoring result includes: The answer to be evaluated is scored using the target answer and the accurate scoring strategy to obtain a target scoring result.

8. A model performance evaluation device, characterized in that: The device comprises: a memory, a processor, and a model performance evaluation program stored in the memory and executable on the processor, wherein the model performance evaluation program is configured to implement the steps of the model performance evaluation method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium stores a model performance evaluation program, which, when executed by a processor, implements the steps of the model performance evaluation method according to any one of claims 1 to 7.

10. A model performance evaluation device, characterized in that: The model performance evaluation device comprises: a question and answer library generation module, an answer generation module, a scoring module and a performance evaluation module; The question and answer library generation module is used to generate a business evaluation question and answer library based on the business question document and the preset question generation model; The answer generation module is used to generate an answer to be evaluated for a target question in the business evaluation question and answer library through a model to be evaluated, wherein the business evaluation question and answer library includes a target answer corresponding to the target question; The scoring module is used to score the answer to be evaluated by using the target answer to obtain a target scoring result; The performance evaluation module is used to perform performance evaluation on the model to be evaluated based on the target scoring result.

Images