Assessment method and assessment device of large language model, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an evaluation method and device of a large language model, electronic equipment and a storage medium, and belongs to the technical field of model detection. The method comprises the following steps: acquiring feature information of each application scene in a service scene; according to the feature information of each application scene, determining a model evaluation mode corresponding to each application scene; for each application scene, evaluating the application of the large language model in the application scene by adopting a model evaluation mode corresponding to the application scene to obtain a model evaluation result corresponding to the application scene; and determining a target evaluation result of the large language model according to the model evaluation result corresponding to each application scene. According to the embodiment of the invention, different model evaluation modes can be selected based on different application scenes to evaluate the large language model, so that the accuracy of large language model evaluation can be improved.

Description

Technical Field

[0001] The present application relates to the field of model detection technology, and in particular to an evaluation method and evaluation device, electronic equipment and storage medium for a large language model. Background Art

[0002] By evaluating the large language model, you can determine how accurate the large language model is in generating answers, understanding text, and performing tasks.

[0003] The existing large language model evaluation generally uses a comparative evaluation method to evaluate the large language model. However, the comparative evaluation method can only evaluate questions with fixed standard answers. If the large language model is used to generate answers to questions without fixed standard answers, the comparative evaluation method will result in a low evaluation accuracy of the large language model. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a large language model evaluation method and evaluation device, electronic device and storage medium, aiming to improve the accuracy of large language model evaluation.

[0005] To achieve the above purpose, a first aspect of an embodiment of the present application proposes a method for evaluating a large language model, the method comprising:

[0006] Obtain characteristic information of each application scenario in the business scenario;

[0007] Determine the model evaluation method corresponding to each application scenario based on the characteristic information of each application scenario;

[0008] For each application scenario, the model evaluation method corresponding to the application scenario is used to evaluate the application of the large language model in the application scenario, and the model evaluation result corresponding to the application scenario is obtained;

[0009] Determine the target evaluation result of the large language model based on the model evaluation results corresponding to each application scenario.

[0010] In some embodiments, the characteristic information of the application scenario includes the question type of the test question, and the question type includes subjective questions and objective questions;

[0011] According to the characteristic information of each application scenario, determine the model evaluation method corresponding to each application scenario, including:

[0012] When the test question type of the application scenario is an objective question, the model evaluation method corresponding to the application scenario is determined to be a comparative evaluation;

[0013] When the question type of the test question of the application scenario is a subjective question, the model evaluation method corresponding to the application scenario is determined to be the referee model evaluation.

[0014] In some embodiments, the business scenario includes a first application scenario, and the model evaluation method corresponding to the first application scenario is a referee model evaluation;

[0015] For each application scenario, the model evaluation method corresponding to the application scenario is used to evaluate the application of the large language model in the application scenario, and the model evaluation results corresponding to the application scenario are obtained, including:

[0016] Obtaining a first test question for a first application scenario, a preset benchmark answer to the first test question, and a first large language model answer generated by the large language model based on the first test question;

[0017] Inputting the first information into the multiple referee models respectively, obtaining first score sets corresponding to the multiple referee models respectively, wherein the first information includes the first test question, the preset benchmark answer and the first large language model answer; the first score set corresponding to each referee model includes the first score of the referee model for the preset benchmark answer and the second score for the large language model answer;

[0018] According to the first score set corresponding to each referee model, a model evaluation result corresponding to the first application scenario is determined.

[0019] In some embodiments, in the first information, the preset reference answer and the first language model answer are arranged in a first arrangement order;

[0020] After obtaining the first test question of the first application scenario, the preset benchmark answer to the first test question, and the large language model answer generated by the large language model based on the first test question, before determining the model evaluation result corresponding to the first application scenario according to the score set corresponding to each referee model, the evaluation method further includes:

[0021] Inputting the second information into the plurality of referee models respectively, obtaining second score sets corresponding to the plurality of referee models respectively, wherein the second information includes the same information as the first information, but in the second information, the preset benchmark answer and the answer of the large language model to be tested are arranged in a second arrangement order opposite to the first arrangement order; the second score set corresponding to each referee model includes a third score of the referee model for the preset benchmark answer and a fourth score for the answer of the large language model;

[0022] Determining a model evaluation result corresponding to the first application scenario according to the first score set corresponding to each referee model includes:

[0023] According to the first score set and the second score set corresponding to each referee model, a model evaluation result corresponding to the first application scenario is determined.

[0024] In some embodiments, determining a model evaluation result corresponding to the first application scenario according to a first score set corresponding to each referee model includes:

[0025] Obtaining a first proportional score corresponding to each referee model, wherein the first proportional score corresponding to each referee model is a ratio of a second score in a first score set corresponding to the referee model to the first score;

[0026] According to the first proportional score corresponding to each referee model, a model evaluation result corresponding to the first application scenario is determined.

[0027] In some embodiments, the business scenario includes a second application scenario, and the model evaluation method corresponding to the second application scenario is comparative evaluation;

[0028] For each application scenario, the model evaluation method corresponding to the application scenario is used to evaluate the application of the large language model in the application scenario, and the model evaluation results corresponding to the application scenario are obtained, including:

[0029] Obtaining a second test question for a second application scenario, a historical answer to the second test question, and a second large language model answer generated by the large language model based on the second test question;

[0030] According to the historical answers and the large language model answer to the second test question, a model evaluation result corresponding to the second application scenario is determined.

[0031] In some embodiments, determining a model evaluation result corresponding to the second application scenario based on the historical answers and the large language model answers to the second type of test questions includes:

[0032] Determine the absolute value of the first difference between the historical answer and the answer of the second largest language model;

[0033] determining an evaluation score for the second test question based on a second difference in absolute value relative to a preset threshold;

[0034] According to the evaluation score, a model evaluation result corresponding to the second application scenario is determined.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes an evaluation device for a large language model, the evaluation device comprising:

[0036] The acquisition module is used to obtain the characteristic information of each application scenario in the business scenario;

[0037] A first determination module is used to determine a model evaluation method corresponding to each application scenario according to feature information of each application scenario;

[0038] An evaluation module is used to evaluate the application of the large language model in each application scenario using a model evaluation method corresponding to the application scenario, and obtain a model evaluation result corresponding to the application scenario;

[0039] The second determination module is used to determine the target evaluation result of the large language model according to the model evaluation results corresponding to each application scenario.

[0040] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0041] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0042] The evaluation method and evaluation device, electronic device and storage medium of the large language model proposed in the present application obtain the characteristic information of each application scenario in the business scenario, and determine the model evaluation method corresponding to each application scenario according to the characteristic information of each application scenario. For each application scenario, the model evaluation method corresponding to the application scenario is adopted to evaluate the application of the large language model in the application scenario, and the model evaluation result corresponding to the application scenario is obtained. Finally, the target evaluation result of the large language model is determined according to the model evaluation result corresponding to each application scenario. It can be seen that when evaluating the application of the large language model in the business scenario, the application of the large language model in each application scenario in the business scenario can be evaluated separately, and the evaluation of the application of the large language model in each application scenario can be implemented by using the model evaluation method determined based on the characteristic information of the application scenario. In this way, the evaluation method of the large language model in each application scenario can be matched with each application scenario, so that the evaluation accuracy of the large language model in each application scenario can be improved, and then the evaluation accuracy of the large language model in the business scenario can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a method for evaluating a large language model provided in an embodiment of the present application;

[0044] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0045] Figure 3 yes Figure 1 A flowchart of an embodiment of step S103 in FIG.

[0046] Figure 4 yes Figure 3 Flow chart of step S303 in FIG.

[0047] Figure 5 yes Figure 1 A flowchart of another embodiment of step S103 in FIG.

[0048] Figure 6 yes Figure 5 Flow chart of step S502 in FIG.

[0049] Figure 7 It is a structural schematic diagram of an evaluation device for a large language model provided in an embodiment of the present application;

[0050] Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] First, some nouns involved in this application are analyzed:

[0055] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0056] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0057] Based on this, the embodiments of the present application provide a large language model evaluation method and evaluation device, electronic device and storage medium, aiming to improve the accuracy of large language model evaluation.

[0058] The evaluation method, evaluation device, electronic device and storage medium of the large language model provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the evaluation method in the embodiments of the present application is described.

[0059] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0060] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0061] The evaluation method of the large language model provided in the embodiment of the present application relates to the field of artificial intelligence technology. The evaluation method of the large language model provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the evaluation method of the large language model, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0064] Figure 1 is an optional flowchart of the large language model evaluation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0065] Step S101, obtaining characteristic information of each application scenario in the business scenario.

[0066] In an embodiment of the present application, the business scenario may be any business scenario in which a large language model is applied. For example, the business scenario may include but is not limited to business scenarios in the insurance field.

[0067] Application scenarios can be understood as specific application environments in business scenarios. For example, business scenarios in the insurance field can include the following application scenarios: marketing application scenarios, underwriting application scenarios, claims application scenarios, service application scenarios, and basic capability application scenarios.

[0068] The characteristic information of the application scenario may include, but is not limited to, the type of the test questions of the application scenario, and the length of the standard answer or the historical answer.

[0069] Step S102: determining a model evaluation method corresponding to each application scenario based on characteristic information of each application scenario.

[0070] In the embodiment of the present application, each application scenario corresponds to a model evaluation method. The model evaluation method corresponding to each application scenario can be determined according to the characteristic information of the application scenario.

[0071] In some embodiments, the model evaluation method may include but is not limited to comparative evaluation and referee model evaluation. When the characteristic information of the application scenario is the length of the standard answer or the historical answer, and the length of the standard answer or the historical answer is shorter, the model evaluation method corresponding to the current application scenario may be comparative evaluation. When the characteristic information of the application scenario is the length of the standard answer or the historical answer, and the length of the standard answer or the historical answer is longer, the model evaluation method corresponding to the current application scenario may be referee model evaluation.

[0072] Step S103, after determining the model evaluation method corresponding to each application scenario, the model evaluation method corresponding to each application scenario is adopted to evaluate the application of the large language model in each application scenario to obtain the model evaluation result corresponding to the application scenario.

[0073] In an embodiment of the present application, for each application scenario, a model evaluation method corresponding to each application scenario is obtained respectively, and the application of each application scenario is evaluated using the model evaluation method corresponding to each application scenario respectively, so that a model evaluation result corresponding to each application scenario can be obtained.

[0074] In some embodiments, the large language model includes marketing application scenarios, underwriting application scenarios, claims application scenarios, service application scenarios, and basic capability application scenarios in the insurance field business scenarios. In one example, in the marketing application scenario, it includes modules such as insurance product consultation, personalized recommendation of insurance products, agent sales assistance, and marketing material design. For different modules, the model evaluation method used is also different. Test questions for modules such as insurance product consultation, personalized recommendation of insurance products, agent sales assistance, and marketing material design are obtained respectively, and the model evaluation method for modules such as insurance product consultation, personalized recommendation of insurance products, agent sales assistance, and marketing material design is determined according to the type of question.

[0075] In some embodiments, if the question type of the test question of the insurance product consulting module is determined to be a subjective question, then the model evaluation method of the insurance product consulting module is the referee model evaluation; if the question type of the test question of the insurance product personalized recommendation module is determined to be an objective question, then the model evaluation method of the insurance product personalized recommendation module is a comparative evaluation. If the question type of the test question of the agent sales assistance module is determined to be a subjective question, then the model evaluation method of the agent sales assistance module is the referee model evaluation. If the question type of the test question of the marketing material design module is determined to be a subjective question, then the model evaluation method of the marketing material design module is the referee model evaluation.

[0076] Step S104, determining the target evaluation result of the large language model according to the model evaluation results corresponding to each application scenario.

[0077] In the embodiment of the present application, by performing model evaluation on the application of each application scenario, the model evaluation results corresponding to each application scenario can be obtained. The model evaluation results corresponding to each application scenario are then summarized to obtain the target evaluation results of the large language model. The target evaluation result refers to the accuracy of the evaluation of the large language model.

[0078] In order to eliminate the deviation caused by randomness, the model evaluation result corresponding to each application scenario can be the average of multiple evaluations.

[0079] In one example, when determining the target evaluation result of the large language model, the sum of the scores obtained by the large language model in each type of evaluation above multiplied by the weight of the type of evaluation is determined as the target evaluation result of the large language model, that is, the performance of the large language model in the insurance business scenario. In another example, the average value of the model evaluation results corresponding to each application scenario can be determined as the target evaluation result of the large language model.

[0080] Steps S101 to S104 shown in the embodiment of the present application obtain characteristic information of each application scenario in the business scenario, and determine the model evaluation method corresponding to each application scenario based on the characteristic information of each application scenario. For each application scenario, the model evaluation method corresponding to the application scenario is used to evaluate the application of the large language model in the application scenario, and the model evaluation result corresponding to the application scenario is obtained. Finally, based on the model evaluation results corresponding to each application scenario, the target evaluation result of the large language model is determined. In this way, different model evaluation methods can be selected based on different application scenarios to evaluate the large language model, thereby improving the accuracy of the large language model evaluation.

[0081] See also Figure 2 In some implementations, the characteristic information of the application scenario includes the type of the test question, and the question type includes a subjective question and an objective question; step S102 may include but is not limited to steps S201 to S202:

[0082] Step S201, when the question type of the test question of the application scenario is an objective question, determining that the model evaluation method corresponding to the application scenario is a comparative evaluation;

[0083] Step S202, when the question type of the test question of the application scenario is a subjective question, determine that the model evaluation method corresponding to the application scenario is a referee model evaluation.

[0084] Specifically, the characteristic information of the application scenario may include, but is not limited to, the type of the test question, etc. The type of the test question may include subjective questions and objective questions. Subjective questions may be understood as questions without fixed answers. Objective questions may be understood as questions with fixed answers.

[0085] In step S201 of some embodiments, when the question type of the test question of the application scenario is an objective question, a comparative evaluation model evaluation method can be used for evaluation. Comparative evaluation can be understood as an evaluation method that compares the answer to be tested with the preset answer. The preset answer may include but is not limited to the standard answer or historical answer to the question. When performing the comparative evaluation, the evaluation can be performed based on the similarity between the answer to be tested and the preset answer. When the similarity is greater than a preset threshold, the answer to be tested is scored.

[0086] In step S202 of some embodiments, when the question type of the test question of the application scenario is a subjective question, the model evaluation method of the referee model evaluation can be used for evaluation. The referee model evaluation can be understood as a model evaluation method that uses other large language models as referee models to evaluate the current large language model. When performing the referee model evaluation, the score of the current large language model can be determined based on the referee model's score for the answer to be tested and the score for the benchmark answer.

[0087] By determining the type of test questions based on application scenarios and the model evaluation method corresponding to the application scenarios, we can specifically determine the model evaluation method that is adapted to the application scenarios, thereby improving the accuracy of large language model evaluation.

[0088] See also Figure 3 In some implementations, the business scenario includes a first application scenario, and the model evaluation method corresponding to the first application scenario is referee model evaluation; step S103 may include but is not limited to steps S301 to S303:

[0089] Step S301, obtaining a first test question of a first application scenario, a preset benchmark answer to the first test question, and a first large language model answer generated by a large language model based on the first test question;

[0090] Step S302, inputting the first information into the plurality of referee models respectively, obtaining first score sets corresponding to the plurality of referee models respectively, wherein the first information includes the first test question, the preset benchmark answer, and the first large language model answer; the first score set corresponding to each referee model includes the first score of the referee model for the preset benchmark answer, and the second score for the large language model answer;

[0091] Step S303, determining the model evaluation result corresponding to the first application scenario according to the first score set corresponding to each referee model.

[0092] Specifically, the business scenario may include a first application scenario, and the model evaluation method corresponding to the first application scenario is referee model evaluation. The first is only used to distinguish similar objects, and is not used to describe a specific order or sequence.

[0093] In step S301 of some embodiments, first obtain a first test question for a first application scenario, a preset benchmark answer to the first test question, and a first large language model answer generated by a large language model based on the first test question. The first test question can be understood as a test question corresponding to the first application scenario, and the first test question can refer to a question or a class of questions. The preset benchmark answer to the first test question can be obtained by inputting the first test question into other large language models and generating an answer through other large language models. For example, other large language models may include but are not limited to gpt4.0, etc.

[0094] In step S302 of some embodiments, the first information can be understood as information including a first test question, a preset benchmark answer, and a first large language model answer. The first information is input into a plurality of referee models respectively. After receiving the input first information, each referee model scores the preset benchmark answer and the first large language model answer respectively according to the first test question in the first information, and obtains the first score of the preset benchmark answer and the second score of the large language model answer by the referee model. The first score of each referee model for the preset benchmark answer and the second score of the large language model answer constitute the first score set corresponding to each referee model. Among them, the first and the second are only used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0095] In step S303 of some embodiments, after obtaining the first score set corresponding to each referee model, the model evaluation result corresponding to the first application scenario can be determined according to the first score set corresponding to each referee model. In one example, after obtaining the first score set corresponding to each referee model, the ratio of the second score of the referee model corresponding to each referee model for the large language model answer to the first score of the referee model for the preset benchmark answer is determined. The ratios corresponding to each referee model are summed to obtain the model evaluation result corresponding to the first application scenario.

[0096] By selecting an adapted referee model evaluation method for the first application scenario to perform large language model evaluation, the accuracy of the large language model evaluation can be improved.

[0097] In some implementations, in the first information, the preset reference answer and the first language model answer are arranged in a first arrangement order;

[0098] After obtaining the first test question of the first application scenario, the preset benchmark answer to the first test question, and the large language model answer generated by the large language model based on the first test question, before determining the model evaluation result corresponding to the first application scenario according to the score set corresponding to each referee model, the evaluation method further includes:

[0099] Inputting the second information into the plurality of referee models respectively, obtaining second score sets corresponding to the plurality of referee models respectively, wherein the second information includes the same information as the first information, but in the second information, the preset benchmark answer and the answer of the large language model to be tested are arranged in a second arrangement order opposite to the first arrangement order; the second score set corresponding to each referee model includes a third score of the referee model for the preset benchmark answer and a fourth score for the answer of the large language model;

[0100] Determining a model evaluation result corresponding to the first application scenario according to the first score set corresponding to each referee model includes:

[0101] According to the first score set and the second score set corresponding to each referee model, a model evaluation result corresponding to the first application scenario is determined.

[0102] Specifically, the first information may include a first test question, a preset reference answer, and a first language model answer. The preset reference answer and the first language model answer may be arranged in the order of the preset reference answer first and the first language model answer later, i.e., the first arrangement order.

[0103] Since the referee model is more inclined to prioritize the input result when comparing the pros and cons of multiple answers at the same time, in order to avoid the positional bias of the referee model, the first information is arranged in a second order to form the second information. The second information includes the same information as the first information, but in the second order, the preset benchmark answer and the first language model answer can be arranged in the order of the first language model answer first and the preset benchmark answer later.

[0104] The second information is inputted into a plurality of referee models respectively, and a second score set corresponding to each of the plurality of referee models can be obtained. The second score set corresponding to each referee model includes a third score of the referee model for the preset benchmark answer and a fourth score for the large language model answer.

[0105] To improve the accuracy of the large language model evaluation, the average value of the first score set and the second score set may be determined as the model evaluation result corresponding to the first application scenario.

[0106] By inputting the preset benchmark answer to the first test question and the large language model into the referee model based on different arrangement orders, the position bias of the referee model can be avoided and the accuracy of the large language model evaluation can be improved.

[0107] See also Figure 4 In some implementations, step S303 may include but is not limited to steps S401 to S402:

[0108] Step S401, obtaining a first proportional score corresponding to each referee model, wherein the first proportional score corresponding to each referee model is a ratio of a second score to a first score in a first score set corresponding to the referee model;

[0109] Step S402, determining a model evaluation result corresponding to the first application scenario according to the first proportional score corresponding to each referee model.

[0110] In step S401 of some embodiments, after the first information is respectively input into a plurality of referee models, a first score set corresponding to each of the plurality of referee models can be obtained. The first score set includes a first score of the referee model for a preset benchmark answer and a second score for the large language model answer. The ratio of the second score of the large language model answer to the first score of the preset benchmark answer is determined as the first proportional score corresponding to each referee model.

[0111] In step S402 of some embodiments, after obtaining the first proportional scores corresponding to the referee models, the first proportional scores corresponding to the referee models are aggregated to obtain the model evaluation result corresponding to the first application scenario. The model evaluation result corresponding to the first application scenario can be expressed as:

[0112]

[0113] By determining a first proportional score based on the first score of the referee model for the preset benchmark answer and the second score of the large language model answer, and aggregating the proportional scores to obtain the model evaluation result corresponding to the first application scenario, the accuracy of the model evaluation result corresponding to the first application scenario can be improved.

[0114] See also Figure 5 In some implementations, the business scenario includes a second application scenario, and the model evaluation method corresponding to the second application scenario is a comparative evaluation; step S103 may include but is not limited to steps S501 to S502:

[0115] Step S501, obtaining a second test question of a second application scenario, a historical answer to the second test question, and a second largest language model answer generated by the large language model based on the second test question;

[0116] Step S502: Determine a model evaluation result corresponding to the second application scenario based on the historical answers and the large language model answer to the second test question.

[0117] Specifically, the business scenario may also include a second application scenario, and the model evaluation method corresponding to the second application scenario is comparative evaluation. The second is only used to distinguish similar objects, and is not used to describe a specific order or sequence.

[0118] In step S501 of some embodiments, first obtain the second test question of the second application scenario, the historical answer to the second test question, and the second large language model answer generated by the large language model based on the second test question. The second test question can be understood as a test question corresponding to the second application scenario, and the second test question can refer to a question or a class of questions. The preset benchmark answer to the second test question can be obtained by inputting the second test question into other large language models and generating an answer through other large language models. For example, other large language models may include but are not limited to gpt4.0, etc.

[0119] In step S502 of some embodiments, after obtaining the historical answer and the large language model answer to the second test question, the model evaluation result corresponding to the second application scenario can be determined based on the historical answer and the large language model answer to the second test question.

[0120] In some embodiments, the similarity between the historical answer and the large language model answer to the second test question can be compared, and the model evaluation result corresponding to the second application scenario can be determined based on the similarity.

[0121] In some embodiments, a first difference between a historical answer and a large language model answer to a second test question can be determined, and then a second difference between the first difference and a preset threshold can be calculated, and a model evaluation result corresponding to the second application scenario can be determined based on the second difference.

[0122] By performing model evaluation in the second application scenario using a comparative evaluation method corresponding to the second application scenario, the accuracy of the model evaluation can be improved.

[0123] See also Figure 6 In some implementations, step S502 may include but is not limited to steps S601 to S602:

[0124] S601, determining an absolute value of a first difference between a historical answer and a second largest language model answer;

[0125] S602, determining an evaluation score for a second test question according to a second difference between a preset threshold and an absolute value;

[0126] S603: Determine a model evaluation result corresponding to the second application scenario according to the evaluation score.

[0127] Specifically, when determining the model evaluation result corresponding to the second application scenario based on the historical answer and the large language model answer to the second test question, the absolute value of the first difference between the historical answer and the second large language model answer can be determined first. Then determine the second difference between the preset threshold and the absolute value of the first difference. According to the second difference, the evaluation score of the second test question can be determined. Finally, the evaluation scores obtained for each piece of data are summed up to obtain the model evaluation result corresponding to the second application scenario. The model evaluation result corresponding to the second application scenario can be expressed as:

[0128]

[0129] Wherein, m represents a preset threshold.

[0130] In some embodiments, based on the application scenarios, the performance of the large language model in the insurance industry is divided into the following parts: marketing application scenarios, underwriting application scenarios, claims application scenarios, service application scenarios and basic capability application scenarios.

[0131] In some embodiments, the marketing application scenario includes an insurance product consulting module. In the insurance product consulting module, the evaluation method may include:

[0132] Obtain random insurance product corpus text, including insurance liability, insurance type, insurance period, insurance conditions, insurance terms, premiums, etc.

[0133] Randomly delete part of the insurance product corpus text, such as deleting the insurance conditions.

[0134] Generate test questions based on the deleted parts, such as: Based on the following information, what are the insurance conditions for this insurance and what details need to be paid attention to.

[0135] Use GPT4.0 to independently generate answers and use them as benchmark answers.

[0136] Adopt multiple referee models, such as qwen, cl aude, gemin i, etc.

[0137] The test question is combined with the benchmark answer generated by gpt4.0 and the answer generated by the large language model to be tested into a triple

[0138] The triples are given to multiple referee models for scoring, and the answer score of the large language model to be tested / the benchmark answer score is used as the score of the large language model to be tested.

[0139]

[0140] The benchmark answer generated by gpt4.0 is swapped with the answer generated by the large language model to be tested and then given to multiple referee models for re-scoring.

[0141] The average of the two large language model scores to be tested is taken as the final score of the large language model to be tested.

[0142] In some embodiments, the marketing application scenario includes a personalized insurance product recommendation module. In the personalized insurance product recommendation module, the evaluation method may include:

[0143] Obtain historical sales data and user portraits of randomly anonymized insurance products.

[0144] Input the user portrait into the large language model to be tested and generate test questions, such as: Based on the following user information, what insurance product would you recommend?

[0145] Compare the answers of the large language model with the actual historical sales data of insurance products and calculate the similarity.

[0146] The score of the large language model to be tested in this type of test is obtained based on the similarity. The higher the similarity, the higher the score.

[0147] In some embodiments, the marketing application scenario includes an agent sales assistance module. In the agent sales assistance module, the evaluation method may include:

[0148] Obtain existing random insurance products and desensitized customer profiles that have randomly purchased the products.

[0149] Combine them into questions, such as: We have this product, how do we recommend it to this customer? Please design a script.

[0150] Use GPT4.0 to independently generate answers and use them as benchmark answers.

[0151] Adopt multiple referee models, such as qwen, cl aude, gemin i, etc.

[0152] The test question is combined into a triple with the benchmark answer generated by gpt4.0 and the answer generated by the large language model to be tested.

[0153] The triples are given to multiple referee models for scoring, and the answer score of the large model to be tested / the benchmark answer score is used as the score of the large language model to be tested.

[0154]

[0155] The benchmark answer generated by gpt4.0 is swapped with the answer generated by the large language model to be tested and then given to multiple referee models for re-scoring.

[0156] The average of the two large model scores to be tested is taken as the final score of the large model to be tested.

[0157] In some embodiments, the marketing application scenario includes a marketing material design module. In the marketing material design module, the evaluation method may include:

[0158] Obtain historical marketing materials and material themes.

[0159] Construct evaluation questions through material themes, such as: Generate marketing slogans / circle of friends copywriting / marketing text messages / WeChat public account articles based on the following themes.

[0160] Use GPT4.0 to independently generate answers and use them as benchmark answers.

[0161] Adopt multiple referee models, such as qwen, cl aude, gemin i, etc.

[0162] The test question is combined into a triple with the benchmark answer generated by gpt4.0 and the answer generated by the large language model to be tested.

[0163] The triples are given to multiple referee models for scoring, and the answer score of the large model to be tested / the benchmark answer score is used as the score of the large language model to be tested.

[0164]

[0165] The benchmark answer generated by gpt4.0 is swapped with the answer generated by the large language model to be tested and then given to multiple referee models for re-scoring.

[0166] The average of the two large model scores to be tested is taken as the final score of the large model to be tested.

[0167] In some embodiments, the underwriting application scenario includes a risk assessment module. In the risk assessment module, the assessment method may include:

[0168] Obtain policy information, customer portraits, customer insurance history, customer claims data and external data submitted by historical randomly anonymized customers.

[0169] Input customer policy information, customer profile, customer insurance history and external data into the large language model to be tested, and generate questions, such as: Based on the following information, evaluate the risks of this policy.

[0170] The answer of the large language model is compared with the customer's claim data. If it is consistent with the claim information, it will be scored; if it is inconsistent, no score will be given.

[0171] Get the score of the large language model to be tested in this type of test.

[0172] In some embodiments, the underwriting application scenario includes an automatic underwriting module. In the automatic underwriting module, the evaluation method may include:

[0173] Obtain policy information submitted by historical randomly anonymized customers, customer portraits, customer historical insurance, external data, and historical underwriting results of manual underwriters.

[0174] Input customer policy information, customer profile, customer insurance history and external data into the large language model to be tested, and generate questions, such as: Based on the following information, evaluate whether the policy has been approved.

[0175] The answers of the large language model are compared with the historical underwriting results of manual underwriters. Points are awarded if the conclusions are consistent, and no points are awarded if the conclusions are inconsistent.

[0176] Get the score of the large language model to be tested in this type of test.

[0177] In some embodiments, the claim settlement application scenario includes an intelligent damage assessment module. In the intelligent damage assessment module, the assessment method may include:

[0178] Obtain historical randomly anonymized vehicle damage photos submitted by customers and historical manual damage assessment results.

[0179] The vehicle damage photos submitted by the customer are input into the large language model to be tested, and questions are generated, such as: Based on the following information, evaluate the vehicle damage amount.

[0180] Compare the answer of the large language model with the historical manual damage assessment results, take the threshold m as a fixed number, and if the difference is greater than m, it is recorded as 0 points.

[0181]

[0182] Get the score of the large language model to be tested in this type of test.

[0183] In some embodiments, the claims application scenario includes a risk warning module. In the risk warning module, the evaluation method may include:

[0184] Obtain historical data on random common natural disasters, such as typhoon paths, meteorological disasters, etc.

[0185] The historical data of natural disasters are input into the large language model to be tested, and questions are generated, such as: Based on the following information, evaluate the path of the typhoon in the next few days / whether meteorological disasters will occur in the next few days.

[0186] Compare the answer of the large language model with the historical data. If they are the same, score them; if they are different, no score is given.

[0187] Get the score of the large language model to be tested in this type of test.

[0188] In some embodiments, the claims application scenario includes a risk anti-fraud module. In the risk anti-fraud module, the evaluation method may include:

[0189] Obtain historical randomly anonymized report information submitted by customers and their claims anti-fraud judgment results, such as intentional traffic accidents, vehicle license plate duplication, etc.

[0190] The case report information submitted by the customer is input into the large language model to be tested, and questions are generated, such as: Based on the following information, evaluate whether there is a risk of fraud in this case.

[0191] The answer of the large language model is compared with the facts of its claims anti-fraud judgment results. If they are the same, points will be awarded, otherwise no points will be awarded.

[0192] Get the score of the large language model to be tested in this type of test.

[0193] In some embodiments, the service scenario includes a call summary module. In the call summary module, the evaluation method may include:

[0194] Obtain historical randomly desensitized call texts between agents and customers, as well as historical manually annotated customer intent, key information and other data.

[0195] Input the conversation text between the agent and the customer into the large language model to be tested, and generate questions, such as: Based on the following information, evaluate the customer's intention and key information.

[0196] The answers of the large language model are compared with the historical manually annotated data facts such as customer intent and key information. Points are scored if they are consistent, and no points are given if they are inconsistent.

[0197] Get the score of the large language model to be tested in this type of test.

[0198] In some embodiments, the service scenario includes an intelligent quality inspection module. In the intelligent quality inspection module, the evaluation method may include:

[0199] Obtain historical random desensitized agent compliance rules, call texts between agents and customers, and historical manually reviewed compliance issues.

[0200] The agent compliance rules and the conversation texts between the agent and the customer are input into the large language model to be tested, and questions are generated, such as: Based on the following information, determine whether the agent has compliance issues.

[0201] The answers of the large language model are compared with the historical manual review results. Points are awarded if they are consistent, and no points are awarded if they are inconsistent.

[0202] Get the score of the large language model to be tested in this type of test.

[0203] In one embodiment, in the basic capability module, the current industry-wide evaluation standards are used, including reliability, security, fairness, potential for abuse, reasoning ability, social norms, robustness, contextual relevance, language fluency, etc., so as to obtain the basic capability score of the large language model to be tested.

[0204] See also Figure 7 The embodiment of the present application further provides a large language model evaluation device, which can implement the above large language model evaluation method, and the device includes:

[0205] The acquisition module is used to obtain the characteristic information of each application scenario in the business scenario;

[0206] A first determination module is used to determine a model evaluation method corresponding to each application scenario according to feature information of each application scenario;

[0207] An evaluation module is used to evaluate the application of the large language model in each application scenario using a model evaluation method corresponding to the application scenario, and obtain a model evaluation result corresponding to the application scenario;

[0208] The second determination module is used to determine the target evaluation result of the large language model according to the model evaluation results corresponding to each application scenario.

[0209] The specific implementation of the large language model evaluation device is basically the same as the specific implementation of the large language model evaluation method described above, and will not be repeated here.

[0210] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned large language model evaluation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0211] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0212] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0213] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the evaluation method of the large language model of the embodiment of this application;

[0214] Input / output interface 803, used to implement information input and output;

[0215] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0216] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0217] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0218] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned large language model evaluation method.

[0219] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0220] The evaluation method and evaluation device, electronic device and storage medium of the large language model provided in the embodiment of the present application obtain the characteristic information of each application scenario in the business scenario; determine the model evaluation method corresponding to each application scenario according to the characteristic information of each application scenario; for each application scenario, use the model evaluation method corresponding to the application scenario to evaluate the application of the large language model in the application scenario, and obtain the model evaluation result corresponding to the application scenario; determine the target evaluation result of the large language model according to the model evaluation result corresponding to each application scenario. It can be seen that when evaluating the application of the large language model in the business scenario, the application of the large language model in each application scenario in the business scenario can be evaluated separately, and the evaluation of the application of the large language model in each application scenario can be implemented by using the model evaluation method determined based on the characteristic information of the application scenario. In this way, the evaluation method of the large language model in each application scenario can be matched with each application scenario, so as to improve the evaluation accuracy of the large language model in each application scenario, and then improve the evaluation accuracy of the large language model in the business scenario.

[0221] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0222] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0223] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0225] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0226] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0228] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0229] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0230] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0231] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for evaluating a large language model, characterized in that: The evaluation methods include: Obtain characteristic information of each application scenario in the business scenario; Determine the model evaluation method corresponding to each application scenario based on the characteristic information of each application scenario; For each application scenario, a model evaluation method corresponding to the application scenario is used to evaluate the application of the large language model in the application scenario to obtain a model evaluation result corresponding to the application scenario; According to the model evaluation results corresponding to each application scenario, the target evaluation result of the large language model is determined.

2. The evaluation method according to claim 1, characterized in that: The characteristic information of the application scenario includes the type of test questions, and the type of questions includes subjective questions and objective questions; Determining the model evaluation method corresponding to each application scenario according to the characteristic information of each application scenario includes: In a case where the question type of the test question of the application scenario is the objective question, determining that the model evaluation method corresponding to the application scenario is comparative evaluation; In the case where the question type of the test question of the application scenario is the subjective question, it is determined that the model evaluation method corresponding to the application scenario is the referee model evaluation.

3. The evaluation method according to claim 1, characterized in that: The business scenario includes a first application scenario, and the model evaluation method corresponding to the first application scenario is referee model evaluation; For each application scenario, the model evaluation method corresponding to the application scenario is adopted to evaluate the application of the large language model in the application scenario, and the model evaluation result corresponding to the application scenario is obtained, including: Obtaining a first test question for the first application scenario, a preset benchmark answer to the first test question, and a first large language model answer generated by the large language model based on the first test question; Inputting the first information into a plurality of referee models respectively, and obtaining first score sets corresponding to the plurality of referee models respectively, wherein the first information includes the first test question, the preset benchmark answer, and the first large language model answer; the first score set corresponding to each referee model includes a first score of the referee model for the preset benchmark answer, and a second score for the large language model answer; According to the first score set corresponding to each referee model, the model evaluation result corresponding to the first application scenario is determined.

4. The evaluation method according to claim 3, characterized in that: In the first information, the preset reference answer and the first language model answer are arranged in a first arrangement order; After obtaining the first test question of the first application scenario, the preset benchmark answer to the first test question, and the large language model answer generated by the large language model based on the first test question, and before determining the model evaluation result corresponding to the first application scenario according to the score set corresponding to each referee model, the evaluation method further includes: Inputting the second information into the multiple referee models respectively, obtaining second score sets corresponding to the multiple referee models respectively, wherein the information included in the second information is the same as the information included in the first information, but in the second information, the preset benchmark answer and the answer of the large language model to be tested are arranged in a second arrangement order opposite to the first arrangement order; the second score set corresponding to each referee model includes a third score of the referee model for the preset benchmark answer and a fourth score for the answer of the large language model; The step of determining the model evaluation result corresponding to the first application scenario according to the first score set corresponding to each referee model includes: According to the first score set and the second score set corresponding to each referee model, the model evaluation result corresponding to the first application scenario is determined.

5. The evaluation method according to claim 3, characterized in that: The step of determining the model evaluation result corresponding to the first application scenario according to the first score set corresponding to each referee model includes: Obtaining a first proportional score corresponding to each referee model, wherein the first proportional score corresponding to each referee model is a ratio of the second score to the first score in a first score set corresponding to the referee model; According to the first proportional score corresponding to each referee model, the model evaluation result corresponding to the first application scenario is determined.

6. The evaluation method according to claim 1, characterized in that: The business scenario includes a second application scenario, and the model evaluation method corresponding to the second application scenario is comparative evaluation; For each application scenario, the model evaluation method corresponding to the application scenario is adopted to evaluate the application of the large language model in the application scenario, and the model evaluation result corresponding to the application scenario is obtained, including: Obtaining a second test question for the second application scenario, a historical answer to the second test question, and a second large language model answer generated by the large language model based on the second test question; Determine a model evaluation result corresponding to the second application scenario based on the historical answers and the large language model answer to the second test question.

7. The evaluation method according to claim 6, characterized in that: Determining the model evaluation result corresponding to the second application scenario according to the historical answers and the large language model answer to the second type of test question includes: Determine an absolute value of a first difference between the historical answer and the second largest language model answer; determining an evaluation score for the second test question according to a second difference between a preset threshold and the absolute value; According to the evaluation score, a model evaluation result corresponding to the second application scenario is determined.

8. A large language model evaluation device, characterized in that: The evaluation device comprises: The acquisition module is used to obtain the characteristic information of each application scenario in the business scenario; A first determination module is used to determine a model evaluation method corresponding to each application scenario according to feature information of each application scenario; An evaluation module is used to evaluate the application of the large language model in each application scenario by using a model evaluation method corresponding to the application scenario to obtain a model evaluation result corresponding to the application scenario; The second determination module is used to determine the target evaluation result of the large language model according to the model evaluation results corresponding to each application scenario.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the large language model evaluation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for evaluating a large language model according to any one of claims 1 to 7 is implemented.

Images