Assessment method and system of large language model, electronic equipment and storage medium

Abstract

The invention discloses an evaluation method and system of a large language model, electronic equipment and a storage medium, and relates to the technical field of model evaluation, and the method comprises the steps: generating a plurality of candidate answers of a preset question; generating an answer to the preset question by using the to-be-evaluated large language model; calculating a similarity mean value between the generated answer and each candidate answer by using each preset similarity calculation model; and according to the maximum similarity mean value, evaluating the to-be-evaluated large language model to obtain an evaluation result. According to the method, interference caused by subjective factors can be avoided, and compared with a keyword comparison mode, the method utilizes multiple similarity calculation models and evaluates the evaluation large language model with the maximum similarity mean value, so that the evaluation result is more accurate and fair, and the evaluation efficiency is improved. And the applicability of the large language model in specific fields such as the insurance field can be accurately evaluated.

Description

Background Art

[0002] Currently, evaluation methods for large language models can be divided into two main categories: one is manual evaluation of the answers to subjective questions generated by the large language model; the other is automatic evaluation of large language models using keyword comparison (such as metrics such as BLEU, ROUGE, and METEOR scores); and the other is automated evaluation of large language models based on the Bert model. These methods each have their advantages and disadvantages, but all have certain limitations, specifically:

[0003] When evaluating answers to subjective questions, traditional manual scoring is susceptible to personal bias and judgment, resulting in a lack of standardization and consistency. While keyword matching provides a certain degree of standardization, it can lead to inaccurate or unfair scoring when dealing with the diverse and creative answers generated by the model.

[0004] Moreover, existing evaluation methods cannot meet the application requirements of large models in specific industries (such as the insurance industry) and cannot accurately evaluate the capabilities and adaptability of large language models in specific fields. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and specifically provide a large language model evaluation method, system, electronic device and storage medium, as follows:

[0006] 1) In a first aspect, the present invention provides a method for evaluating a large language model, the specific technical solution of which is as follows:

[0007] Generate multiple candidate answers to pre-set questions;

[0008] Generate answers to pre-set questions using the large language model to be evaluated;

[0009] Using each preset similarity calculation model, calculate the mean similarity between the generated answer and each candidate answer;

[0010] According to the maximum similarity mean, the large language model to be evaluated is evaluated to obtain the evaluation result.

[0011] The beneficial effects of the large language model evaluation method provided by the present invention are as follows:

[0012] It can avoid the interference caused by subjective factors, and compared with the keyword comparison method, this application uses multiple similarity calculation models and evaluates the large language model with the maximum similarity mean, making the evaluation results more accurate and fair, and can accurately evaluate the applicability of the large language model in specific fields such as the insurance field.

[0013] Based on the above solution, the evaluation method of a large language model of the present invention can be further improved as follows.

[0014] Furthermore, the large language model to be evaluated is evaluated based on the maximum similarity mean, and the evaluation results are obtained, including:

[0015] Extract basic information and keywords from the generated answers;

[0016] The maximum similarity mean, extracted basic information and keywords are weighted respectively, and the large language model to be evaluated is evaluated according to the total weight to obtain the evaluation result.

[0017] The beneficial effect of adopting the above further technical solution is: combining the basic information and keywords in the generated answers to further improve the accuracy of the evaluation results.

[0018] Furthermore, the multiple candidate answers include: manually annotated candidate answers and candidate answers generated by a preset large language model.

[0019] Furthermore, it also includes: when the evaluation result meets the expected standard, using the large language model to be evaluated to generate an answer to the question raised by the user and provide it to the user.

[0020] Furthermore, the default questions are questions about insurance.

[0021] 2) In a second aspect, the present invention further provides an evaluation system for a large language model, the specific technical solution of which is as follows:

[0022] It includes candidate answer generation module, answer generation module, similarity calculation module and evaluation module;

[0023] The candidate answer generation module is used to: generate multiple candidate answers to preset questions;

[0024] The answer generation module is used to: generate answers to preset questions using the large language model to be evaluated;

[0025] The similarity calculation module is used to: calculate the mean similarity between the generated answer and each candidate answer using each preset similarity calculation model;

[0026] The evaluation module is used to evaluate the large language model to be evaluated based on the maximum similarity mean and obtain the evaluation result.

[0027] Based on the above solution, the large language model evaluation system of the present invention can also be improved as follows.

[0028] Furthermore, the evaluation module is specifically used to:

[0029] Extract basic information and keywords from the generated answers;

[0030] The maximum similarity mean, extracted basic information and keywords are weighted respectively, and the large language model to be evaluated is evaluated according to the total weight to obtain the evaluation result.

[0031] Furthermore, the multiple candidate answers include: manually annotated candidate answers and candidate answers generated by a preset large language model.

[0032] Furthermore, an interactive module is included, which is used to: when the evaluation result meets the expected standard, use the large language model to be evaluated to generate an answer to the question raised by the user and provide it to the user.

[0033] Furthermore, the default questions are questions about insurance.

[0034] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above-mentioned large language model evaluation methods.

[0035] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor so that the computer implements any of the above-mentioned large language model evaluation methods.

[0036] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 Schematic diagram of a flow chart of a method for evaluating a large language model according to an embodiment of the present invention;

[0039] Figure 2 Schematic diagram of the structure of a large language model evaluation system according to an embodiment of the present invention;

[0040] Figure 3 The figure is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0042] like Figure 1 As shown, a large language model evaluation method according to an embodiment of the present invention includes the following steps:

[0043] S1. Generate multiple candidate answers to preset questions;

[0044] Among them, the preset questions can be questions about insurance or questions in other fields, and can be set according to actual conditions.

[0045] The multiple candidate answers include: manually annotated candidate answers and candidate answers generated by a preset large language model. Specifically:

[0046] 1) Manually labeled candidate answers refer to the answers obtained by manually answering preset questions. The number of manually labeled candidate answers can be set according to actual circumstances, for example, 3 manually labeled candidate answers or 4 manually labeled candidate answers.

[0047] 2) The candidate answers generated by the preset large language model refer to the answers obtained by answering the preset questions using the preset large language model. It should be noted that, generally speaking, each time the preset question is input into the preset large language model, the candidate answers output by the preset large language model are different each time. Therefore, by inputting the preset question into the preset large language model multiple times, multiple candidate answers can be obtained.

[0048] S2. Generate answers to preset questions using the large language model to be evaluated;

[0049] Among them, the preset large language model can be ChatGPT, and the large language model to be evaluated is an open source model, such as Alibaba's Qwen, Baichuan's Baichuan, Zhipu's ChatGLM, Zero One Wanwu's Yi, etc. It can also be a large language model trained by an enterprise or individual.

[0050] S3. Calculate the mean similarity between the generated answer and each candidate answer using each preset similarity calculation model. Specifically:

[0051] Using any preset similarity calculation model, calculate the similarity between the generated answer and each candidate answer, and then calculate the mean to obtain the mean similarity between the generated answer and each candidate answer when the preset similarity calculation model is used for calculation, until the mean similarity calculated using different preset similarity calculation models is obtained.

[0052] The principle of calculating similarity of the BGE model is as follows:

[0053] The generated answer is embedded (embedding is the process of mapping high-dimensional text data to a low-dimensional space, which will generate a real-valued vector) to obtain a real-valued vector. Any candidate answer is also embedded to obtain a real-valued vector. The similarity between the two real-valued vectors is calculated through matrix multiplication, which is the similarity between the generated answer and the candidate answer.

[0054] Among them, multiple preset similarity calculation models include: BGE model, BCE model and GTE model, which can also be set according to actual conditions.

[0055] S4. Evaluate the large language model to be evaluated based on the maximum similarity mean to obtain an evaluation result.

[0056] Among them, the maximum similarity mean can be used as the score of the large language model to be evaluated, and the correspondence between scores in different ranges and evaluation results can also be pre-set. For example, when the score is between 0.85 and 1, the evaluation result is excellent; when the score is between 0.7 and 0.85, the evaluation result is good; when the score is lower than 0.7, the evaluation result is poor; based on this correspondence, the evaluation result of the large language model to be evaluated can be determined.

[0057] It should be noted that other evaluation criteria may also be set to implement the process of “evaluating the large language model to be evaluated based on the maximum similarity mean”.

[0058] A large language model evaluation method according to an embodiment of the present invention can avoid interference caused by subjective factors. Compared with the keyword comparison method, the present application utilizes multiple similarity calculation models and evaluates the large language model with the maximum similarity mean, making the evaluation results more accurate and fair, and can accurately evaluate the applicability of the large language model in specific fields such as the insurance field.

[0059] Optionally, in S4, the large language model to be evaluated is evaluated based on the maximum similarity mean, and an evaluation result is obtained, including:

[0060] S40, extracting basic information and keywords from the generated answers;

[0061] Among them, basic information and keywords can be set according to actual conditions. For example, the preset question is: "How much was the financial expenses of a certain bank in 2019?" The three candidate answers are: "The financial expenses of Industrial and Commercial Bank of China in 2019 were 12,345,678.9 yuan", "The financial expenses of Industrial and Commercial Bank of China in 2019 were 12,345,678.9 yuan", and "The financial expenses of a certain bank in 2019 were 12,345,678.9 yuan." The basic information is: "financial expenses" and "12,345,678.9 yuan", and the keywords are "financial expenses" and "2019".

[0062] Among them, the regular expression comparison method is used to extract the basic information and keywords in the generated answers. The training classification model can also be used to extract the basic information and keywords in the generated answers. The training classification model is CNN, RNN, LSTM, BiLSTM or Bert, etc. The keyword exact matching method can also be used to extract the basic information and keywords in the generated answers.

[0063] S41. Assign weights to the maximum similarity mean, the extracted basic information, and the keywords respectively, and evaluate the large language model to be evaluated based on the total weight to obtain an evaluation result.

[0064] Among them, the specific rules for weight assignment can be set according to actual conditions. The total weight can be used as the score of the large language model to be evaluated. The correspondence between the score and the evaluation result can be set in advance. Based on this correspondence, the evaluation result of the large language model to be evaluated is determined. The evaluation results can also be divided into excellent, good and poor.

[0065] In this embodiment, the basic information and keywords in the generated answers are combined to further improve the accuracy of the evaluation results.

[0066] Optionally, in the above technical solution, the following is further included:

[0067] S5. When the evaluation results meet the expected standards, the large language model to be evaluated is used to generate the answer to the question raised by the user and provide it to the user.

[0068] For example, if the expected standard is excellent, then when the evaluation result is excellent, the large language model to be evaluated is used to generate answers to the questions raised by the user and provide them to the user to achieve interaction, so as to improve the transaction rate of insurance policies.

[0069] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the scope of protection of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0070] like Figure 2 As shown, a large language model evaluation system 200 according to an embodiment of the present invention includes a candidate answer generation module 201, an answer generation module 202, a similarity calculation module 203 and an evaluation module 204;

[0071] The candidate answer generation module 201 is used to: generate multiple candidate answers to a preset question;

[0072] The answer generation module 202 is used to: generate answers to preset questions using the large language model to be evaluated;

[0073] The similarity calculation module 203 is used to: calculate the mean similarity between the generated answer and each candidate answer using each preset similarity calculation model;

[0074] The evaluation module 204 is used to evaluate the large language model to be evaluated according to the maximum similarity mean to obtain an evaluation result.

[0075] Optionally, in the above solution, the evaluation module 204 is specifically configured to:

[0076] Extract basic information and keywords from the generated answers;

[0077] The maximum similarity mean, extracted basic information and keywords are weighted respectively, and the large language model to be evaluated is evaluated according to the total weight to obtain the evaluation result.

[0078] Optionally, in the above technical solution, the multiple candidate answers include: manually annotated candidate answers and candidate answers generated by a preset large language model.

[0079] Optionally, the above technical solution further includes an interaction module, which is used to: when the evaluation result meets the expected standard, use the large language model to be evaluated to generate an answer to the question raised by the user and provide it to the user.

[0080] Optionally, in the above technical solution, the preset question is a question about insurance.

[0081] It should be noted that the beneficial effects of the large language model evaluation system 200 provided in the above embodiment are the same as the beneficial effects of the large language model evaluation method provided in the above embodiment, and will not be repeated here. In addition, when the system provided in the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0082] like Figure 3 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, which is coupled to a memory 310. The memory 310 stores at least one computer program 330. The at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above-mentioned large language model evaluation methods. Specifically:

[0083] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320 to enable the electronic device 300 to implement any large language model evaluation method provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for realizing device functions, which will not be described in detail here. The electronic device may specifically be a computer, etc.

[0084] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above-mentioned large language model evaluation methods.

[0085] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0086] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the aforementioned large language model evaluation methods.

[0087] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and to define a specific order or precedence. Where appropriate, the order used for similar objects may be interchanged, such that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0088] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented in the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0089] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0090] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for evaluating a large language model, characterized in that: include: Generate multiple candidate answers to pre-set questions; Generate answers to the preset questions using the large language model to be evaluated; Using each preset similarity calculation model, calculate the mean similarity between the generated answer and each candidate answer; The large language model to be evaluated is evaluated according to the maximum similarity mean to obtain an evaluation result.

2. The method for evaluating a large language model according to claim 1, wherein: The large language model to be evaluated is evaluated based on the maximum similarity mean to obtain an evaluation result, including: Extract basic information and keywords from the generated answers; Weights are assigned to the maximum similarity mean, the extracted basic information, and the keywords respectively, and the large language model to be evaluated is evaluated according to the total weight to obtain an evaluation result.

3. The method for evaluating a large language model according to claim 1, wherein: The multiple candidate answers include: manually annotated candidate answers and candidate answers generated by a preset large language model.

4. The method for evaluating a large language model according to any one of claims 1 to 3, wherein: Also includes: When the evaluation result meets the expected standard, the large language model to be evaluated is used to generate an answer to the question raised by the user and provide it to the user.

5. The method for evaluating a large language model according to any one of claims 1 to 3, characterized in that: The preset questions are questions about insurance.

6. A large language model evaluation system, characterized in that: It includes candidate answer generation module, answer generation module, similarity calculation module and evaluation module; The candidate answer generation module is used to: generate multiple candidate answers to preset questions; The answer generation module is used to: generate an answer to the preset question using the large language model to be evaluated; The similarity calculation module is used to: calculate the mean similarity between the generated answer and each candidate answer using each preset similarity calculation model; The evaluation module is used to evaluate the large language model to be evaluated according to the maximum similarity mean to obtain an evaluation result.

7. The large language model evaluation system according to claim 6, characterized in that: The evaluation module is specifically used for: Extract basic information and keywords from the generated answers; Weights are assigned to the maximum similarity mean, the extracted basic information, and the keywords respectively, and the large language model to be evaluated is evaluated according to the total weight to obtain an evaluation result.

8. The large language model evaluation system according to claim 6, characterized in that: The multiple candidate answers include: manually annotated candidate answers and candidate answers generated by a preset large language model.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, and the memory stores at least one computer program. The at least one computer program is loaded and executed by the processor so that the electronic device implements a large language model evaluation method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement a large language model evaluation method according to any one of claims 1 to 5.

Images