Large language model evaluation method and device based on judgment model, equipment and medium

Abstract

The invention relates to a judgment model-based big language model evaluation method, device and equipment and a medium, and the method comprises the steps: inputting pre-obtained test problems into a plurality of to-be-evaluated big language models, and obtaining an output result of each big language model; inputting the test problem, the output result of each large language model and a preset first guide instruction into a pre-established judgment model to obtain a model evaluation result output by the judgment model, the first guide instruction is used for guiding the judgment model to evaluate the model performance of the multiple large language models by adopting target evaluation standards such as accuracy, correlation, creativity, logic continuity and information integrity, and the model evaluation result comprises information of the target large language model with the optimal performance. According to the method, the output results of the multiple large language models for the same problem are input into the judgment model, the judgment model is guided to evaluate the output results of the large language models from multiple angles, and the large language models can be automatically and comprehensively evaluated.

Description

Technical Field

[0001] The present disclosure relates to the technical field of natural language processing, and in particular to a large language model evaluation method, device, equipment and medium based on a referee model. Background Art

[0002] With the development of artificial intelligence technology, large language models (LLMs) have made significant progress in the field of natural language processing and are widely used in question-answering models, bringing unprecedented convenience to people's lives. Large language models can understand and generate natural language by learning from large-scale corpora, thereby realizing intelligent question-answering in various scenarios. However, in practical applications, the performance of different large language models varies greatly, which directly affects the service quality and user experience of the model. In order to screen out models with excellent performance and strong adaptability and continuously optimize them, it is necessary to objectively and comprehensively evaluate the model performance.

[0003] Existing large language model evaluation methods mainly include static evaluation methods and manual evaluation methods. Among them, the static evaluation method scores based on the consistency and similarity between the model output and the reference answer. The evaluation index is single and it is difficult to comprehensively evaluate the model performance. Although the manual evaluation method can provide flexible evaluation criteria, it is difficult to promote and apply it in large-scale model evaluation tasks due to high labor costs and time consumption. Therefore, how to comprehensively evaluate large language models through technical means has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a large language model evaluation method, device, equipment and medium based on a referee model.

[0005] A first aspect of the embodiments of the present disclosure provides a large language model evaluation method based on a referee model, the method comprising:

[0006] Input the pre-acquired test questions into the multiple large language models to be evaluated respectively, and obtain the output results of each large language model;

[0007] The test question, the output results of each large language model and a preset first guiding instruction are input into a pre-established referee model to obtain a model evaluation result output by the referee model, wherein the first guiding instruction is used to guide the referee model to evaluate the model performance of the multiple large language models using target evaluation criteria, wherein the target evaluation criteria include accuracy, relevance, creativity, logical coherence and information completeness, and the model evaluation result includes information of a target large language model with the best model performance among the multiple large language models.

[0008] A second aspect of the embodiments of the present disclosure provides a large language model evaluation device based on a referee model, the device comprising:

[0009] An input module, used to input the pre-acquired test questions into the multiple large language models to be evaluated, and obtain the output results of each large language model;

[0010] An evaluation module is used to input the test question, the output results of each large language model and a preset first guiding instruction into a pre-established referee model to obtain a model evaluation result output by the referee model, wherein the first guiding instruction is used to guide the referee model to evaluate the model performance of the multiple large language models using a target evaluation standard, wherein the target evaluation standard includes accuracy, relevance, creativity, logical coherence and information completeness, and the model evaluation result includes information of a target large language model with the best model performance among the multiple large language models.

[0011] A third aspect of an embodiment of the present disclosure provides a computer device, including a memory and a processor, and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the large language model evaluation method based on the referee model as described in the first aspect above is implemented.

[0012] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the large language model evaluation method based on the referee model as described in the first aspect above is implemented.

[0013] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0014] In the large language model evaluation method, device, equipment and medium based on the referee model provided in the embodiments of the present disclosure, by inputting the pre-acquired test questions into the multiple large language models to be evaluated respectively, and obtaining the output results of each large language model, the test questions, the output results of each large language model and the preset first guiding instruction are input into the pre-established referee model to obtain the model evaluation result output by the referee model, the first guiding instruction is used to guide the referee model to use the target evaluation standard to evaluate the model performance of the multiple large language models, the target evaluation standard includes accuracy, relevance, creativity, logical coherence and information integrity, the model evaluation result includes the information of the target large language model with the best model performance among the multiple large language models, the output results of multiple large language models for the same problem can be input into the referee model, and the referee model is guided to evaluate the output results of each large language model from multiple angles such as accuracy, relevance, creativity, logical coherence and information integrity, so as to achieve a comprehensive evaluation of the model performance, and at the same time replace the manual evaluation method, reduce the labor cost and improve the evaluation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 is a flow chart of a large language model evaluation method based on a referee model provided by an embodiment of the present disclosure;

[0018] Figure 2 is a flow chart of a method for integrating evaluation basis provided by an embodiment of the present disclosure;

[0019] Figure 3 is a flow chart of a method for determining an evaluation dimension corresponding to an evaluation basis provided by an embodiment of the present disclosure;

[0020] Figure 4 is a structural diagram of a large language model evaluation device based on a referee model provided by an embodiment of the present disclosure;

[0021] Figure 5 It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0024] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0025] Figure 1 This is a flowchart of a large language model evaluation method based on a referee model provided by an embodiment of the present disclosure. The method can be executed by a large language model evaluation device based on a referee model. The large language model evaluation device based on a referee model can be implemented in software and / or hardware. The large language model evaluation device based on a referee model can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc. Figure 1 As shown, the large language model evaluation method based on the referee model provided in this embodiment includes the following steps:

[0026] S101. Input the pre-acquired test questions into a plurality of large language models to be evaluated respectively, and obtain the output results of each large language model.

[0027] The multiple large language models to be evaluated in the disclosed embodiments may be generative large language models defined by different architectures, training data or fine-tuning methods, and their main function is to generate natural language output to answer given input questions.

[0028] In the disclosed embodiment, the large language model evaluation device based on the referee model can obtain information of multiple large language models to be evaluated and obtain one or more test questions from a preset question library when receiving evaluation instructions from the user for multiple large language models. It can also receive test questions uploaded by the user, and after obtaining the test questions, input the test questions into the multiple large language models to be evaluated respectively, and the multiple large language models analyze and process the test questions, generate answers corresponding to the test questions and output them. The large language model evaluation device based on the referee model can obtain the output results of each large language model and record the corresponding relationship between the output results and the large language model.

[0029] In an exemplary implementation of the disclosed embodiment, the large language model evaluation device based on the referee model can, after obtaining the output results of each large language model, splice the test question and the output results of each large language model according to a preset structured template. For example, for the test question Q, the output result of the large language model A can be expressed as A(Q), and the output result of the large language model B can be expressed as B(Q). The spliced ​​content can be expressed as follows:

[0030] User question: the core question Q entered;

[0031] Answer A: The answer A(Q) generated by model A for question Q;

[0032] Answer B: Answer B(Q) generated by model B for question Q.

[0033] S102. Input the test question, the output results of each large language model and the preset first guiding instruction into a pre-established referee model to obtain a model evaluation result output by the referee model. The first guiding instruction is used to guide the referee model to evaluate the model performance of multiple large language models using target evaluation criteria. The target evaluation criteria include accuracy, relevance, creativity, logical coherence and information completeness. The model evaluation result includes information of the target large language model with the best model performance among the multiple large language models.

[0034] In the disclosed embodiments, the target evaluation criteria may include accuracy, relevance, creativity, logical coherence and information completeness, wherein accuracy is used to evaluate the consistency between the model answer and the standard answer, relevance is used to evaluate the relevance between the model answer and the input question, creativity is used to evaluate the innovation and uniqueness displayed by the model answer, logical coherence is used to evaluate the rationality of the model answer in semantics and reasoning, and information completeness is used to evaluate whether the model answer comprehensively covers the core points of the question. The standard answer in the accuracy evaluation can be input into the referee model with the first guidance instruction, or it can be determined by the referee model searching for relevant information in a pre-established knowledge graph based on the test question, and there is no limitation here.

[0035] The referee model in the embodiment of the present disclosure can be understood as a large language model containing large-scale parameters, which has powerful natural language understanding and generation capabilities.

[0036] The first guidance instruction in the disclosed embodiment can be understood as a pre-set prompt for guiding the referee model to evaluate the model performance of multiple large language models. The first guidance instruction includes the target evaluation criteria used for model evaluation. Optionally, for different application scenarios, the target evaluation criteria may also include domain-specific indicators, such as terminology accuracy in the corresponding field. The first guidance instruction may also include weight coefficients corresponding to each evaluation criterion in the target evaluation criterion, and the content of the model evaluation result that the referee model needs to output, wherein the model evaluation result includes at least information about the target large language model with the best model performance among the multiple large language models, such as identification information of the target large language model, and may also include evaluation reasons, scores of each large language model, etc., which are not limited here.

[0037] In the disclosed embodiment, the large language model evaluation device based on the referee model can, after obtaining the output results of each large language model for a test question, input the test question, the output results of each large language model and a preset first guidance instruction into a pre-established referee model. After receiving the input data, the referee model analyzes and compares the output results of each large language model based on its own language comprehension ability, determines which answer is better, generates an evaluation of each answer, and determines the information of the large language model corresponding to the best answer, and outputs it as the model evaluation result.

[0038] In an exemplary implementation of the disclosed embodiment, a large language model evaluation device based on a referee model can input the test questions and the output results of each large language model into the referee model according to the content spliced ​​by a preset structured template and a first guide instruction, so that the referee model can accurately compare and analyze the output results of each large language model based on the structured content.

[0039] The disclosed embodiment inputs pre-acquired test questions into multiple large language models to be evaluated respectively, and obtains the output results of each large language model, and inputs the test questions, the output results of each large language model and a preset first guiding instruction into a pre-established referee model to obtain a model evaluation result output by the referee model, wherein the first guiding instruction is used to guide the referee model to evaluate the model performance of the multiple large language models using target evaluation criteria, wherein the target evaluation criteria include accuracy, relevance, creativity, logical coherence and information integrity, and the model evaluation result includes information of a target large language model with the best model performance among the multiple large language models, and the output results of multiple large language models for the same question can be input into the referee model, guiding the referee model to evaluate the output results of each large language model from multiple angles such as accuracy, relevance, creativity, logical coherence and information integrity, thereby achieving a comprehensive evaluation of the model performance, and at the same time replacing the manual evaluation method, thereby reducing labor costs and improving evaluation efficiency.

[0040] Optionally, the model evaluation result also includes an evaluation basis text associated with the target large language model. After executing S102, the large language model evaluation device based on the referee model can input the evaluation basis text associated with the target large language model and a preset second guidance instruction into a pre-established integration model, and the integration model refines the evaluation basis text to obtain a set of key evaluation basis associated with the target large language model. The second guidance instruction is used to guide the integration model to extract key evaluation basis from the evaluation basis text.

[0041] Specifically, the first guiding instruction is also used to guide the referee model to output the information of the target large language model with the best model performance among multiple large language models, and output the evaluation basis text for the judgment result that the target large language model is the model with the best performance among multiple large language models. For example, the evaluation basis text can be: the answer of model A is more accurate because it clearly points out the key steps to solve the problem, while the answer content of model B is comprehensive but not directly related to the problem. The evaluation basis text is usually expressed in natural language form, containing rich contextual information and analysis details. In order to facilitate user viewing and subsequent further processing, the large language model evaluation device based on the referee model can use the evaluation basis text as the input of the integrated model, and input it into the integrated model together with the preset second guiding instruction. The integrated model extracts the evaluation basis text to obtain a set of key evaluation bases associated with the target large language model. Among them, the integrated model can be specifically another large language model, and the second guiding instruction can be understood as a prompt for guiding the integrated model to extract a preset number of key points within a preset length from the evaluation basis text to form a set of key evaluation bases associated with the target large language model.

[0042] Figure 2 is a flow chart of a method for integrating evaluation basis provided by an embodiment of the present disclosure, such as Figure 2 As shown, based on the above embodiments, the evaluation basis can be integrated through the following method.

[0043] S201. Obtain multiple groups of key evaluation bases corresponding to multiple test questions, and the correlation between each group of key evaluation bases and multiple large language models. The multiple groups of key evaluation bases are obtained by respectively refining multiple evaluation base texts corresponding to multiple test questions output by the referee model based on the second guidance instruction by the integrated model.

[0044] In the disclosed embodiment, the large language model evaluation device based on the referee model can perform multiple tests on multiple large language models based on multiple test questions. For the output results of each of the multiple large language models, the referee model is used to obtain the model evaluation results corresponding to the test questions, and the evaluation basis text in the model evaluation results is input into the integration model to obtain multiple groups of key evaluation basis corresponding to the multiple test questions. Based on the information of the target large language model in the model evaluation results, the correlation relationship between each group of key evaluation basis and the multiple large language models is determined.

[0045] S202: Determine the evaluation dimensions corresponding to each key evaluation basis in the multiple groups of key evaluation basis.

[0046] The evaluation dimensions in the embodiments of the present disclosure can be understood as multiple key directions for comprehensive consideration of model performance. For example, the evaluation dimensions may be the same as the target evaluation criteria, or may be different from the target evaluation criteria. The evaluation dimensions may also be manually preset, or may be extracted from the evaluation basis text or key evaluation basis, which is not limited here.

[0047] In the disclosed embodiment, the large language model evaluation device based on the referee model can determine the evaluation dimension corresponding to the key evaluation basis for each key evaluation basis in the multiple groups of key evaluation basis after obtaining multiple groups of key evaluation basis corresponding to multiple test questions.

[0048] In an exemplary implementation of the disclosed embodiment, a large language model evaluation device based on a referee model can search whether the key evaluation criteria contain the keywords associated with each evaluation dimension according to the keywords associated with each evaluation dimension obtained in advance, thereby determining the evaluation dimension corresponding to each key evaluation criterion.

[0049] Optionally, the large language model evaluation device based on the referee model can cluster multiple groups of key evaluation criteria to obtain a target number of evaluation dimensions before determining the evaluation dimensions corresponding to each of the multiple groups of key evaluation criteria.

[0050] Specifically, after obtaining multiple groups of key evaluation criteria corresponding to multiple test questions, the large language model evaluation device based on the referee model can cluster the multiple groups of key evaluation criteria into a target number of summary titles, and determine the obtained summary titles as evaluation dimensions.

[0051] Optionally, a large language model evaluation device based on a referee model can cluster multiple groups of key evaluation criteria to obtain a target number of evaluation dimensions, and then input the multiple groups of key evaluation criteria and a preset third guidance instruction into an integration model to obtain the target number of evaluation dimensions output by the integration model. The third guidance instruction is used to guide the integration model to cluster the key evaluation criteria.

[0052] Specifically, the large language model evaluation device based on the referee model can input multiple groups of key evaluation criteria and a preset third guiding instruction into the integration model when clustering multiple groups of key evaluation criteria, and the integration model clusters multiple key evaluation criteria contained in the multiple groups of key evaluation criteria to obtain a target number of summary titles, and determine the obtained summary titles as evaluation dimensions, wherein the third guiding instruction can be understood as a prompt for guiding the integration model to cluster multiple key evaluation criteria.

[0053] Among them, when there are N groups of key evaluation bases, each group of key evaluation bases contains m key evaluation bases, and the total length of N×m key evaluation bases exceeds the maximum input limit of the integrated model, the large language model evaluation device based on the referee model can determine the fixed number k of key evaluation bases input each time according to the maximum input limit of the integrated model, and divide the N×m key evaluation bases according to the fixed number k, and divide them into N*m / k batches in total, and input each batch of key evaluation bases together with the third guidance instruction into the integrated model to obtain the target number of summary titles, and then divide the summary titles corresponding to each batch of key evaluation bases again according to the fixed number k, and input them into the integrated model together with the third guidance instruction until the total number of output summary titles is less than or equal to the fixed number k, and the summary titles are no longer divided, and are directly input into the integrated model together with the third guidance instruction to obtain the target number of evaluation dimensions.

[0054] S203. Count the first quantity ratios of the key evaluation criteria corresponding to each evaluation dimension.

[0055] In the disclosed embodiment, a large language model evaluation device based on a referee model can, after determining the evaluation dimensions corresponding to each key evaluation basis, organize and count the key evaluation basis according to the evaluation dimensions, determine the first number of key evaluation basis corresponding to each evaluation dimension, and then determine the first quantity ratio of the key evaluation basis corresponding to each evaluation dimension.

[0056] S204. Count the second quantity ratios of key evaluation criteria associated with each large language model in each evaluation dimension.

[0057] In the disclosed embodiment, the large language model evaluation device based on the referee model can, after arranging the key evaluation basis according to the evaluation dimension, determine, for each evaluation dimension, the second number of key evaluation basis associated with each large language model among all the key evaluation basis corresponding to the evaluation dimension in combination with the association relationship between the key evaluation basis and the large language model, and then determine the ratio of the second data volume corresponding to each large language model to the number of all key evaluation basis under the evaluation dimension as the second quantity ratio of the key evaluation basis associated with each large language model in the evaluation dimension, wherein the second quantity ratio can reflect the situation in which each large language model has an advantage in each evaluation dimension.

[0058] S205. Based on the second quantity ratio, determine a third quantity ratio of key evaluation criteria associated with each large language model in all evaluation dimensions.

[0059] In the disclosed embodiment, the large language model evaluation device based on the referee model can, after determining the second quantity ratio of the key evaluation criteria associated with each large language model in each evaluation dimension, sort out the second quantity ratios of each large language model in each evaluation dimension, and calculate the average value of the second quantity ratios of each evaluation dimension for each large language model to obtain the third quantity ratio of the key evaluation criteria associated with each large language model in all evaluation dimensions, wherein the second quantity ratio can reflect the situation in which each large language model has an advantage in all evaluation dimensions.

[0060] The disclosed embodiment obtains multiple groups of key evaluation bases corresponding to multiple test questions, and the association relationship between each group of key evaluation bases and multiple large language models. The multiple groups of key evaluation bases are obtained by respectively refining multiple evaluation base texts corresponding to multiple test questions output by the referee model based on the second guiding instruction by the integration model, determining the evaluation dimensions corresponding to each key evaluation base in the multiple groups of key evaluation bases, counting the first quantity ratio of the key evaluation bases corresponding to each evaluation dimension, counting the second quantity ratio of the key evaluation bases associated with each large language model in each evaluation dimension, and determining the third quantity ratio of the key evaluation bases associated with each large language model in all evaluation dimensions based on the second quantity ratio. While providing evaluation results, a large number of scattered evaluation bases can be summarized into a small number of evaluation dimensions with commonality through cluster analysis technology, thereby enhancing the transparency and interpretability of the evaluation. At the same time, the relative importance or occurrence frequency of different evaluation dimensions in the overall evaluation is quantified based on the first quantity ratio, and the relative performance of each model in each evaluation dimension and all dimensions combined is quantified based on the second quantity ratio and the third quantity ratio, so that users can more scientifically and reasonably screen and optimize the models based on these data.

[0061] Figure 3is a flow chart of a method for determining an evaluation dimension corresponding to an evaluation basis provided by an embodiment of the present disclosure, such as Figure 3 As shown, based on the above embodiment, the evaluation dimension corresponding to the evaluation basis can be determined by the following method.

[0062] S301. For each key evaluation basis in the multiple groups of key evaluation basis, calculate the similarity between each key evaluation basis and each evaluation dimension.

[0063] In the embodiment of the present disclosure, the large language model evaluation device based on the referee model can convert each key evaluation basis and each evaluation dimension into an embedded vector based on a preset vector conversion model (such as BAAI General Embedding, BGE model) when determining the evaluation dimension corresponding to each key evaluation basis, and calculate the similarity between each key evaluation basis and each evaluation dimension, which can be specifically expressed as follows:

[0064]

[0065] Where Sim(r,h) represents the similarity between the key evaluation basis r and the evaluation dimension h, v r and v h They represent the embedding vectors corresponding to the key evaluation basis r and the evaluation dimension h respectively.

[0066] S302: Based on the similarity, determine the evaluation dimensions corresponding to each key evaluation basis.

[0067] In the disclosed embodiment, the large language model evaluation device based on the referee model can further determine the evaluation dimensions corresponding to each key evaluation basis after determining the similarity between each key evaluation basis and each evaluation dimension. Specifically, the similarity between the key evaluation basis and the evaluation dimension can be compared with a preset threshold. If the similarity is greater than or equal to the preset threshold, it is determined that the key evaluation basis corresponds to the evaluation dimension. If the similarity between the same key evaluation basis and multiple evaluation dimensions is greater than or equal to the preset threshold, the evaluation dimension with the highest similarity is determined as the evaluation dimension corresponding to the key evaluation basis.

[0068] The disclosed embodiment calculates the similarity between each key evaluation basis and each evaluation dimension for each key evaluation basis in multiple groups of key evaluation basis, and determines the evaluation dimensions corresponding to each key evaluation basis based on the similarity. This can improve the accuracy of determining the correspondence between the key evaluation basis and the evaluation dimensions, and further improve the accuracy of a series of statistical analysis results subsequently obtained based on the correspondence.

[0069] Optionally, the large language model evaluation device based on the referee model can determine the third quantitative ratio of the key evaluation criteria associated with each large language model in all evaluation dimensions based on the second quantitative ratio, and then visualize the first quantitative ratio of the key evaluation criteria corresponding to each evaluation dimension, the second quantitative ratio corresponding to each large language model in each evaluation dimension, and the third quantitative ratio corresponding to each large language model in all evaluation dimensions.

[0070] Specifically, the large language model evaluation device based on the referee model can use a bar chart or a pie chart to visualize the first quantity ratio, the second quantity ratio and the third quantity ratio, so as to intuitively reflect the performance differences of multiple large language models to be evaluated in different scenarios, and provide strong support for model optimization.

[0071] Figure 4 Schematic diagram of a large language model evaluation device based on a referee model provided by an embodiment of the present disclosure. Figure 4 As shown, the large language model evaluation device 400 based on the referee model includes: an input module 410 and an evaluation module 420, wherein the input module 410 is used to input the pre-acquired test questions into the multiple large language models to be evaluated respectively, and obtain the output results of each large language model; the evaluation module 420 is used to input the test questions, the output results of each large language model and a preset first guiding instruction into the pre-established referee model to obtain the model evaluation result output by the referee model, the first guiding instruction is used to guide the referee model to use the target evaluation standard to evaluate the model performance of the multiple large language models, the target evaluation standard includes accuracy, relevance, creativity, logical coherence and information integrity, and the model evaluation result includes the information of the target large language model with the best model performance among the multiple large language models.

[0072] Optionally, the model evaluation result also includes an evaluation basis text associated with the target large language model, and the large language model evaluation device 400 based on the referee model also includes: a refining module, used to input the evaluation basis text associated with the target large language model and a preset second guiding instruction into a pre-established integration model, and the integration model refines the evaluation basis text to obtain a set of key evaluation basis associated with the target large language model, and the second guiding instruction is used to guide the integration model to extract the key evaluation basis from the evaluation basis text.

[0073] Optionally, the large language model evaluation device 400 based on the referee model also includes: an acquisition module, used to obtain multiple groups of key evaluation criteria corresponding to multiple test questions, and the association relationship between each group of key evaluation criteria and the multiple large language models, the multiple groups of key evaluation criteria are obtained by the integration model based on the second guidance instruction to extract multiple evaluation criteria texts corresponding to the multiple test questions output by the referee model; a first determination module, used to determine the evaluation dimensions corresponding to each key evaluation basis in the multiple groups of key evaluation criteria; a first statistical module, used to count the first quantity ratio of the key evaluation basis corresponding to each evaluation dimension; a second statistical module, used to count the second quantity ratio of the key evaluation basis associated with each large language model in each evaluation dimension; a second determination module, used to determine the third quantity ratio of the key evaluation basis associated with each large language model in all evaluation dimensions based on the second quantity ratio.

[0074] Optionally, the first determination module includes: a calculation unit, used to calculate the similarity between each key evaluation basis in the multiple groups of key evaluation basis and each evaluation dimension; and a determination unit, used to determine the evaluation dimensions corresponding to each key evaluation basis based on the similarity.

[0075] Optionally, the large language model evaluation device 400 based on the referee model further includes: a clustering module, used for clustering the multiple groups of key evaluation criteria to obtain a target number of evaluation dimensions.

[0076] Optionally, the clustering module is specifically used to input the multiple groups of key evaluation criteria and a preset third guidance instruction into the integration model to obtain the target number of evaluation dimensions output by the integration model, and the third guidance instruction is used to guide the integration model to cluster the key evaluation criteria.

[0077] Optionally, the large language model evaluation device 400 based on the referee model also includes: a display module, which is used to visualize the first quantitative ratio of the key evaluation criteria corresponding to each evaluation dimension, the second quantitative ratio corresponding to each large language model in each evaluation dimension, and the third quantitative ratio corresponding to each large language model in all evaluation dimensions.

[0078] The large language model evaluation device based on the referee model provided in this embodiment can execute the method described in any of the above embodiments. Its execution method and beneficial effects are similar and will not be repeated here.

[0079] Figure 5 It is a structural diagram of a computer device provided in an embodiment of the present disclosure.

[0080] like Figure 5As shown, the computer device may include a processor 510 and a memory 520 storing computer program instructions.

[0081] Specifically, the processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0082] The memory 520 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 520 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 520 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 520 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 520 is a non-volatile solid-state memory. In a particular embodiment, the memory 520 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0083] The processor 510 reads and executes the computer program instructions stored in the memory 520 to perform the steps of the large language model evaluation method based on the referee model provided in the embodiment of the present disclosure.

[0084] In one example, the computer device may further include a transceiver 530 and a bus 540. Figure 5 As shown, the processor 510, the memory 520 and the transceiver 530 are connected via a bus 540 and communicate with each other.

[0085] The bus 540 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 540 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0086] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the large language model evaluation method based on the referee model provided by the embodiments of the present disclosure.

[0087] The above-mentioned storage medium may, for example, include a memory 520 of computer program instructions, and the above-mentioned instructions may be executed by the processor 510 of the large language model evaluation device based on the referee model to complete the large language model evaluation method based on the referee model provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc. The above-mentioned computer program may be written in any combination of one or more programming languages ​​to perform the program code of the operation of the embodiment of the present disclosure, and the programming language includes an object-oriented programming language, such as Java, C++, etc., and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device, partially on the remote computing device, or entirely on the remote computing device or server.

[0088] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0089] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large language model evaluation method based on a referee model, characterized in that: The method comprises: Input the pre-acquired test questions into the multiple large language models to be evaluated respectively, and obtain the output results of each large language model; The test question, the output results of each large language model and a preset first guiding instruction are input into a pre-established referee model to obtain a model evaluation result output by the referee model, wherein the first guiding instruction is used to guide the referee model to evaluate the model performance of the multiple large language models using target evaluation criteria, wherein the target evaluation criteria include accuracy, relevance, creativity, logical coherence and information completeness, and the model evaluation result includes information of a target large language model with the best model performance among the multiple large language models.

2. The method according to claim 1, characterized in that: The model evaluation result also includes an evaluation basis text associated with the target large language model. After inputting the test question, the output results of each large language model and the preset first guiding instruction into a pre-established referee model to obtain the model evaluation result output by the referee model, the method further includes: The evaluation basis text associated with the target large language model and a preset second guiding instruction are input into a pre-established integration model, and the integration model refines the evaluation basis text to obtain a set of key evaluation basis associated with the target large language model. The second guiding instruction is used to guide the integration model to extract the key evaluation basis from the evaluation basis text.

3. The method according to claim 2, characterized in that After the evaluation basis text associated with the target large language model and the preset second guiding instruction are input into a pre-established integration model, and the integration model is used to refine the evaluation basis text to obtain a set of key evaluation basis associated with the target large language model, the method further includes: Acquire multiple groups of key evaluation bases corresponding to multiple test questions, and the correlation between each group of key evaluation bases and the multiple large language models, wherein the multiple groups of key evaluation bases are obtained by respectively refining the multiple evaluation base texts corresponding to the multiple test questions output by the referee model based on the second guiding instruction by the integrated model; Determine the evaluation dimensions corresponding to each of the multiple groups of key evaluation criteria; Count the first quantity ratio of the key evaluation basis corresponding to each evaluation dimension; Counting the second quantity ratio of key evaluation basis associated with each large language model in each evaluation dimension; Based on the second quantity ratio, a third quantity ratio of key evaluation criteria associated with each large language model in all evaluation dimensions is determined.

4. The method according to claim 3, characterized in that Determining the evaluation dimensions corresponding to each of the multiple groups of key evaluation criteria includes: For each key evaluation basis in the multiple groups of key evaluation basis, calculating the similarity between each key evaluation basis and each evaluation dimension; Based on the similarity, evaluation dimensions corresponding to each of the key evaluation criteria are determined.

5. The method according to claim 3, characterized in that: Before determining the evaluation dimensions corresponding to each of the multiple groups of key evaluation criteria, the method further includes: The multiple groups of key evaluation criteria are clustered to obtain a target number of evaluation dimensions.

6. The method according to claim 5, characterized in that The clustering of the multiple groups of key evaluation criteria to obtain a target number of evaluation dimensions includes: The multiple groups of key evaluation criteria and a preset third guiding instruction are input into the integrated model to obtain the evaluation dimensions of the target number output by the integrated model. The third guiding instruction is used to guide the integrated model to perform clustering processing on the key evaluation criteria.

7. The method according to claim 3, characterized in that After determining the third quantity ratio of the key evaluation criteria associated with each large language model in all evaluation dimensions based on the second quantity ratio, the method further includes: The first quantity ratio of the key evaluation criteria corresponding to each evaluation dimension, the second quantity ratio corresponding to each large language model in each evaluation dimension, and the third quantity ratio corresponding to each large language model in all evaluation dimensions are visualized.

8. A large language model evaluation device based on a referee model, characterized in that: The device comprises: An input module, used to input the pre-acquired test questions into the multiple large language models to be evaluated, and obtain the output results of each large language model; An evaluation module is used to input the test question, the output results of each large language model and a preset first guiding instruction into a pre-established referee model to obtain a model evaluation result output by the referee model, wherein the first guiding instruction is used to guide the referee model to evaluate the model performance of the multiple large language models using a target evaluation standard, wherein the target evaluation standard includes accuracy, relevance, creativity, logical coherence and information completeness, and the model evaluation result includes information of a target large language model with the best model performance among the multiple large language models.

9. A computer device, characterized in that: include: Memory; processor; And a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the large language model evaluation method based on the referee model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the large language model evaluation method based on the referee model as described in any one of claims 1 to 7 is implemented.

Images