Assessment method and device of large language model system and related equipment

Abstract

The invention provides an evaluation method and device of a large language model system and related equipment, and relates to the technical field of computers. The method comprises the following steps: generating multiple groups of question and answer pairs, wherein each group of question and answer pair comprises an evaluation question and a standard answer; inputting the evaluation question of each question and answer pair into a to-be-evaluated large language model question and answer system, and obtaining a question answer output by the large language model question and answer system; and based on a pre-constructed quantitative evaluation model, according to the evaluation questions and the standard answers of the multiple groups of question and answer pairs, evaluating the questions and answers output by the large language model question and answer system. According to the method, the pre-constructed quantitative evaluation model is utilized, the answers output by the model are scored according to the standard answers, the problems of subjectivity and inconsistency in a traditional evaluation method are solved, the evaluation efficiency is improved, a large amount of data is automatically processed, and manpower and time cost is saved. The objectivity of evaluation is enhanced, personal errors are reduced, and the reliability of evaluation results is ensured.

Description

Background Art

[0002] With the rapid development of artificial intelligence technology, large language models have made significant progress in the field of natural language processing. These models are trained on large amounts of text data through deep learning algorithms and are able to understand and generate natural language text, thus playing an important role in various question-answering, dialogue systems, and information retrieval applications. However, as the model size continues to increase and the complexity increases, how to effectively evaluate the performance of these large language model systems has become a key issue.

[0003] In the current technical context, the methods for evaluating large language model systems mainly rely on manual evaluation and simple automated testing. Although manual evaluation can provide detailed feedback, it is time-consuming and highly subjective, making it difficult to apply on a large scale. Simple automated testing often cannot fully reflect the performance of the model in actual applications. Therefore, in this technical context, a more objective, systematic, and scalable evaluation method is needed to accurately measure the performance of large language model systems.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a method, apparatus and related equipment for evaluating a large language model system, which at least to a certain extent overcome the problem that related technologies cannot implement performance evaluation of a large language model system.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a method for evaluating a large language model system is provided, comprising: generating multiple groups of question-answer pairs, each group of question-answer pairs comprising: evaluation questions and standard answers; inputting the evaluation questions of each group of question-answer pairs into the large language model question-answering system to be evaluated, and obtaining answers to the questions output by the large language model question-answering system; and evaluating the answers to the questions output by the large language model question-answering system based on the evaluation questions and standard answers of the multiple groups of question-answer pairs based on a pre-constructed quantitative evaluation model.

[0008] In some embodiments, the large language model question answering system includes a retrieval enhancement generation (RAG) system.

[0009] In some embodiments, the evaluation questions and standard answers of the multiple sets of question-answer pairs are manually compiled and generated based on the knowledge base of the RAG system or are generated based on the knowledge base of the RAG system and a large language model, and the generated evaluation questions and standard answers of the multiple sets of question-answer pairs are manually verified.

[0010] In some embodiments, after inputting the evaluation questions of each set of question and answer pairs into the large language model question and answer system to be evaluated and obtaining the answers to the questions output by the large language model question and answer system, the method further includes: obtaining the context information retrieved when the RAG system outputs the answers to the questions; based on a pre-constructed quantitative evaluation model, evaluating the answers to the questions output by the large language model question and answer system according to the evaluation questions and standard answers of the multiple sets of question and answer pairs and the context information.

[0011] In some embodiments, based on a pre-constructed quantitative evaluation model, the answers to questions output by the large language model question and answer system are evaluated according to the evaluation questions and standard answers of the multiple groups of question and answer pairs, including: obtaining at least one pre-set evaluation indicator; obtaining a corresponding prompt word template according to the evaluation indicator, wherein the prompt word template is used to generate a prompt word; inputting the prompt word into a pre-trained quantitative evaluation model, and outputting an evaluation value calculated for the evaluation indicator; and merging the evaluation values ​​of each evaluation indicator to obtain a comprehensive evaluation result of the answer to the question output by the large language model question and answer system.

[0012] According to another aspect of the present disclosure, there is also provided an evaluation device for a large language model system, including: a question-answer pair generation module, used to generate multiple groups of question-answer pairs, each group of question-answer pairs including: evaluation questions and standard answers; a question-answer output module, used to input the evaluation questions of each group of question-answer pairs into the large language model question-answering system to be evaluated, and obtain the answers to the questions output by the large language model question-answering system; a question-answer evaluation module, used to evaluate the answers to the questions output by the large language model question-answering system based on a pre-constructed quantitative evaluation model and according to the evaluation questions and standard answers of the multiple groups of question-answer pairs.

[0013] According to another aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-described large language model system evaluation methods by executing the executable instructions.

[0014] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the evaluation method of the large language model system described in any one of the above is implemented.

[0015] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, which implements any of the above-mentioned large language model system evaluation methods when executed by a processor.

[0016] The evaluation method, device and related equipment of the large language model system provided in the embodiments of the present disclosure solve the problem of performance evaluation of the large language model question-answering system by generating multiple sets of question-answer pairs and using a pre-built quantitative evaluation model, and realize an objective and quantitative evaluation of the quality of the model output. By generating multiple sets of question-answer pairs, the comprehensiveness and representativeness of the evaluation questions are ensured, so that the evaluation results can truly reflect the performance of the large language model system in practical applications. Using the pre-built quantitative evaluation model, the answers output by the model are scored according to the standard answers, which solves the problems of subjectivity and inconsistency in traditional evaluation methods. The quantitative evaluation model can provide specific performance indicators, such as fidelity, answer accuracy, etc., making the evaluation results more intuitive and convincing. The efficiency of the evaluation is improved, a large amount of data is automatically processed, and manpower and time costs are saved. The objectivity of the evaluation is enhanced, human errors are reduced, and the reliability of the evaluation results is ensured. The continuous optimization of the large language model system is promoted, and the R&D team is guided to improve the model in a targeted manner by identifying performance shortcomings.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 A schematic diagram showing an exemplary application system architecture of an evaluation method using a large language model system in an embodiment of the present disclosure is shown;

[0020] Figure 2 A flow chart of an evaluation method for a large language model system in an embodiment of the present disclosure is shown;

[0021] Figure 3 A flow chart of another method for evaluating a large language model system in an embodiment of the present disclosure is shown;

[0022] Figure 4 A flow chart of another method for evaluating a large language model system in an embodiment of the present disclosure is shown;

[0023] Figure 5 A detailed schematic diagram of a RAG evaluation device is shown;

[0024] Figure 6 A schematic diagram of a specific RAG system evaluation method flow in an embodiment of the present disclosure is shown;

[0025] Figure 7 A schematic diagram of a large language model system according to an embodiment of the present disclosure is shown;

[0026] Figure 8 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0029] Figure 1 FIG. 2 shows an exemplary application system architecture diagram to which the evaluation method of the large language model system in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0030] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0031] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0032] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0033] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0034] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0035] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0036] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0037] Under the above system architecture, an evaluation method for a large language model system is provided in an embodiment of the present disclosure. The method can be executed by any electronic device with computing and processing capabilities.

[0038] In some embodiments, the evaluation method of the large language model system provided in the embodiments of the present disclosure can be executed by the terminal device of the above-mentioned system architecture; in other embodiments, the evaluation method of the large language model system provided in the embodiments of the present disclosure can be executed by the server in the above-mentioned system architecture; in other embodiments, the evaluation method of the large language model system provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0039] Figure 2 A flow chart of an evaluation method for a large language model system in an embodiment of the present disclosure is shown. Figure 2 As shown, the evaluation method of the large language model system provided in the embodiment of the present disclosure includes the following steps:

[0040] S202, generating multiple sets of question-answer pairs, each set of question-answer pairs including: an evaluation question and a standard answer.

[0041] It should be noted that the large language model system to be evaluated in the embodiments of the present disclosure can be any large language model system that needs to evaluate the performance of question-answering effects. Specifically, the large language model system in the embodiments of the present disclosure can be a RAG (Retrieval-Augmented Generation) system. The RAG system is a technology that combines a retrieval model and a generation model. RAG technology is mainly used to assist text generation by using information from private or proprietary knowledge bases to enhance the relevance and accuracy of generated content. Compared with other large language models, the RAG system can provide external information sources to help generate more accurate and more contextual answers. The RAG system is particularly suitable for tasks such as question-answering systems and content generation. It improves the relevance of the search experience by obtaining background information from more data sources and supplementing the original knowledge base of the large language model through training.

[0042] When evaluating a large language model system, in the disclosed embodiment, it is first necessary to generate multiple sets of question-answer pairs, where each set of question-answer pairs consists of an evaluation question and a corresponding standard answer. The evaluation question can be any question used to test the effectiveness of the answers output by the large language model during the evaluation process. The standard answer is the correct answer or expected answer to each evaluation question. The standard answer is used as a benchmark for evaluating the accuracy of the answers output by the large language model. By generating multiple sets of question-answer pairs, a more comprehensive evaluation of the effectiveness of the answers output by the large language model system can be ensured.

[0043] S204: Input the evaluation questions of each question-answer pair into the large language model question-answering system to be evaluated, and obtain the answers to the questions output by the large language model question-answering system.

[0044] It should be noted that the evaluation questions in the multiple sets of question-answer pairs generated previously are input as input data to the large language model question-answering system to be evaluated. The large language model question-answering system will process the evaluation question according to its internal algorithm and training data, and finally generate the answer to the evaluation question as output data. For the RAG system, the evaluation question of each set of question-answer pairs is input into the RAG system to be evaluated, and the RAG system will output the answer to the evaluation question, that is, the RAG answer.

[0045] S206, based on a pre-built quantitative evaluation model, evaluate the answers to questions output by the large language model question and answer system according to the evaluation questions and standard answers of multiple groups of question and answer pairs.

[0046] It should be noted that the quantitative evaluation model can be any pre-built model for quantitative evaluation of a large language model system. The quantitative evaluation model includes a series of evaluation indicators and algorithms for scoring the output of the model.

[0047] The evaluation indicators may include accuracy, recall, precision scores, etc., which are used to measure the quality of the answers to questions output by the large language model system from different perspectives. For each question-answer pair, there is a correct or optimal answer, the standard answer, which will be compared with the answer to the question output by the large language model system to determine the accuracy or relevance of the answer to the question output by the system.

[0048] The answers output by the large language model question-answering system are input into the quantitative evaluation model together with the corresponding standard answers. The quantitative evaluation model will score each answer according to the predetermined evaluation indicators. For example, the quantitative evaluation model can calculate the text similarity between the output question answer and the standard answer, or evaluate the grammatical correctness and information completeness of the output question answer. Through quantitative evaluation, a series of performance indicator values ​​can be obtained, which can intuitively show the performance of the large language model system when processing input questions. By analyzing these performance indicators, the overall performance of the large language model system can be comprehensively evaluated, and the advantages and disadvantages of the large language model system can be identified.

[0049] The disclosed embodiment generates multiple sets of question-answer pairs including evaluation questions and corresponding standard answers, uses the standard answers as evaluation benchmarks, and inputs the evaluation questions into a pre-built quantitative evaluation model to complete the evaluation of the large language model system to be evaluated. This can objectively evaluate the performance of the large language model system, facilitate the development, optimization and application of the large language model system, and also provide users and developers with reliable information about the performance of the large language model system.

[0050] In some embodiments of the present disclosure, the evaluation questions and standard answers for the multiple sets of question-answer pairs are manually compiled and generated based on the knowledge base of the RAG system or are generated based on the knowledge base of the RAG system and a large language model, and the generated evaluation questions and standard answers for the multiple sets of question-answer pairs are manually verified.

[0051] It should be noted that the question-answer pair in the question-answer pair can be generated by two methods. In some embodiments of the present disclosure, the evaluation questions and the corresponding standard answers in the question-answer pair are written by domain experts or the evaluation team based on the knowledge base content of the RAG system, ensuring that the evaluation questions are directly related to the knowledge base of the RAG system and ensuring the consistency of the questions with the RAG system capabilities. Since the evaluation questions are compiled based on the knowledge base of the RAG system, they can test the ability of the RAG system to retrieve and utilize the knowledge base.

[0052] In other embodiments of the present disclosure, in addition to using the knowledge base of the RAG system, the generation of question-answer pairs can also be combined with a large language model. The large language model can provide richer language expressions and question diversity. The large language model automatically generates evaluation questions and corresponding standard answers based on the knowledge base content of the RAG system. In this way, question-answer pairs can be generated quickly on a large scale, improving the efficiency of evaluation.

[0053] Whether it is manually compiled based on the RAG system knowledge base or automatically generated based on the RAG system knowledge base combined with a large language model, the evaluation questions and standard answers of the generated multiple sets of question-answer pairs need to be manually verified. The manual verification step ensures the quality of the evaluation questions and standard answers, thereby improving the reliability and effectiveness of the evaluation results.

[0054] By generating question-answer pairs through these two methods, the disclosed embodiments can more accurately evaluate the performance of the large language model system when dealing with questions related to the RAG system knowledge base. The question-answer pairs generated in this way can more comprehensively and accurately evaluate the effectiveness of the RAG system in retrieval and generation, especially in evaluating whether the RAG system can effectively retrieve relevant information in the knowledge base, evaluating whether the answers generated by the RAG system accurately reflect the information in the knowledge base, and evaluating the performance of the RAG system when dealing with questions of different types and difficulty. The disclosed embodiments help ensure the reliability and effectiveness of the RAG system in practical applications.

[0055] In some embodiments of the present disclosure, Figure 3 As shown, after inputting the evaluation questions of each set of question-answer pairs into the large language model question-answering system to be evaluated and obtaining the answers to the questions output by the large language model question-answering system, the following steps are also included:

[0056] S302, obtaining context information retrieved when the RAG system outputs the answer to the question;

[0057] S304, based on the pre-built quantitative evaluation model, evaluate the answers to the questions output by the large language model question and answer system according to the evaluation questions and standard answers of multiple groups of question and answer pairs and context information.

[0058] It should be noted that in the RAG system, when generating answers to input evaluation questions, the RAG system will retrieve relevant documents or information in the knowledge base to enhance the relevance and accuracy of the generated content. The evaluation method of the disclosed embodiment requires obtaining the contextual information of these retrievals, that is, the relevant information used by the RAG system when generating answers.

[0059] By combining the evaluation questions and standard answers with the retrieval context information of the RAG system, the quantitative evaluation model can more comprehensively evaluate the performance of the RAG system. This comprehensive evaluation method takes into account the relevant information retrieved by the RAG system when generating the answer to the question, which helps to more accurately evaluate the performance of the large language model system in practical applications. Obtaining the retrieval context information when the RAG system outputs the answer to the question makes the evaluation more accurate and relevant because this information reflects the knowledge base content that the RAG system relies on when generating the answer.

[0060] In some embodiments of the present disclosure, the quantitative evaluation model includes at least one predefined evaluation indicator and a calculation formula corresponding to each evaluation indicator, and the evaluation indicator includes one or more of the following: context fidelity, answer accuracy, answer relevance, context relevance, and context recall.

[0061] It should be noted that the evaluation indicators are pre-set evaluation indicators used to measure and evaluate the effectiveness of the RAG system. The evaluation indicators in the disclosed embodiment include one or more of the following: context fidelity, answer accuracy, answer relevance, context relevance, and context recall. The following is a detailed description of these evaluation indicators:

[0062] Context fidelity, answer accuracy, answer relevance, context relevance, and context recall are several key evaluation indicators used to evaluate the performance of large language model systems. Their importance in evaluating the large language model system answering questions is as follows:

[0063] Contextual fidelity is used to evaluate the fidelity between the answer to the question output by the large language model system and the retrieved context information. It measures whether the model can accurately reflect the retrieved context information and ensure the consistency of the answer with the question background.

[0064] Answer accuracy is used to evaluate the similarity or consistency between the answer to the question output by the large language model system and the standard answer. It focuses on the accuracy of the answer, that is, whether the answer correctly answers the question.

[0065] Answer relevance is used to evaluate the relevance between the answer to the question output by the large language model system and the evaluation question. It measures whether the answer is directly related to the question. Even if the answer is correct, if it is not relevant to the question, then the answer is not ideal.

[0066] Context relevance is used to evaluate the relevance between the answer to the question output by the large language model system and the context information retrieved by the RAG system. It focuses on whether the answer is relevant to the retrieved context information. Even if the answer is accurate and relevant, if it is not relevant to the context information, then the answer may not be optimal.

[0067] The context recall rate is used to evaluate the degree of recall between the retrieval context information contained in the question answer output by the large language model system and the actual retrieved context information. It measures whether the model can recall all the context information related to the question to ensure the comprehensiveness of the answer.

[0068] The combined use of these evaluation indicators can help evaluate whether the large language model system is faithful to the context information when answering questions, the accuracy and relevance of the answer, the relevance of the context information, and the recall of the context information. Through these indicators, the performance of the large language model system can be more comprehensively evaluated and guide the further optimization and adjustment of the model.

[0069] The calculation formula is a mathematical expression used to calculate the specific value of the evaluation indicator. When evaluating the performance of a large language model system, the calculation formula will be used to convert the collected data into specific values, which can be used to quantify the performance of the model.

[0070] Contextual fidelity can be evaluated by calculating the degree of match between the information mentioned in the answer to the question output by the large language model system and the retrieved contextual information. The calculation formula may involve using a text matching algorithm, such as the Jaccard coefficient or the vector space model, to calculate the similarity between the answer and the contextual information.

[0071] Answer accuracy can be evaluated by calculating the similarity between the answer to the question output by the large language model system and the standard answer. The calculation formula may involve using methods such as cosine similarity, edit distance, or BLEU score to compare the answer to the standard answer.

[0072] Answer relevance can be evaluated by calculating the correlation between the answer to the question output by the large language model system and the evaluation question. The calculation formula may involve using a relevance scoring method in information retrieval, such as BM25 or TF-IDF, to evaluate the relevance between the answer and the question.

[0073] Context relevance can be evaluated by calculating the relevance between the answer to the question output by the large language model system and the context information retrieved by the RAG system. The calculation formula may involve using a text matching algorithm to calculate the similarity between the answer and the context information.

[0074] The context recall rate can be evaluated by calculating the recall degree between the retrieval context information contained in the answer to the question output by the large language model system and the actual retrieved context information. The calculation formula may involve calculating the recall rate between the information mentioned in the answer and the actual retrieved context information.

[0075] These calculation formulas are the mathematical basis for evaluating the performance of large language model systems. They help convert complex data and information into quantifiable and comparable values, thereby providing a quantitative assessment of the model's performance.

[0076] In some embodiments of the present disclosure, Figure 4 As shown, based on the pre-built quantitative evaluation model, the answer to the question output by the large language model question answering system is evaluated, and the following steps are also included:

[0077] S402, obtaining at least one pre-set evaluation indicator;

[0078] S404, obtaining a corresponding prompt word template according to the evaluation index, wherein the prompt word template is used to generate a prompt word;

[0079] S406, inputting the prompt word into a pre-trained quantitative evaluation model, and outputting an evaluation value calculated for the evaluation index;

[0080] S408, combining the evaluation values ​​of the various evaluation indicators to obtain a comprehensive evaluation result of the answer to the question output by the large language model question answering system.

[0081] It should be noted that at least one pre-set evaluation indicator is selected, and the evaluation indicator is a standard for measuring the quality of the answer to the question output by the large language model system. According to the selected evaluation indicator, the corresponding prompt word template is obtained. The prompt word template is a framework or structure for generating prompt words, which can help the evaluation model calculate the evaluation indicator more accurately. Generate prompt words using the prompt word template. The prompt words will be used in the further evaluation process, and the prompt words can help the quantitative evaluation model better understand the answer output by the model and calculate the evaluation indicator accordingly.

[0082] The generated prompt words are input into a pre-trained quantitative evaluation model. The quantitative evaluation model is specially designed to calculate evaluation indicators, and the quantitative evaluation model can calculate evaluation values ​​based on the prompt words. The quantitative evaluation model outputs the evaluation values ​​calculated for the evaluation indicators based on the prompt words. The evaluation value is a specific numerical value used to measure the quality of the answers to questions output by the large language model system. The evaluation values ​​of each evaluation indicator are combined to obtain a comprehensive evaluation result of the answers to the questions output by the large language model question and answer system. This comprehensive evaluation result is an overall evaluation of the answers to the questions output by the model, which takes into account the combined impact of multiple evaluation indicators. The disclosed embodiment can more comprehensively evaluate the performance of the large language model system in answering questions.

[0083] In some embodiments of the present disclosure, the RAG system is taken as an example of a large language model system. Figure 5-Figure 6 As shown, how to evaluate a large language model system in the embodiments of the present disclosure is explained in detail.

[0084] Figure 5 A detailed schematic diagram of a RAG evaluation device is shown, which uses a data-centric architecture to organize various components, uses a database to store and manage data, and various components work together based on shared data, such as Figure 5 As shown, the schematic diagram of the RAG evaluation device includes modules such as RAG system, question generation, manual assistance, shared data source, and RAG evaluation system.

[0085] Among them, the RAG system module is a question-answering system built based on the retrieval enhancement technology of the large language model. It is also the object we want to evaluate. It is responsible for giving answers based on the evaluation questions and saving the retrieved RAG context content.

[0086] The question generation module is responsible for the generation of evaluation questions. It usually obtains relevant knowledge bases and generates evaluation questions by calling large language models that are recognized in the industry as having strong language understanding and generation capabilities, such as GPT-4, a large language model released by OpenAI, which is the fourth generation of its GPT series models.

[0087] Human assistance module: introduces human assistance, usually industry experts, who assist in the selection of assessment questions, review of standard answers, etc.

[0088] Shared data source module: a shared database system where all components can access and update data for interaction between systems. The main data for interaction include assessment questions, RAG answers, standard answers, RAG context, etc.

[0089] Evaluation system module: It is the core module of the embodiment of the present disclosure, and is composed of four sub-modules: LLM (Large Language Model) reasoning service, prompt word engineering, evaluation index system, and Agent.

[0090] The LLM reasoning service module is the large language model reasoning service, which is used to evaluate the answers to questions given by the RAG system and give an evaluation score for the answers provided by the RAG system based on a predefined evaluation index system.

[0091] Evaluation index system module: It is a quantitative measurement target used to measure the effectiveness of the RAG system’s responses. This solution includes the following indicators and supports the expansion of indicators.

[0092] Fidelity: Measures the factual consistency of the generated answer relative to the given context.

[0093] Answer accuracy: Evaluates the accuracy of the generated answers compared to the standard answers.

[0094] Answer relevance: Evaluates how relevant the generated answer is to the given prompt.

[0095] Contextual relevance: How relevant is the retrieval context to answering a given question.

[0096] Context recall: measures the consistency between the retrieved context and the ground truth answer.

[0097] Prompt word engineering module: Generates corresponding prompt words based on predefined evaluation indicators for calculation of the indicator by the large language model.

[0098] Agent module: It is a concatenation of the entire evaluation process, including obtaining evaluation questions, RAG answers, standard answers, RAG context and other data, obtaining indicators that need to be evaluated, generating corresponding prompt words, and sending them to the large language model for calculation to obtain calculation results.

[0099] Figure 6 The following is a flow chart of a specific RAG system evaluation method in an embodiment of the present disclosure. The method specifically includes the following steps:

[0100] S602, question generation: This step is used to test the generation of evaluation questions and standard answers of the RAG system.

[0101] S6021, knowledge base selection: Randomly and abstractly select the knowledge base content of the RAG system to be used as the context fragment for question generation.

[0102] S6022, question generation: There are two ways. One is to manually collect or edit relevant questions and standard answers based on the knowledge base content selected in step S6021 through the Internet or other channels; the other is to select a large language model with strong language understanding and generation capabilities recognized by the industry, and generate a set of questions and answers based on the knowledge base context fragment of step S6021.

[0103] S6023, manual verification: Business experts in related fields select, review, and rewrite the generated question and answer pairs to form accurate and usable question and answer pairs.

[0104] S604, the RAG system answers the question.

[0105] S6041, problem acquisition: the Agent module accesses the shared data source to acquire the problem generated in step S602.

[0106] S6042, RAG system question and answer: Agent calls the RAG system interface, requests the RAG system to obtain answers for the question group output in step S602, and saves the answers and the context information retrieved.

[0107] S606, answer quality assessment: evaluate and score the questions answered by the RAG system.

[0108] S6061, Build an evaluation index library: Based on industry standards and personalized evaluation requirements, define evaluation indicators and their calculation formulas, such as context fidelity, answer accuracy, answer relevance, context relevance, context recall, etc.

[0109] S6062, define prompt word template library: define the corresponding large model prompt word template library based on the evaluation index system, and bind the calculation of each index to a fixed template.

[0110] S6063, Select evaluation indicators: Based on the actual needs of the scenario, select the evaluation indicators that are most important to the scenario as the test objects.

[0111] S6064, Evaluation: The Agent reads relevant data from the shared data source, obtains the prompt word template according to each calculation indicator, generates prompt words, sends them to the evaluation model, and obtains the calculation scores of each indicator for each question.

[0112] S6065, evaluation report output: output the calculation evaluation results, assemble them into an evaluation report, and output them to the user.

[0113] The data-centric large model RAG evaluation system architecture proposed in the disclosed embodiment defines the core data structure of the shared data source. Each component exchanges information based on the shared data source, and drives the evaluation of the large model RAG system based on the shared data source. It can flexibly and efficiently complete the implementation of the system, and has good scalability. The large language model RAG effect evaluation method effectively completes the evaluation of its effect through the generation of problems, the establishment of an indicator system, and the implementation of the evaluation method, and has replicability, convenience and effectiveness.

[0114] Based on the same inventive concept, the present disclosure also provides a device for a large language model system, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0115] Figure 7 A schematic diagram of a large language model system in an embodiment of the present disclosure is shown. Figure 7 As shown, the device comprises:

[0116] A question-answer pair generating module 701 is used to generate multiple sets of question-answer pairs, each set of question-answer pairs including: an evaluation question and a standard answer;

[0117] A question answer output module 702 is used to input the evaluation question of each set of question-answer pairs into the large language model question-answering system to be evaluated, and obtain the answer to the question output by the large language model question-answering system;

[0118] The question answer evaluation module 703 is used to evaluate the question answers output by the large language model question answering system based on a pre-built quantitative evaluation model and according to the evaluation questions and standard answers of the multiple groups of question and answer pairs.

[0119] In some embodiments of the present disclosure, the above-mentioned question and answer output module 702 is also used to obtain context information retrieved when the RAG system outputs question answers; based on a pre-constructed quantitative evaluation model, the question answers output by the large language model question and answer system are evaluated according to the evaluation questions and standard answers of the multiple groups of question and answer pairs and the context information.

[0120] In some embodiments of the present disclosure, the question answer evaluation module 703 is further used to obtain at least one pre-set evaluation indicator; obtain a corresponding prompt word template according to the evaluation indicator, and the prompt word template is used to generate a prompt word; input the prompt word into a pre-trained quantitative evaluation model, and output an evaluation value calculated for the evaluation indicator; and combine the evaluation values ​​of each evaluation indicator to obtain a comprehensive evaluation result of the question answer output by the large language model question and answer system.

[0121] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiments. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0122] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0123] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0124] like Figure 8As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0125] Among them, the storage unit stores a program code, and the program code can be executed by the processing unit 810, so that the processing unit 810 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 810 can execute the following steps of the above method embodiment: generate multiple groups of question and answer pairs, each group of question and answer pairs includes: evaluation questions and standard answers; input the evaluation questions of each group of question and answer pairs into the large language model question and answer system to be evaluated, and obtain the answers to the questions output by the large language model question and answer system; based on the pre-constructed quantitative evaluation model, evaluate the answers to the questions output by the large language model question and answer system according to the evaluation questions and standard answers of the multiple groups of question and answer pairs.

[0126] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0127] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0128] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0129] The electronic device 800 may also communicate with one or more external devices 840 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0130] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0131] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the method in the above embodiment when executed by a processor.

[0132] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0133] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0135] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0136] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0137] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0138] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0139] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0140] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for evaluating a large language model system, characterized in that: include: Generate multiple sets of question-answer pairs, each set of question-answer pairs includes: an evaluation question and a standard answer; Inputting the evaluation questions of each set of question-answer pairs into the large language model question-answering system to be evaluated, and obtaining the answers to the questions output by the large language model question-answering system; Based on a pre-built quantitative evaluation model, the answers to the questions output by the large language model question and answer system are evaluated according to the evaluation questions and standard answers of the multiple groups of question and answer pairs.

2. The evaluation method of a large language model system according to claim 1, characterized in that: The large language model question answering system includes a retrieval enhancement generation RAG system.

3. The evaluation method of a large language model system according to claim 2, characterized in that: The evaluation questions and standard answers for the multiple sets of question-answer pairs are manually compiled and generated based on the knowledge base of the RAG system or are generated based on the knowledge base of the RAG system and a large language model. The generated evaluation questions and standard answers for the multiple sets of question-answer pairs are manually verified.

4. The evaluation method of a large language model system according to claim 2, characterized in that: After inputting the evaluation questions of each set of question-answer pairs into the large language model question-answering system to be evaluated and obtaining the answers to the questions output by the large language model question-answering system, the method further includes: Obtaining context information retrieved when the RAG system outputs an answer to a question; Based on a pre-built quantitative evaluation model, the answers to the questions output by the large language model question and answer system are evaluated according to the evaluation questions and standard answers of the multiple groups of question and answer pairs and the context information.

5. The evaluation method of a large language model system according to claim 4, characterized in that: The quantitative evaluation model includes at least one predefined evaluation indicator and a calculation formula corresponding to each evaluation indicator, and the evaluation indicator includes one or more of the following: context fidelity, answer accuracy, answer relevance, context relevance, and context recall.

6. The evaluation method of a large language model system according to claim 5, characterized in that: Based on the pre-built quantitative evaluation model, the answers to the questions output by the large language model question answering system are evaluated according to the evaluation questions and standard answers of the multiple groups of question answering pairs, including: obtaining at least one pre-set evaluation indicator; Acquire a corresponding prompt word template according to the evaluation index, wherein the prompt word template is used to generate a prompt word; Inputting the prompt word into a pre-trained quantitative evaluation model, and outputting an evaluation value calculated for the evaluation index; The evaluation values ​​of each evaluation indicator are combined to obtain a comprehensive evaluation result of the answer to the question output by the large language model question answering system.

7. An evaluation device for a large language model system, characterized in that: include: A question-answer pair generation module is used to generate multiple sets of question-answer pairs, each set of question-answer pairs includes: an evaluation question and a standard answer; A question answer output module, used to input the evaluation question of each set of question-answer pairs into the large language model question-answering system to be evaluated, and obtain the answer to the question output by the large language model question-answering system; A question answer evaluation module is used to evaluate the answers to questions output by the large language model question answering system based on a pre-built quantitative evaluation model and according to the evaluation questions and standard answers of the multiple groups of question and answer pairs.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the large language model system evaluation method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the evaluation method of the large language model system according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, the evaluation method of the large language model system described in any one of claims 1 to 6 is implemented.

Images