RAG system performance evaluation method and device, computer equipment and storage medium

Abstract

The invention discloses an RAG system performance evaluation method and device, computer equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: respectively carrying out the fine adjustment of three preset LLMs based on an answer authenticity evaluation index, an answer correlation evaluation index and a context correlation evaluation index, and obtaining three LLMs; corresponding answer authenticity fine tuning LLM, answer correlation fine tuning LLM and context correlation fine tuning LLM are obtained; based on the answer authenticity fine tuning LLM, the answer correlation fine tuning LLM and the context correlation fine tuning LLM, performance evaluation is carried out on the target RAG system, and a corresponding answer authenticity score, a corresponding answer correlation score and a corresponding context correlation score are obtained; and generating a performance evaluation result of the target RAG system based on the answer authenticity score, the answer correlation score and the context correlation score. Through the method provided by the invention, the performance of the target RAG system can be quickly, efficiently and accurately evaluated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a RAG system performance evaluation method, device, computer equipment and non-volatile computer-readable storage medium. Background Art

[0002] Currently, in the field of artificial intelligence (AI), with the rise of large language models (LLM), some researchers have found that although LLM captures enough knowledge to be comparable to human performance in various question-answering benchmarks, LLM cannot answer questions about events that occurred after the model was trained. The solution to this problem is to rely on retrieval-augmented generation (RAG), so RAG is one of the most popular large model cutting-edge technologies at present.

[0003] RAG has also been widely used in the insurance field, such as the A-end claims assistant, group underwriting robot, and easy-to-read business analysis assistant. The usefulness of RAG is obvious, but their implementation requires a lot of adjustments. This is because the overall performance of the RAG system will be affected by the retrieval model, corpus, LLM or prompt expression, etc. In order to ensure that the RAG system can run efficiently and provide high-quality performance, developers must accurately adjust and optimize it. Therefore, automatic performance evaluation of the RAG system is crucial.

[0004] At present, there are two main methods for evaluating the performance of RAG systems: 1. Using public data sets in nlp tasks for evaluation, such as machine reading comprehension question and answer, automatic summarization, entity extraction, etc. On the one hand, the data of these nlp tasks are significantly different from the data required by RAG, and there will be significant adaptability issues. Therefore, professional labelers are required to perform secondary processing on them. This process is undoubtedly time-consuming and labor-intensive. On the other hand, the knowledge in the insurance field is highly professional, and the differences between different sub-fields are significant. For example, there are essential differences in the knowledge points and business logic involved in the claims processing and underwriting processes. This means that these public data cannot effectively evaluate the performance of insurance RAGs; 2. Writing some simple prompts and using the reasoning ability of LLM to evaluate RAGs, such as RAGAS (Retrieval-Augmented Generation Systems), but this method has almost no adaptability to RAGs in vertical fields, so this method cannot effectively evaluate insurance RAGs.

[0005] Based on this, how to provide a RAG system performance evaluation method, device, computer equipment and non-volatile computer-readable storage medium that can achieve fast, efficient and accurate evaluation of the performance of the target RAG system is a problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide a RAG system performance evaluation method, apparatus, computer equipment and non-volatile computer-readable storage medium that can be used in financial technology or other related fields, which can realize fast, efficient and accurate evaluation of the performance of the target RAG system.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A RAG system performance evaluation method, comprising:

[0009] Based on the answer authenticity evaluation index, answer relevance evaluation index and context relevance evaluation index, the three preset LLMs are fine-tuned respectively to obtain the corresponding answer authenticity fine-tuned LLM, answer relevance fine-tuned LLM and context relevance fine-tuned LLM;

[0010] Based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, the target RAG system is evaluated for performance to obtain corresponding answer authenticity scores, answer relevance scores and context relevance scores;

[0011] Based on the answer authenticity score, the answer relevance score and the context relevance score, a performance evaluation result of the target RAG system is generated.

[0012] In a further technical solution, the RAG system performance evaluation method, wherein the three preset LLMs are fine-tuned based on the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index to obtain the corresponding answer authenticity fine-tuned LLM, answer relevance fine-tuned LLM and context relevance fine-tuned LLM, includes:

[0013] Organize and obtain knowledge in the target field;

[0014] Based on the target domain knowledge, according to the requirements of the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, the corresponding answer authenticity training data, the answer relevance training data and the context relevance training data are sorted out;

[0015] Based on the answer authenticity training data, the answer relevance training data and the context relevance training data, the LoRA algorithm is used to fine-tune the three preset LLMs respectively to obtain the corresponding answer authenticity fine-tuning LLM, answer relevance fine-tuning LLM and context relevance fine-tuning LLM.

[0016] In a further technical solution, the RAG system performance evaluation method, wherein, based on the target domain knowledge, according to the requirements of the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, the corresponding answer authenticity training data, the answer relevance training data and the context relevance training data are sorted out, wherein,

[0017] The answer authenticity training data includes answer text, retrieval context text, and score labels of the correlation between the two;

[0018] The answer relevance training data includes answer text, question text, and score labels of the relevance between the two;

[0019] The context relevance training data includes context text, question text, and score labels of the relevance between the context text and question text.

[0020] In a further technical solution, the RAG system performance evaluation method, wherein the performance of the target RAG system is evaluated based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, to obtain the corresponding answer authenticity score, answer relevance score and context relevance score, wherein,

[0021] Based on the answer authenticity fine-tuning LLM, the performance of the target RAG system is evaluated to obtain the corresponding answer authenticity score, including:

[0022] Design the first prompt based on the given first target answer text;

[0023] Input the first prompt and the first target answer text into the preset LLM to generate an answer short sentence;

[0024] The answer phrase and the target retrieval context text are converted into vector form using a preset text conversion model, and a similarity score r1 between the vectors of the two is calculated;

[0025] Inputting the first target answer text and the target retrieval context text into the answer authenticity fine-tuning LLM to obtain a score r2;

[0026] The answer authenticity score corresponding to the target RAG system is calculated as:

[0027] f1=W1*r1+W2*r2,

[0028] Among them, W1 and W2 are hyperparameters.

[0029] In a further technical solution, the RAG system performance evaluation method, wherein the performance of the target RAG system is evaluated based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, to obtain the corresponding answer authenticity score, answer relevance score and context relevance score, wherein,

[0030] Based on the answer relevance, the LLM is fine-tuned to evaluate the performance of the target RAG system and obtain the corresponding answer relevance score, including:

[0031] Design the second prompt based on the given second target answer text;

[0032] Inputting the second prompt and the second target answer text into the preset LLM to generate a candidate question sentence;

[0033] The candidate question and the original first target question text are converted into vector form using the preset text conversion model, and the similarity score ri between the vector of each candidate question and the vector of the first target question text is calculated, and the average score r3 of all similarity scores ri is calculated as output:

[0034]

[0035] Inputting the first target question text and the second target answer text into the answer relevance fine-tuning LLM to obtain a score r4;

[0036] The answer relevance score corresponding to the target RAG system is calculated as:

[0037] f2=W3*r3+W4*r4,

[0038] Among them, W3 and W4 are hyperparameters.

[0039] In a further technical solution, the RAG system performance evaluation method, wherein the performance of the target RAG system is evaluated based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, to obtain the corresponding answer authenticity score, answer relevance score and context relevance score, wherein,

[0040] Based on the context relevance, the LLM is fine-tuned to evaluate the performance of the target RAG system to obtain the corresponding context relevance score, including:

[0041] Design the third prompt based on the given target context text and the second target question text;

[0042] Input the third prompt, the target context text and the second target question text into the preset LLM to generate candidate context sentences;

[0043] The candidate context sentences and the second target question text are converted into vector form using the preset text-to-text model, and the similarity score rj between the vector of each candidate context sentence and the vector of the second target question text is calculated, and the average score r5 of all similarity scores rj is calculated as output:

[0044]

[0045] Inputting the second target question text and the target context text into the context-relevance fine-tuning LLM to obtain a score r6;

[0046] The context relevance score corresponding to the target RAG system is calculated as:

[0047] f3=W5*r5+W6*r6,

[0048] Among them, W5 and W6 are hyperparameters.

[0049] In a further technical solution, the RAG system performance evaluation method is described, wherein the text steering model is a bge-large model.

[0050] A RAG system performance evaluation device, comprising:

[0051] An LLM fine-tuning module is used to fine-tune the three preset LLMs based on the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, respectively, to obtain the corresponding answer authenticity fine-tuning LLM, answer relevance fine-tuning LLM and context relevance fine-tuning LLM;

[0052] A performance evaluation module, used to perform performance evaluation on the target RAG system based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, to obtain corresponding answer authenticity scores, answer relevance scores and context relevance scores;

[0053] A result generation module is used to generate a performance evaluation result of the target RAG system based on the answer authenticity score, the answer relevance score and the context relevance score.

[0054] A computer device, wherein the computer device comprises at least one processor; and

[0055] a memory communicatively connected to the at least one processor; wherein,

[0056] The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the RAG system performance evaluation method as described in any one of the above items can be implemented.

[0057] A non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, it can implement the RAG system performance evaluation method as described in any one of the above items.

[0058] Compared with the prior art, the present invention provides a RAG system performance evaluation method, device, computer equipment and non-volatile computer-readable storage medium, wherein the method includes: based on the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, respectively fine-tuning the three preset LLMs to obtain the corresponding answer authenticity fine-tuning LLM, answer relevance fine-tuning LLM and context relevance fine-tuning LLM; based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively evaluating the performance of the target RAG system to obtain the corresponding answer authenticity score, answer relevance score and context relevance score; based on the answer authenticity score, the answer relevance score and the context relevance score, generating the performance evaluation result of the target RAG system. In this way, the method of the present invention can achieve a fast, efficient and accurate evaluation of the performance of the target RAG system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A flowchart of a RAG system performance evaluation method provided by an embodiment of the present invention.

[0061] Figure 2 A schematic diagram of functional modules of a RAG system performance evaluation device provided in an embodiment of the present invention.

[0062] Figure 3A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0064] In the description of the present invention, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps therein is not limited and can be appropriately adjusted as needed.

[0065] Various non-limiting embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0066] Currently, in the field of artificial intelligence (AI), with the rise of large language models (LLM), some researchers have found that although LLM captures enough knowledge to be comparable to human performance in various question-answering benchmarks, LLM cannot answer questions about events that occurred after the model was trained. The solution to this problem is to rely on retrieval-augmented generation (RAG), so RAG is one of the most popular large model cutting-edge technologies at present.

[0067] RAG has also been widely used in the insurance field, such as the A-end claims assistant, group underwriting robot, and easy-to-read business analysis assistant. The usefulness of RAG is obvious, but their implementation requires a lot of adjustments. This is because the overall performance of the RAG system will be affected by the retrieval model, corpus, LLM or prompt expression, etc. In order to ensure that the RAG system can run efficiently and provide high-quality performance, developers must accurately adjust and optimize it. Therefore, automatic performance evaluation of the RAG system is crucial.

[0068] At present, there are two main methods for evaluating the performance of RAG systems: 1. Using public data sets in nlp tasks for evaluation, such as machine reading comprehension question and answer, automatic summarization, entity extraction, etc. On the one hand, the data of these nlp tasks are significantly different from the data required by RAG, and there will be significant adaptability issues. Therefore, professional labelers are required to perform secondary processing on them. This process is undoubtedly time-consuming and labor-intensive. On the other hand, the knowledge in the insurance field is highly professional, and the differences between different sub-fields are significant. For example, there are essential differences in the knowledge points and business logic involved in the claims processing and underwriting processes. This means that these public data cannot effectively evaluate the performance of insurance RAGs; 2. Writing some simple prompts and using the reasoning ability of LLM to evaluate RAGs, such as RAGAS (Retrieval-Augmented Generation Systems), but this method has almost no adaptability to RAGs in vertical fields, so this method cannot effectively evaluate insurance RAGs.

[0069] Based on this, how to provide a RAG system performance evaluation method, device, computer equipment and non-volatile computer-readable storage medium that can achieve fast, efficient and accurate evaluation of the performance of the target RAG system is a problem that needs to be urgently solved by technical personnel in this field.

[0070] Therefore, to solve the above problem, please refer to Figure 1 The embodiment of the present invention provides a RAG system performance evaluation method, wherein the method comprises the steps of:

[0071] S100, based on the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, respectively fine-tune the three preset LLMs to obtain the corresponding answer authenticity fine-tuned LLM, answer relevance fine-tuned LLM and context relevance fine-tuned LLM;

[0072] S200, based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, the target RAG system is evaluated for performance, and corresponding answer authenticity scores, answer relevance scores and context relevance scores are obtained;

[0073] S300. Generate a performance evaluation result of the target RAG system based on the answer authenticity score, the answer relevance score and the context relevance score.

[0074] Furthermore, the RAG system performance evaluation method, wherein the three preset LLMs are fine-tuned based on the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index to obtain the corresponding answer authenticity fine-tuned LLM, answer relevance fine-tuned LLM and context relevance fine-tuned LLM, includes:

[0075] Organize and obtain knowledge in the target field;

[0076] Based on the target domain knowledge, according to the requirements of the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, the corresponding answer authenticity training data, the answer relevance training data and the context relevance training data are sorted out;

[0077] Based on the answer authenticity training data, the answer relevance training data and the context relevance training data, the LoRA algorithm is used to fine-tune the three preset LLMs respectively to obtain the corresponding answer authenticity fine-tuning LLM, answer relevance fine-tuning LLM and context relevance fine-tuning LLM.

[0078] Furthermore, the RAG system performance evaluation method, wherein, based on the target domain knowledge, according to the requirements of the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, the corresponding answer authenticity training data, the answer relevance training data and the context relevance training data are sorted out, wherein,

[0079] The answer authenticity training data includes answer text, retrieval context text, and score labels of the correlation between the two;

[0080] The answer relevance training data includes answer text, question text, and score labels of the relevance between the two;

[0081] The context relevance training data includes context text, question text, and score labels of the relevance between the context text and question text.

[0082] In the specific implementation, in this embodiment, first, the target domain knowledge (such as basic characteristic knowledge in the insurance field) is sorted out; then, based on the target domain knowledge, according to the requirements of the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, the corresponding answer authenticity training data, answer relevance training data and context relevance training data are sorted out; specifically, the answer authenticity training data includes the answer text, the search context text, and the score label of the relevance between the two (the value is 0-10), the answer relevance training data includes the answer text, the question text, and the score label of the relevance between the two (the value is 0-10), and the context relevance training data includes the context text, the question text, and the score label of the relevance between the two (the value is 0-10); finally, based on the answer authenticity training data, the answer relevance training data and the context relevance training data, the LoRA algorithm is used to fine-tune the three preset LLMs respectively to obtain the corresponding answer authenticity fine-tuning LLM, answer relevance fine-tuning LLM and context relevance fine-tuning LLM;

[0083] Among them, the LoRA algorithm (Low-Rank Adaptation) is a fine-tuning technology that updates parameters by introducing a low-rank structure in the model's weight matrix. The core idea of ​​this method is to update only a small part of the model's weight matrix to achieve fast and effective adjustment of the model; specifically:

[0084] -Low rank structure: The LoRA algorithm introduces two low-rank matrices in each weight matrix, which are multiplied with the original weight matrix to adjust the weights. Low rank means that these matrices can approximate the transformation of the original matrix with fewer parameters;

[0085] - Parameter efficiency: Since only a portion of the weight matrix is ​​updated, the LoRA algorithm can significantly reduce the number of parameters of the model, thereby reducing storage and computation requirements;

[0086] - Adaptability: LoRA allows the model to quickly adapt to the characteristics of new tasks or datasets while maintaining the original knowledge;

[0087] - Application scenarios: LoRA is particularly suitable for fine-tuning large pre-trained models, as it can improve the performance of the model for specific tasks without significantly increasing the computational burden;

[0088] Fine-tuning is a common technique in deep learning. It is the process of applying a model pre-trained on a large dataset to a more specific task. This usually involves further training on a dataset for a specific task to adjust the parameters of the model to better suit the task. The following are the key points related to fine-tuning:

[0089] - Pre-trained model: Fine-tuning usually starts from a model that has been pre-trained on a large-scale dataset, which has learned rich feature representations;

[0090] - Task-specific: The purpose of fine-tuning is to make the model better at performing a specific downstream task, which usually involves additional training on a task-specific dataset;

[0091] - Parameter update: During fine-tuning, some parameters of the model are updated to adapt to the requirements of the new task, which is usually achieved by adding additional layers to the pre-trained model or adjusting the parameters of the existing layers;

[0092] - Avoid overfitting: Fine-tuning can help the model avoid overfitting on new tasks because it leverages the general features learned during pre-training;

[0093] -Performance improvement: Through fine-tuning, the model can more accurately capture the data distribution and patterns of a specific task, thereby improving task performance;

[0094] -Flexibility: Fine-tuning provides a flexibility that allows the model to be customized for different tasks without having to train from scratch;

[0095] In this embodiment, the LoRA algorithm is used to fine-tune the three preset LLMs, which means that while keeping most of the model parameters unchanged, a small number of parameters are adjusted to optimize the model's performance on answer authenticity, answer relevance, and context relevance evaluation indicators. This can effectively improve the model's performance on specific tasks while maintaining computational efficiency.

[0096] Furthermore, the RAG system performance evaluation method, wherein the performance of the target RAG system is evaluated based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, to obtain the corresponding answer authenticity score, answer relevance score and context relevance score, wherein,

[0097] Based on the answer authenticity fine-tuning LLM, the performance of the target RAG system is evaluated to obtain the corresponding answer authenticity score, including:

[0098] Design the first prompt based on the given first target answer text;

[0099] Input the first prompt and the first target answer text into the preset LLM to generate an answer short sentence;

[0100] The answer phrase and the target retrieval context text are converted into vector form using a preset text conversion model, and a similarity score r1 between the vectors of the two is calculated;

[0101] Inputting the first target answer text and the target retrieval context text into the answer authenticity fine-tuning LLM to obtain a score r2;

[0102] The answer authenticity score corresponding to the target RAG system is calculated as:

[0103] f1=W1*r1+W2*r2,

[0104] Among them, W1 and W2 are hyperparameters.

[0105] In the specific implementation, in this embodiment, first, a first prompt is designed based on a given first target answer text, such as designing a first prompt: "Given a first target answer text, please generate one or more highlighted sentences based on the first target answer text";

[0106] Then, the first prompt and the first target answer text are input into the preset LLM to generate an answer short sentence that can summarize the core content of the answer;

[0107] Then, a preset text conversion model (such as the bge-large model) is used to convert the answer short sentence and the target retrieval context text (retrieved context text) into a vector form, and a similarity score between the vectors of the two is calculated, which is recorded as r1;

[0108] Then, the first target answer text and the target search context text are input into the answer authenticity fine-tuning LLM to obtain a score r2 (with a value between 0 and 10);

[0109] Finally, the answer authenticity score corresponding to the target RAG system is calculated as:

[0110] f1=W1*r1+W2*r2,

[0111] Among them, W1 and W2 are hyperparameters used to balance the contribution of the two scores.

[0112] Furthermore, the RAG system performance evaluation method, wherein the performance of the target RAG system is evaluated based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, to obtain the corresponding answer authenticity score, answer relevance score and context relevance score, wherein,

[0113] Based on the answer relevance, the LLM is fine-tuned to evaluate the performance of the target RAG system and obtain the corresponding answer relevance score, including:

[0114] Design the second prompt based on the given second target answer text;

[0115] Inputting the second prompt and the second target answer text into the preset LLM to generate a candidate question sentence;

[0116] The candidate question and the original first target question text are converted into vector form using the preset text conversion model, and the similarity score ri between the vector of each candidate question and the vector of the first target question text is calculated, and the average score r3 of all similarity scores ri is calculated as output:

[0117]

[0118] Inputting the first target question text and the second target answer text into the answer relevance fine-tuning LLM to obtain a score r4;

[0119] The answer relevance score corresponding to the target RAG system is calculated as:

[0120] f2=W3*r3+W4*r4,

[0121] Among them, W3 and W4 are hyperparameters.

[0122] In the specific implementation, in this embodiment, first, the second prompt is designed based on the given second target answer text, such as designing the second prompt: "Given the second target answer text, first you need to find one or more key information contained in the second target answer text, and then you are asked to generate one or more questions from the second target answer text based on these key contents.";

[0123] Then, the second prompt and the second target answer text are input into the preset LLM to generate a candidate question sentence that can reflect the core content of the answer text;

[0124] Then, each candidate question and the original first target question text are converted into a vector form using the preset text conversion model, and the similarity score ri between the vector of each candidate question and the vector of the first target question text is calculated, and the average score r3 of all similarity scores ri is calculated as output:

[0125]

[0126] Then, the first target question text and the second target answer text are input into the answer relevance fine-tuning LLM to obtain a score r4 (with a value between 0 and 10);

[0127] Finally, the answer relevance score corresponding to the target RAG system is calculated as:

[0128] f2=W3*r3+W4*r4,

[0129] Among them, W3 and W4 are hyperparameters used to balance the contribution of the two scores.

[0130] Furthermore, the RAG system performance evaluation method, wherein the performance of the target RAG system is evaluated based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, to obtain the corresponding answer authenticity score, answer relevance score and context relevance score, wherein,

[0131] Based on the context relevance, the LLM is fine-tuned to evaluate the performance of the target RAG system to obtain the corresponding context relevance score, including:

[0132] Design the third prompt based on the given target context text and the second target question text;

[0133] Input the third prompt, the target context text and the second target question text into the preset LLM to generate candidate context sentences;

[0134] The candidate context sentences and the second target question text are converted into vector form using the preset text-to-text model, and the similarity score rj between the vector of each candidate context sentence and the vector of the second target question text is calculated, and the average score r5 of all similarity scores rj is calculated as output:

[0135]

[0136] Inputting the second target question text and the target context text into the context-relevance fine-tuning LLM to obtain a score r6;

[0137] The context relevance score corresponding to the target RAG system is calculated as:

[0138] f3=W5*r5+W6*r6,

[0139] Among them, W5 and W6 are hyperparameters.

[0140] In the specific implementation, in this embodiment, first, the third prompt is designed based on the given target context text and the second target question text, for example, the third prompt is designed: "Given a target context text and a second target question text, please extract sentences from the target context text that are helpful in answering the question based on the second target question text";

[0141] Then, the third prompt, the target context text and the second target question text are input into the preset LLM to generate a series of candidate context sentences, which should be related to the question text and may contain key information of the answer;

[0142] Then, each candidate context sentence and the second target question text are converted into a vector form using the preset text-to-text model, and the similarity score rj between the vector of each candidate context sentence and the vector of the second target question text is calculated, and the average score r5 of all similarity scores rj is calculated as output:

[0143]

[0144] Then, the second target question text and the target context text are input into the context-relevance fine-tuning LLM to obtain a score r6;

[0145] Finally, the context relevance score corresponding to the target RAG system is calculated as:

[0146] f3=W5*r5+W6*r6,

[0147] Among them, W5 and W6 are hyperparameters used to balance the contribution of the two scores.

[0148] Furthermore, in the RAG system performance evaluation method, the text steering model is a bge-large model.

[0149] In the specific implementation, in this embodiment, the bge-large model is a large-scale Chinese pre-trained language model. The following is a brief introduction to the bge-large model: 1. Specialization: The bge-large model is mainly optimized for Chinese text and is suitable for processing Chinese natural language processing tasks; 2. Pre-training: The model is pre-trained on a large amount of Chinese data to learn the patterns and structures of the language; 3. Vectorization capability: It can convert Chinese text into high-quality vector representations to capture the semantic information of the text; 4. Applicability: It is suitable for various NLP tasks, such as text classification, sentiment analysis, question-answering systems, etc.; 5. Model architecture: Based on the Transformer architecture, it uses the self-attention mechanism to process sequence data; 6. Fine-tuning: The pre-trained model can be fine-tuned for specific tasks to improve task performance; The bge-large model is an important resource in the field of Chinese NLP, providing researchers and developers with powerful language understanding capabilities.

[0150] It can be seen from the above method embodiments that the RAG system performance evaluation method provided by the present invention at least includes: based on the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, respectively fine-tuning the three preset LLMs to obtain the corresponding answer authenticity fine-tuning LLM, answer relevance fine-tuning LLM and context relevance fine-tuning LLM; based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively evaluating the performance of the target RAG system to obtain the corresponding answer authenticity score, answer relevance score and context relevance score; based on the answer authenticity score, the answer relevance score and the context relevance score, generating the performance evaluation result of the target RAG system; etc. In this way, the method of the present invention can achieve a fast, efficient and accurate evaluation of the performance of the target RAG system.

[0151] Specifically, the RAG system performance evaluation method provided by the present invention has at least the following beneficial technical effects:

[0152] 1. This invention proposes a RAG system performance evaluation method based on the insurance field. This evaluation method aims to provide an efficient and convenient testing method for scholars and developers engaged in insurance RAG research, reducing the cost of manual annotation. This evaluation method evaluates answer authenticity, answer relevance, and context relevance from the perspective of evaluating retrieval quality and generation quality. Through this method, researchers can understand the performance of their own RAG system in real time, so as to make precise adjustments and optimizations to ensure the efficient operation and excellent performance of the RAG system.

[0153] 2. For the three evaluation indicators, we use insurance expertise to fine-tune the three LLMs. Compared with traditional methods, this allows LLMs with rich knowledge to fully learn the knowledge in the insurance field, thereby improving the accuracy and authenticity of the evaluation;

[0154] 3. Based on the specific needs of the insurance field, three prompts are designed and combined with the evaluation results of fine-tuning LLM, aiming to use the powerful reasoning ability of LLM to complete the evaluation of the consistency between the answer and the retrieval context, the evaluation of the consistency between the answer and the question, and the evaluation of the consistency between the context and the question; this design is more in line with the needs of the insurance field and takes into account the architecture and algorithm details of LLM, so as to improve the accuracy of RAG system performance evaluation.

[0155] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0156] Based on the above method embodiment, please refer to Figure 2 Another embodiment of the present invention further provides a RAG system performance evaluation device, wherein the device comprises:

[0157] An LLM fine-tuning module 11 is used to fine-tune the three preset LLMs based on the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, respectively, to obtain the corresponding answer authenticity fine-tuning LLM, answer relevance fine-tuning LLM and context relevance fine-tuning LLM;

[0158] A performance evaluation module 12 is used to perform performance evaluation on the target RAG system based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, to obtain corresponding answer authenticity scores, answer relevance scores and context relevance scores;

[0159] The result generating module 13 is used to generate the performance evaluation result of the target RAG system based on the answer authenticity score, the answer relevance score and the context relevance score.

[0160] The specific implementation method is shown in the above method embodiment, which will not be described again here.

[0161] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides a computer device, wherein the computer device 10 comprises:

[0162] memory 120 and one or more processors 110, Figure 3 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a communication bus or other means. Figure 3 The example of connecting via a communication bus is taken as an example.

[0163] The processor 110 is used to complete various control logics of the computer device 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP, or any other such configuration.

[0164] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the computer program corresponding to the RAG system performance evaluation method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the computer device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the RAG system performance evaluation method in the above method embodiment.

[0165] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device or at least one function; the data storage area may store data created according to the use of the computer device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the computer device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0166] One or more units are stored in the memory 120, and when executed by one or more processors 110, the RAG system performance evaluation method in any of the above method embodiments can be implemented, for example, the above described Figure 1 The method comprises steps S100 to S300.

[0167] Those skilled in the art will understand that Figure 3 The hardware structure diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] Based on the above method embodiments, another embodiment of the present invention further provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the RAG system performance evaluation method in any of the above method embodiments can be implemented, for example, the above described Figure 1 The method comprises steps S100 to S300.

[0169] As an example, the non-volatile storage medium can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable ROM (EEPROM), or a flash memory. Volatile memory can include a random access memory (RAM) as an external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synch link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0170] Based on the above method embodiments, another embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a processor, the RAG system performance evaluation method in any of the above method embodiments can be implemented, for example, the above described Figure 1 The method comprises steps S100 to S300.

[0171] The embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can exist in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0173] Conditional language such as "can," "can," "might," or "may," unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey, among other things, that a particular embodiment can include (while other embodiments do not) a particular feature, element, and / or operation. Thus, such conditional language is also generally intended to imply that a feature, element, and / or operation is required for one or more embodiments anyway or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular embodiment.

[0174] What has been described herein, in this specification and in the accompanying drawings includes examples that can provide a method, apparatus, computer device, and non-volatile computer-readable storage medium for evaluating the performance of a RAG system. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it is recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is obvious that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure, but all these various modifications should fall within the scope of the claims attached to the present invention. In addition, or in the alternative, other embodiments of the present disclosure may be apparent from consideration of the present specification and the accompanying drawings and from the practice of the present disclosure as presented herein. It is intended that the examples presented in the present specification and the accompanying drawings be considered to be illustrative and not restrictive in all respects. Although specific terms are employed herein, they are used in a general and descriptive sense and are not used for limiting purposes.

Claims

1. A RAG system performance evaluation method, characterized in that: include: Based on the answer authenticity evaluation index, answer relevance evaluation index and context relevance evaluation index, the three preset LLMs are fine-tuned respectively to obtain the corresponding answer authenticity fine-tuned LLM, answer relevance fine-tuned LLM and context relevance fine-tuned LLM; Based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, the target RAG system is evaluated for performance to obtain corresponding answer authenticity scores, answer relevance scores and context relevance scores; Based on the answer authenticity score, the answer relevance score and the context relevance score, a performance evaluation result of the target RAG system is generated.

2. The RAG system performance evaluation method according to claim 1, characterized in that: The three preset LLMs are fine-tuned based on the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index to obtain the corresponding answer authenticity fine-tuned LLM, answer relevance fine-tuned LLM and context relevance fine-tuned LLM, including: Organize and obtain knowledge in the target field; Based on the target domain knowledge, according to the requirements of the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, the corresponding answer authenticity training data, the answer relevance training data and the context relevance training data are sorted out; Based on the answer authenticity training data, the answer relevance training data and the context relevance training data, the LoRA algorithm is used to fine-tune the three preset LLMs respectively to obtain the corresponding answer authenticity fine-tuning LLM, answer relevance fine-tuning LLM and context relevance fine-tuning LLM.

3. The RAG system performance evaluation method according to claim 2, characterized in that: Based on the target domain knowledge, according to the requirements of the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, the corresponding answer authenticity training data, the answer relevance training data and the context relevance training data are sorted out, wherein: The answer authenticity training data includes answer text, retrieval context text, and score labels of the correlation between the two; The answer relevance training data includes answer text, question text, and score labels of the relevance between the two; The context relevance training data includes context text, question text, and score labels of the relevance between the context text and question text.

4. The RAG system performance evaluation method according to claim 2 or 3, characterized in that: Based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, the target RAG system is evaluated for performance, and the corresponding answer authenticity score, answer relevance score and context relevance score are obtained, wherein: Based on the answer authenticity fine-tuning LLM, the performance of the target RAG system is evaluated to obtain the corresponding answer authenticity score, including: Design the first prompt based on the given first target answer text; Input the first prompt and the first target answer text into the preset LLM to generate an answer short sentence; The answer phrase and the target retrieval context text are converted into vector form using a preset text conversion model, and a similarity score r1 between the vectors of the two is calculated; Inputting the first target answer text and the target retrieval context text into the answer authenticity fine-tuning LLM to obtain a score r2; The answer authenticity score corresponding to the target RAG system is calculated as: f1=W1*r1+W2*r2, Among them, W1 and W2 are hyperparameters.

5. The RAG system performance evaluation method according to claim 4, characterized in that: Based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, the target RAG system is evaluated for performance, and the corresponding answer authenticity score, answer relevance score and context relevance score are obtained, wherein: Based on the answer relevance, the LLM is fine-tuned to evaluate the performance of the target RAG system and obtain the corresponding answer relevance score, including: Design the second prompt based on the given second target answer text; Inputting the second prompt and the second target answer text into the preset LLM to generate a candidate question sentence; The candidate question and the original first target question text are converted into vector form using the preset text conversion model, and the similarity score ri between the vector of each candidate question and the vector of the first target question text is calculated, and the average score r3 of all similarity scores ri is calculated as output: Inputting the first target question text and the second target answer text into the answer relevance fine-tuning LLM to obtain a score r4; The answer relevance score corresponding to the target RAG system is calculated as: f2=W3*r3+W4*r4, Among them, W3 and W4 are hyperparameters.

6. The RAG system performance evaluation method according to claim 5, characterized in that: Based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, the target RAG system is evaluated for performance, and the corresponding answer authenticity score, answer relevance score and context relevance score are obtained, wherein: Based on the context relevance, the LLM is fine-tuned to evaluate the performance of the target RAG system to obtain the corresponding context relevance score, including: Design the third prompt based on the given target context text and the second target question text; Input the third prompt, the target context text and the second target question text into the preset LLM to generate candidate context sentences; The candidate context sentences and the second target question text are converted into vector form using the preset text-to-text model, and the similarity score rj between the vector of each candidate context sentence and the vector of the second target question text is calculated, and the average score r5 of all similarity scores rj is calculated as output: Inputting the second target question text and the target context text into the context-relevance fine-tuning LLM to obtain a score r6; The context relevance score corresponding to the target RAG system is calculated as: f3=W5*r5+W6*r6, Among them, W5 and W6 are hyperparameters.

7. The RAG system performance evaluation method according to claim 6, characterized in that: The text steering model is a bge-large model.

8. A RAG system performance evaluation device, characterized in that: include: An LLM fine-tuning module is used to fine-tune the three preset LLMs based on the answer authenticity evaluation index, the answer relevance evaluation index and the context relevance evaluation index, respectively, to obtain the corresponding answer authenticity fine-tuning LLM, answer relevance fine-tuning LLM and context relevance fine-tuning LLM; A performance evaluation module, used to perform performance evaluation on the target RAG system based on the answer authenticity fine-tuning LLM, the answer relevance fine-tuning LLM and the context relevance fine-tuning LLM, respectively, to obtain corresponding answer authenticity scores, answer relevance scores and context relevance scores; A result generation module is used to generate a performance evaluation result of the target RAG system based on the answer authenticity score, the answer relevance score and the context relevance score.

9. A computer device, characterized in that: The computer device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the RAG system performance evaluation method according to any one of claims 1 to 7 can be implemented.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the RAG system performance evaluation method according to any one of claims 1 to 7 can be implemented.

Images