RAG evaluation data generation method and device, computer equipment and storage medium

Abstract

The invention discloses an RAG evaluation data generation method and device, computer equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: designing a first prompt based on a collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context; designing a second prompt on the basis of the generated context, and inputting the second prompt and the context into the LLM, so as to generate a corresponding query; and designing a third prompt on the basis of the generated query, and inputting the third prompt, the query and the context into the LLM to generate a corresponding answer. According to the method, the RAG evaluation data can be quickly, efficiently and accurately generated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and non-volatile computer-readable storage medium for generating RAG evaluation data. Background Art

[0002] At present, with the rapid evolution and development of artificial intelligence (AI) technology, Retrieval-augmented Generation (RAG) technology has emerged in the current AI field and is regarded as a cutting-edge technology; especially in the insurance industry, the application of RAG technology is gradually spreading, and its application in automated claims processing, group insurance underwriting process, and business operation analysis is increasing.

[0003] Currently, in order to ensure that the RAG system can run efficiently and provide high-quality performance, developers must accurately adjust and optimize it. This process is usually carried out before the RAG system is put into actual operation to ensure service quality; therefore, during the RAG system development phase, researchers need to use professional test data to evaluate system performance. Especially in the insurance field, most researchers need to manually annotate test questions, contextual text to be retrieved, standard answers, and answers generated by the model. However, this manual annotation process requires the executor to have in-depth knowledge of the insurance field, which makes the annotation work not only time-consuming and labor-intensive, inefficient, but also expensive in labor costs.

[0004] Based on this, how to provide a RAG evaluation data generation method, apparatus, computer device and non-volatile computer-readable storage medium that can quickly, efficiently and accurately generate RAG evaluation data is a problem that urgently needs to be solved by those skilled in the art. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide a RAG evaluation data generation method, apparatus, computer equipment and non-volatile computer-readable storage medium that can be used in financial technology or other related fields, which can generate RAG evaluation data quickly, efficiently and accurately.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for generating RAG evaluation data, comprising:

[0008] Designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context;

[0009] Design a second prompt based on the generated context, input the second prompt and the context into the LLM, and generate a corresponding query;

[0010] A third prompt is designed based on the generated query, and the third prompt, the query and the context are input into the LLM to generate a corresponding answer.

[0011] In a further technical solution, the RAG evaluation data generation method, wherein the first prompt is designed based on the collected target knowledge document, the first prompt and the target knowledge document are input into a preset LLM, and a corresponding context is generated, includes:

[0012] Collect the required target knowledge documents;

[0013] Designing a required first prompt based on the collected target knowledge document;

[0014] The first prompt and the target knowledge document are input into a preset LLM to generate corresponding multiple contexts.

[0015] In a further technical solution, the RAG evaluation data generation method, wherein the first prompt and the target knowledge document are input into a preset LLM to generate a corresponding plurality of contexts, further comprises:

[0016] A high-similarity negative sample and a low-similarity negative sample of each context are calculated.

[0017] In a further technical solution, the RAG evaluation data generation method, wherein the calculation to obtain the high-similarity negative samples and the low-similarity negative samples of each context, comprises:

[0018] Convert each of the context's texts into a vector;

[0019] Calculate the cosine similarity between each context vector and other context vectors;

[0020] According to the similarity results, high-similarity negative samples and low-similarity negative samples of each context are determined.

[0021] In a further technical solution, the RAG evaluation data generation method, wherein after determining the high similarity negative samples and the low similarity negative samples of each context according to the similarity result, further comprises:

[0022] The Kendall rank correlation coefficient algorithm is used to quantitatively analyze the correlation between the high-similarity negative samples and the low-similarity negative samples of each context.

[0023] In a further technical solution, the RAG evaluation data generation method, wherein the text of each context is converted into a vector, comprises:

[0024] The preset BGE-large-model is used to convert the text of each context into a vector.

[0025] In a further technical solution, the RAG evaluation data generation method, wherein the third prompt is designed based on the generated query, the third prompt, the query and the context are input into the LLM, and after the corresponding answer is generated, further comprises:

[0026] Based on the query, the answer, the context, and the high-similarity negative samples and the low-similarity negative samples corresponding to the context, the performance of the target RAG system is evaluated.

[0027] A RAG evaluation data generating device, comprising:

[0028] A context generation module, used for designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context;

[0029] A query generation module, used for designing a second prompt based on the generated context, inputting the second prompt and the context into the LLM, and generating a corresponding query;

[0030] The answer generation module is used to design a third prompt based on the generated query, input the third prompt, the query and the context into the LLM, and generate a corresponding answer.

[0031] A computer device, wherein the computer device comprises at least one processor; and

[0032] a memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the RAG evaluation data generating method as described in any one of the above items can be implemented.

[0034] A non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the RAG evaluation data generation method as described in any one of the above items can be implemented.

[0035] Compared with the prior art, the present invention provides a method, device, computer equipment and non-volatile computer-readable storage medium for generating RAG evaluation data, wherein the method includes: designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context; designing a second prompt based on the generated context, inputting the second prompt and the context into the LLM, and generating a corresponding query; designing a third prompt based on the generated query, inputting the third prompt, the query and the context into the LLM, and generating a corresponding answer. In this way, the method of the present invention can generate RAG evaluation data quickly, efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A schematic flow chart of a method for generating RAG evaluation data provided by an embodiment of the present invention.

[0038] Figure 2 A schematic diagram of functional modules of a RAG evaluation data generating device provided in an embodiment of the present invention.

[0039] Figure 3 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] In the description of the present invention, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps therein is not limited and can be appropriately adjusted as needed.

[0042] Various non-limiting embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0043] At present, with the rapid evolution and development of artificial intelligence (AI) technology, Retrieval-augmented Generation (RAG) technology has emerged in the current AI field and is regarded as a cutting-edge technology; especially in the insurance industry, the application of RAG technology is gradually spreading, and its application in automated claims processing, group insurance underwriting process, and business operation analysis is increasing.

[0044] Currently, in order to ensure that the RAG system can run efficiently and provide high-quality performance, developers must accurately adjust and optimize it. This process is usually carried out before the RAG system is put into actual operation to ensure service quality; therefore, during the RAG system development phase, researchers need to use professional test data to evaluate system performance. Especially in the insurance field, most researchers need to manually annotate test questions, contextual text to be retrieved, standard answers, and answers generated by the model. However, this manual annotation process requires the executor to have in-depth knowledge of the insurance field, which makes the annotation work not only time-consuming and labor-intensive, inefficient, but also expensive in labor costs.

[0045] Based on this, how to provide a RAG evaluation data generation method, apparatus, computer device and non-volatile computer-readable storage medium that can quickly, efficiently and accurately generate RAG evaluation data is a problem that urgently needs to be solved by those skilled in the art.

[0046] Therefore, to solve the above problem, please refer to Figure 1 The embodiment of the present invention provides a method for generating RAG evaluation data, wherein the method comprises the steps of:

[0047] S100, designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context;

[0048] S200, designing a second prompt based on the generated context, inputting the second prompt and the context into the LLM, and generating a corresponding query;

[0049] S300, designing a third prompt based on the generated query, inputting the third prompt, the query and the context into the LLM, and generating a corresponding answer.

[0050] In the specific implementation, in this embodiment, the target knowledge document may be an insurance knowledge document; prompt refers to a form of input used to guide or stimulate the LLM model to generate a specific output; LLM (Large Language Model) is a preset large language model; context is the generated context; query is the generated question;

[0051] In this embodiment, first, a first prompt is designed based on the collected target knowledge document, and the first prompt and the target knowledge document are input into a preset LLM to generate a corresponding context;

[0052] For example:

[0053] (1) Design prompt: "You are an insurance expert and you are given an insurance knowledge document. First, you need to understand the content of the entire document, and then divide the document into paragraphs with highlighted text. Please note that the text in the context must all come from the original document, and no other content is allowed to be generated.";

[0054] (2) Input prompt and insurance knowledge document and use LLM to generate the highlighted context;

[0055] Then, design a second prompt based on the generated context, input the second prompt and the context into the LLM, and generate a corresponding query;

[0056] For example:

[0057] (1) Design a prompt: “You are an insurance expert. You are given a context related to insurance. Your task is to ask a question from the given context that satisfies the following rules:

[0058] 1) The content of the context should be able to correctly answer the question;

[0059] 2) Questions should start with the section containing the key information;

[0060] 3) The difficulty of the questions should be moderate. ";

[0061] (2) Input prompt and context, and use LLM to generate a query of moderate difficulty;

[0062] Then, design a third prompt based on the generated query, input the third prompt, the query and the context into the LLM, and generate a corresponding answer;

[0063] For example:

[0064] (1) Design a prompt: "You are an insurance expert, and users will ask you questions related to insurance. You are provided with a context related to insurance. First, find the content that is highly relevant to the user's question in the given context, then make a well-reasoned inference, and finally answer the user's question professionally and concisely.";

[0065] (2) Input prompt, query, and context, and use LLM to generate the answer;

[0066] In this way, this embodiment can provide scholars and developers engaged in insurance RAG research with a fast, efficient and accurate evaluation data generation method to automatically generate test questions, relevant context documents, reference answers and model-generated responses; in this way, the demand for professional manual labeling in the development and optimization cycle of the insurance RAG system can be significantly reduced, the complexity of the labeling process can be reduced and the related labor costs can be compressed; at the same time, this embodiment designs three prompts according to the characteristics of the insurance field, aiming to use the large language model (LLM) to generate context containing key information, questions of moderate difficulty and professional answers, deeply considers the actual needs of the insurance field, and fully takes into account the architecture and algorithm characteristics of the LLM, in order to improve the relevance, applicability and overall quality of the generated data.

[0067] Furthermore, the RAG evaluation data generation method, wherein the first prompt is designed based on the collected target knowledge document, the first prompt and the target knowledge document are input into a preset LLM, and a corresponding context is generated, includes:

[0068] Collect the required target knowledge documents;

[0069] Designing a required first prompt based on the collected target knowledge document;

[0070] The first prompt and the target knowledge document are input into a preset LLM to generate corresponding multiple contexts.

[0071] Furthermore, the RAG evaluation data generation method, wherein the first prompt and the target knowledge document are input into a preset LLM to generate corresponding multiple contexts, further comprises:

[0072] A high-similarity negative sample and a low-similarity negative sample of each context are calculated.

[0073] Furthermore, in the RAG evaluation data generation method, the step of calculating the high-similarity negative samples and the low-similarity negative samples of each context includes:

[0074] Convert each of the context's texts into a vector;

[0075] Calculate the cosine similarity between each context vector and other context vectors;

[0076] According to the similarity results, high-similarity negative samples and low-similarity negative samples of each context are determined.

[0077] Furthermore, the RAG evaluation data generation method, wherein after determining the high-similarity negative samples and the low-similarity negative samples of each context according to the similarity result, further comprises:

[0078] The Kendall rank correlation coefficient algorithm is used to quantitatively analyze the correlation between the high-similarity negative samples and the low-similarity negative samples of each context.

[0079] Furthermore, the RAG evaluation data generation method, wherein the step of converting the text of each context into a vector, comprises:

[0080] The preset BGE-large-model is used to convert the text of each context into a vector.

[0081] In specific implementation, in this embodiment, while generating a corresponding plurality of context positive samples, a high-similarity negative sample and a low-similarity negative sample of each context positive sample are also calculated;

[0082] Specifically:

[0083] 1) Text vectorization:

[0084] All contextual text data is input into a pre-trained model (such as BGE-large-model) to be converted into corresponding vector representations. These vectors can capture the semantic information of the text and convert it into a format that is easy to handle mathematically.

[0085] Among them, BGE-large-model, which is the large-size model in the BAAI General Embedding (BGE) model series, is a set of pre-trained models for text embedding, which can map text into low-dimensional dense vectors. These vectors can be used for tasks such as retrieval, classification, clustering or semantic matching, and can support the call of external knowledge for large language models;

[0086] 2) Calculate text similarity:

[0087] Using vector representation, calculate the similarity between each pair of context texts, usually measured by cosine similarity; based on the similarity score, determine the topk most similar (high similarity negative samples) and least similar (low similarity negative samples) samples for each context.

[0088] 3) Kendall rank correlation coefficient calculation:

[0089] For each context, there are now two lists: one containing high-similarity negative samples and the other containing low-similarity negative samples; the Kendall rank correlation coefficient algorithm is applied to evaluate the correlation between the two lists;

[0090] Among them, the calculation method of Kendall rank correlation coefficient is as follows:

[0091] ① Rank assignment: In each list, samples are sorted according to the similarity score, and a rank (rank) is assigned to each sample (element). For example, if element a ranks i in list A, then its rank is i.

[0092] ② Count consistent pairs: Calculate the number of sample pairs with the same ranking order in the two lists, denoted as C;

[0093] ③ Count inconsistent pairs: Calculate the number of sample pairs with different ranking orders in the two lists, denoted as D;

[0094] ④Calculate the Kendall rank correlation coefficient:

[0095] Application formula: Kendall rank correlation coefficient = (CD) / (n*(n-1) / 2), where n is the number of elements in the list, that is, the length of the list;

[0096] significance:

[0097] -If the Kendall rank correlation coefficient is close to +1, it means that the ranking between the high-similarity list and the low-similarity list is highly consistent;

[0098] -If the coefficient is close to -1, it means that the sorting is highly inconsistent;

[0099] - If the coefficient is close to 0, it means that there is no significant correlation between the rankings of the two lists;

[0100] application:

[0101] - This algorithm can be used to evaluate the quality of text datasets, such as in information retrieval, recommendation systems, or any scenario where text similarity needs to be evaluated;

[0102] - It can also help understand the relative relationships between different text samples and how they perform in different contexts;

[0103] Specifically corresponding to this embodiment, the Kendall rank correlation coefficient algorithm is used to quantitatively analyze the correlation between high-similarity negative samples and low-similarity negative samples; statistically, if the correlation between the two is low, it indicates that the correlation between them is weak; from the perspective of data quality, this weak correlation implies that the difference between high-similarity negative samples and low-similarity negative samples is large, thereby reflecting that these negative samples have higher data quality; through this algorithm, this embodiment can effectively determine the negative samples of the correct context text, thereby improving the reliability and accuracy of the RAG evaluation data.

[0104] Furthermore, the RAG evaluation data generation method, wherein the third prompt is designed based on the generated query, the third prompt, the query and the context are input into the LLM, and after the corresponding answer is generated, further comprises:

[0105] Based on the query, the answer, the context, and the high-similarity negative samples and the low-similarity negative samples corresponding to the context, the performance of the target RAG system is evaluated.

[0106] In specific implementation, in this embodiment, after obtaining relevant query, answer, context positive sample and its corresponding high similarity negative sample and low similarity negative sample data, the above-mentioned relevant data can be input into the target RAG system for processing to achieve a comprehensive evaluation of the performance of the target RAG system.

[0107] It can be seen from the above method embodiments that the RAG evaluation data generation method provided by the present invention at least includes: designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context; designing a second prompt based on the generated context, inputting the second prompt and the context into the LLM, and generating a corresponding query; designing a third prompt based on the generated query, inputting the third prompt, the query, and the context into the LLM, and generating a corresponding answer; etc. In this way, the RAG evaluation data can be generated quickly, efficiently, and accurately through the method of the present invention.

[0108] Specifically, in general, the RAG evaluation data set can be simply understood as three parts: questions, knowledge, and answers. The data is divided into positive samples and negative samples depending on whether the knowledge is correct or not;

[0109] In the existing technology, if you want to evaluate the performance of RAG, you can only manually annotate the data set, which requires a lot of manpower and financial resources;

[0110] However, in the RAG evaluation data generation method provided by the present invention, a large language model is used to automatically generate the RAG evaluation data set, and only an original insurance knowledge document needs to be input to output the corresponding evaluation data set, which saves a lot of manpower and financial resources.

[0111] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0112] Based on the above method embodiment, please refer to Figure 2 Another embodiment of the present invention further provides a RAG evaluation data generating device, wherein the device comprises:

[0113] A context generation module 11, configured to design a first prompt based on the collected target knowledge document, input the first prompt and the target knowledge document into a preset LLM, and generate a corresponding context;

[0114] A query generation module 12, configured to design a second prompt based on the generated context, input the second prompt and the context into the LLM, and generate a corresponding query;

[0115] The answer generation module 13 is used to design a third prompt based on the generated query, input the third prompt, the query and the context into the LLM, and generate a corresponding answer.

[0116] In the specific implementation, in this embodiment, the target knowledge document may be an insurance knowledge document; prompt refers to a form of input used to guide or stimulate the LLM model to generate a specific output; LLM (Large Language Model) is a preset large language model; context is the generated context; query is the generated question;

[0117] In this embodiment, first, a first prompt is designed based on the collected target knowledge document, and the first prompt and the target knowledge document are input into a preset LLM to generate a corresponding context;

[0118] For example:

[0119] (1) Design prompt: "You are an insurance expert and you are given an insurance knowledge document. First, you need to understand the content of the entire document, and then divide the document into paragraphs with highlighted text. Please note that the text in the context must all come from the original document, and no other content is allowed to be generated.";

[0120] (2) Input prompt and insurance knowledge document and use LLM to generate the highlighted context;

[0121] Then, design a second prompt based on the generated context, input the second prompt and the context into the LLM, and generate a corresponding query;

[0122] For example:

[0123] (1) Design a prompt: “You are an insurance expert. You are given a context related to insurance. Your task is to ask a question from the given context that satisfies the following rules:

[0124] 1) The content of the context should be able to correctly answer the question;

[0125] 2) Questions should start with the section containing the key information;

[0126] 3) The difficulty of the questions should be moderate. ";

[0127] (2) Input prompt and context, and use LLM to generate a query of moderate difficulty;

[0128] Then, design a third prompt based on the generated query, input the third prompt, the query and the context into the LLM, and generate a corresponding answer;

[0129] For example:

[0130] (1) Design a prompt: "You are an insurance expert, and users will ask you questions related to insurance. You are provided with a context related to insurance. First, find the content that is highly relevant to the user's question in the given context, then make a well-reasoned inference, and finally answer the user's question professionally and concisely.";

[0131] (2) Input prompt, query, and context, and use LLM to generate the answer;

[0132] In this way, this embodiment can provide scholars and developers engaged in insurance RAG research with a fast, efficient and accurate evaluation data generation device to automatically generate test questions, relevant context documents, reference answers and model-generated responses; in this way, the demand for professional manual labeling in the development and optimization cycle of the insurance RAG system can be significantly reduced, the complexity of the labeling process can be reduced and the related labor costs can be compressed; at the same time, this embodiment designs three prompts according to the characteristics of the insurance field, aiming to use the large language model (LLM) to generate context containing key information, questions of moderate difficulty and professional answers, deeply considers the actual needs of the insurance field, and fully takes into account the architecture and algorithm characteristics of the LLM, in order to improve the relevance, applicability and overall quality of the generated data.

[0133] Furthermore, the RAG evaluation data generating device, wherein the first prompt is designed based on the collected target knowledge document, the first prompt and the target knowledge document are input into a preset LLM, and a corresponding context is generated, includes:

[0134] Collect the required target knowledge documents;

[0135] Designing a required first prompt based on the collected target knowledge document;

[0136] The first prompt and the target knowledge document are input into a preset LLM to generate corresponding multiple contexts.

[0137] Furthermore, the RAG evaluation data generating device, wherein after the first prompt and the target knowledge document are input into a preset LLM to generate corresponding multiple contexts, further comprises:

[0138] A high-similarity negative sample and a low-similarity negative sample of each context are calculated.

[0139] Furthermore, in the RAG evaluation data generating device, the step of calculating the high-similarity negative samples and the low-similarity negative samples of each context includes:

[0140] Convert each of the context's texts into a vector;

[0141] Calculate the cosine similarity between each context vector and other context vectors;

[0142] According to the similarity results, high-similarity negative samples and low-similarity negative samples of each context are determined.

[0143] Furthermore, the RAG evaluation data generating device, wherein after determining the high similarity negative samples and the low similarity negative samples of each context according to the similarity result, further comprises:

[0144] The Kendall rank correlation coefficient algorithm is used to quantitatively analyze the correlation between the high-similarity negative samples and the low-similarity negative samples of each context.

[0145] Furthermore, the RAG evaluation data generating device, wherein the step of converting the text of each context into a vector comprises:

[0146] The preset BGE-large-model is used to convert the text of each context into a vector.

[0147] In specific implementation, in this embodiment, while generating a corresponding plurality of context positive samples, a high-similarity negative sample and a low-similarity negative sample of each context positive sample are also calculated;

[0148] Specifically:

[0149] 1) Text vectorization:

[0150] All contextual text data is input into a pre-trained model (such as BGE-large-model) to be converted into corresponding vector representations. These vectors can capture the semantic information of the text and convert it into a format that is easy to handle mathematically.

[0151] Among them, BGE-large-model, which is the large-size model in the BAAI General Embedding (BGE) model series, is a set of pre-trained models for text embedding, which can map text into low-dimensional dense vectors. These vectors can be used for tasks such as retrieval, classification, clustering or semantic matching, and can support the call of external knowledge for large language models;

[0152] 2) Calculate text similarity:

[0153] Using vector representation, calculate the similarity between each pair of context texts, usually measured by cosine similarity; based on the similarity score, determine the topk most similar (high similarity negative samples) and least similar (low similarity negative samples) samples for each context.

[0154] 3) Kendall rank correlation coefficient calculation:

[0155] For each context, there are now two lists: one containing high-similarity negative samples and the other containing low-similarity negative samples; the Kendall rank correlation coefficient algorithm is applied to evaluate the correlation between the two lists;

[0156] Among them, the calculation method of Kendall rank correlation coefficient is as follows:

[0157] ① Rank assignment: In each list, samples are sorted according to the similarity score, and a rank (rank) is assigned to each sample (element). For example, if element a ranks i in list A, then its rank is i.

[0158] ② Count consistent pairs: Calculate the number of sample pairs with the same ranking order in the two lists, denoted as C;

[0159] ③ Count inconsistent pairs: Calculate the number of sample pairs with different ranking orders in the two lists, denoted as D;

[0160] ④Calculate the Kendall rank correlation coefficient:

[0161] Application formula: Kendall rank correlation coefficient = (CD) / (n*(n-1) / 2), where n is the number of elements in the list, that is, the length of the list;

[0162] significance:

[0163] -If the Kendall rank correlation coefficient is close to +1, it means that the ranking between the high-similarity list and the low-similarity list is highly consistent;

[0164] -If the coefficient is close to -1, it means that the sorting is highly inconsistent;

[0165] - If the coefficient is close to 0, it means that there is no significant correlation between the rankings of the two lists;

[0166] application:

[0167] - This algorithm can be used to evaluate the quality of text datasets, such as in information retrieval, recommendation systems, or any scenario where text similarity needs to be evaluated;

[0168] - It can also help understand the relative relationships between different text samples and how they perform in different contexts;

[0169] Specifically corresponding to this embodiment, the Kendall rank correlation coefficient algorithm is used to quantitatively analyze the correlation between high-similarity negative samples and low-similarity negative samples; statistically, if the correlation between the two is low, it indicates that the correlation between them is weak; from the perspective of data quality, this weak correlation implies that the difference between high-similarity negative samples and low-similarity negative samples is large, thereby reflecting that these negative samples have higher data quality; through this algorithm, this embodiment can effectively determine the negative samples of the correct context text, thereby improving the reliability and accuracy of the RAG evaluation data.

[0170] Furthermore, the RAG evaluation data generating device, wherein the third prompt is designed based on the generated query, the third prompt, the query and the context are input into the LLM, and after the corresponding answer is generated, further comprises:

[0171] Based on the query, the answer, the context, and the high-similarity negative samples and the low-similarity negative samples corresponding to the context, the performance of the target RAG system is evaluated.

[0172] In specific implementation, in this embodiment, after obtaining relevant query, answer, context positive sample and its corresponding high similarity negative sample and low similarity negative sample data, the above-mentioned relevant data can be input into the target RAG system for processing to achieve a comprehensive evaluation of the performance of the target RAG system.

[0173] It can be seen from the above device embodiments that the RAG evaluation data generation device provided by the present invention at least includes: designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context; designing a second prompt based on the generated context, inputting the second prompt and the context into the LLM, and generating a corresponding query; designing a third prompt based on the generated query, inputting the third prompt, the query, and the context into the LLM, and generating a corresponding answer; etc. In this way, the RAG evaluation data can be generated quickly, efficiently, and accurately through the device of the present invention.

[0174] Specifically, in general, the RAG evaluation data set can be simply understood as three parts: questions, knowledge, and answers. The data is divided into positive samples and negative samples depending on whether the knowledge is correct or not;

[0175] In the existing technology, if you want to evaluate the performance of RAG, you can only manually annotate the data set, which requires a lot of manpower and financial resources;

[0176] However, in the RAG evaluation data generating device provided by the present invention, a large language model is used to automatically generate the RAG evaluation data set, and only an original insurance knowledge document needs to be input to output the corresponding evaluation data set, thus saving a lot of manpower and financial resources.

[0177] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides a computer device, wherein the computer device 10 includes:

[0178] memory 120 and one or more processors 110, Figure 3 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a communication bus or other means. Figure 3 The example of connecting via a communication bus is taken as an example.

[0179] The processor 110 is used to complete various control logics of the computer device 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP, or any other such configuration.

[0180] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the computer program corresponding to the RAG evaluation data generation method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the computer device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the RAG evaluation data generation method in the above method embodiment.

[0181] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device or at least one function; the data storage area may store data created according to the use of the computer device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the computer device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] The computer device 10, wherein one or more units are stored in the memory 120, when executed by one or more processors 110, can implement the following steps:

[0183] S100, designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context;

[0184] S200, designing a second prompt based on the generated context, inputting the second prompt and the context into the LLM, and generating a corresponding query;

[0185] S300, designing a third prompt based on the generated query, inputting the third prompt, the query and the context into the LLM, and generating a corresponding answer.

[0186] In the specific implementation, in this embodiment, the target knowledge document may be an insurance knowledge document; prompt refers to a form of input used to guide or stimulate the LLM model to generate a specific output; LLM (Large Language Model) is a preset large language model; context is the generated context; query is the generated question;

[0187] In this embodiment, first, a first prompt is designed based on the collected target knowledge document, and the first prompt and the target knowledge document are input into a preset LLM to generate a corresponding context;

[0188] For example:

[0189] (1) Design prompt: "You are an insurance expert and you are given an insurance knowledge document. First, you need to understand the content of the entire document, and then divide the document into paragraphs with highlighted text. Please note that the text in the context must all come from the original document, and no other content is allowed to be generated.";

[0190] (2) Input prompt and insurance knowledge document and use LLM to generate the highlighted context;

[0191] Then, design a second prompt based on the generated context, input the second prompt and the context into the LLM, and generate a corresponding query;

[0192] For example:

[0193] (1) Design a prompt: “You are an insurance expert. You are given a context related to insurance. Your task is to ask a question from the given context that satisfies the following rules:

[0194] 1) The content of the context should be able to correctly answer the question;

[0195] 2) Questions should start with the section containing the key information;

[0196] 3) The difficulty of the questions should be moderate. ";

[0197] (2) Input prompt and context, and use LLM to generate a query of moderate difficulty;

[0198] Then, design a third prompt based on the generated query, input the third prompt, the query and the context into the LLM, and generate a corresponding answer;

[0199] For example:

[0200] (1) Design a prompt: "You are an insurance expert, and users will ask you questions related to insurance. You are provided with a context related to insurance. First, find the content that is highly relevant to the user's question in the given context, then make a well-reasoned inference, and finally answer the user's question professionally and concisely.";

[0201] (2) Input prompt, query, and context, and use LLM to generate the answer;

[0202] In this way, this embodiment can provide scholars and developers engaged in insurance RAG research with a fast, efficient and accurate evaluation data generation device to automatically generate test questions, relevant context documents, reference answers and model-generated responses; in this way, the demand for professional manual labeling in the development and optimization cycle of the insurance RAG system can be significantly reduced, the complexity of the labeling process can be reduced and the related labor costs can be compressed; at the same time, this embodiment designs three prompts according to the characteristics of the insurance field, aiming to use the large language model (LLM) to generate context containing key information, questions of moderate difficulty and professional answers, deeply considers the actual needs of the insurance field, and fully takes into account the architecture and algorithm characteristics of the LLM, in order to improve the relevance, applicability and overall quality of the generated data.

[0203] Furthermore, the computer device, wherein the designing of a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context, comprises:

[0204] Collect the required target knowledge documents;

[0205] Designing a required first prompt based on the collected target knowledge document;

[0206] The first prompt and the target knowledge document are input into a preset LLM to generate corresponding multiple contexts.

[0207] Furthermore, the computer device, wherein after inputting the first prompt and the target knowledge document into a preset LLM to generate corresponding multiple contexts, further comprises:

[0208] A high-similarity negative sample and a low-similarity negative sample of each context are calculated.

[0209] Furthermore, in the computer device, the step of calculating the high-similarity negative samples and the low-similarity negative samples of each context includes:

[0210] Convert each of the context's texts into a vector;

[0211] Calculate the cosine similarity between each context vector and other context vectors;

[0212] According to the similarity results, high-similarity negative samples and low-similarity negative samples of each context are determined.

[0213] Furthermore, the computer device, after determining the high-similarity negative samples and the low-similarity negative samples of each context according to the similarity result, further comprises:

[0214] The Kendall rank correlation coefficient algorithm is used to quantitatively analyze the correlation between the high-similarity negative samples and the low-similarity negative samples of each context.

[0215] Furthermore, in the computer device, the step of converting the text of each context into a vector comprises:

[0216] The preset BGE-large-model is used to convert the text of each context into a vector.

[0217] In specific implementation, in this embodiment, while generating a corresponding plurality of context positive samples, a high-similarity negative sample and a low-similarity negative sample of each context positive sample are also calculated;

[0218] Specifically:

[0219] 1) Text vectorization:

[0220] All contextual text data is input into a pre-trained model (such as BGE-large-model) to be converted into corresponding vector representations. These vectors can capture the semantic information of the text and convert it into a format that is easy to handle mathematically.

[0221] Among them, BGE-large-model, which is the large-size model in the BAAI General Embedding (BGE) model series, is a set of pre-trained models for text embedding, which can map text into low-dimensional dense vectors. These vectors can be used for tasks such as retrieval, classification, clustering or semantic matching, and can support the call of external knowledge for large language models;

[0222] 2) Calculate text similarity:

[0223] Using vector representation, calculate the similarity between each pair of context texts, usually measured by cosine similarity; based on the similarity score, determine the topk most similar (high similarity negative samples) and least similar (low similarity negative samples) samples for each context.

[0224] 3) Kendall rank correlation coefficient calculation:

[0225] For each context, there are now two lists: one containing high-similarity negative samples and the other containing low-similarity negative samples; the Kendall rank correlation coefficient algorithm is applied to evaluate the correlation between the two lists;

[0226] Among them, the calculation method of Kendall rank correlation coefficient is as follows:

[0227] ① Rank assignment: In each list, samples are sorted according to the similarity score, and a rank (rank) is assigned to each sample (element). For example, if element a ranks i in list A, then its rank is i.

[0228] ② Count consistent pairs: Calculate the number of sample pairs with the same ranking order in the two lists, denoted as C;

[0229] ③ Count inconsistent pairs: Calculate the number of sample pairs with different ranking orders in the two lists, denoted as D;

[0230] ④Calculate the Kendall rank correlation coefficient:

[0231] Application formula: Kendall rank correlation coefficient = (CD) / (n*(n-1) / 2), where n is the number of elements in the list, that is, the length of the list;

[0232] significance:

[0233] -If the Kendall rank correlation coefficient is close to +1, it means that the ranking between the high-similarity list and the low-similarity list is highly consistent;

[0234] -If the coefficient is close to -1, it means that the sorting is highly inconsistent;

[0235] - If the coefficient is close to 0, it means that there is no significant correlation between the rankings of the two lists;

[0236] application:

[0237] - This algorithm can be used to evaluate the quality of text datasets, such as in information retrieval, recommendation systems, or any scenario where text similarity needs to be evaluated;

[0238] - It can also help understand the relative relationships between different text samples and how they perform in different contexts;

[0239] Specifically corresponding to this embodiment, the Kendall rank correlation coefficient algorithm is used to quantitatively analyze the correlation between high-similarity negative samples and low-similarity negative samples; statistically, if the correlation between the two is low, it indicates that the correlation between them is weak; from the perspective of data quality, this weak correlation implies that the difference between high-similarity negative samples and low-similarity negative samples is large, thereby reflecting that these negative samples have higher data quality; through this algorithm, this embodiment can effectively determine the negative samples of the correct context text, thereby improving the reliability and accuracy of the RAG evaluation data.

[0240] Furthermore, the computer device, wherein the third prompt is designed based on the generated query, the third prompt, the query and the context are input into the LLM, and after the corresponding answer is generated, further comprises:

[0241] Based on the query, the answer, the context, and the high-similarity negative samples and the low-similarity negative samples corresponding to the context, the performance of the target RAG system is evaluated.

[0242] In specific implementation, in this embodiment, after obtaining relevant query, answer, context positive sample and its corresponding high similarity negative sample and low similarity negative sample data, the above-mentioned relevant data can be input into the target RAG system for processing to achieve a comprehensive evaluation of the performance of the target RAG system.

[0243] It can be seen from the above computer device embodiments that the computer device provided by the present invention can at least achieve: designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context; designing a second prompt based on the generated context, inputting the second prompt and the context into the LLM, and generating a corresponding query; designing a third prompt based on the generated query, inputting the third prompt, the query, and the context into the LLM, and generating a corresponding answer; etc. In this way, the computer device of the present invention can generate RAG evaluation data quickly, efficiently, and accurately.

[0244] Specifically, in general, the RAG evaluation data set can be simply understood as three parts: questions, knowledge, and answers. The data is divided into positive samples and negative samples depending on whether the knowledge is correct or not;

[0245] In the existing technology, if you want to evaluate the performance of RAG, you can only manually annotate the data set, which requires a lot of manpower and financial resources;

[0246] However, in the computer device provided by the present invention, a large language model is used to automatically generate the RAG evaluation data set, and only an original insurance knowledge document needs to be input to output the corresponding evaluation data set, which saves a lot of manpower and financial resources.

[0247] Those skilled in the art will understand that Figure 3 The hardware structure diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0248] Based on the above method embodiments, another embodiment of the present invention further provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the RAG evaluation data generation method in any of the above method embodiments can be implemented, for example, the above described Figure 1 The method comprises steps S100 to S300.

[0249] As an example, the non-volatile storage medium can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable ROM (EEPROM), or a flash memory. Volatile memory can include a random access memory (RAM) as an external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synch link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0250] Based on the above method embodiments, another embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a processor, the RAG evaluation data generation method in any of the above method embodiments can be implemented, for example, the above described Figure 1 The method comprises steps S100 to S300.

[0251] The embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0252] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can exist in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0253] Conditional language such as "can," "can," "might," or "may," unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey, among other things, that a particular embodiment can include (while other embodiments do not) a particular feature, element, and / or operation. Thus, such conditional language is also generally intended to imply that a feature, element, and / or operation is required for one or more embodiments anyway or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular embodiment.

[0254] What has been described herein, in this specification and in the drawings includes examples of a method, apparatus, computer device, and non-volatile computer-readable storage medium for generating RAG assessment data. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it is recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is obvious that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure, but all these various modifications should fall within the scope of the claims attached to the present invention. In addition, or in the alternative, other embodiments of the present disclosure may be apparent from consideration of the present specification and the drawings and from the practice of the present disclosure as presented herein. It is intended that the examples set forth in the present specification and the drawings be considered illustrative and not restrictive in all respects. Although specific terms are employed herein, they are used in a general and descriptive sense and are not used for limiting purposes.

Claims

1. A method for generating RAG evaluation data, characterized in that: include: Designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context; Design a second prompt based on the generated context, input the second prompt and the context into the LLM, and generate a corresponding query; A third prompt is designed based on the generated query, and the third prompt, the query and the context are input into the LLM to generate a corresponding answer.

2. The RAG evaluation data generation method according to claim 1, characterized in that: The step of designing a first prompt based on the collected target knowledge document, inputting the first prompt and the target knowledge document into a preset LLM, and generating a corresponding context includes: Collect the required target knowledge documents; Designing a required first prompt based on the collected target knowledge document; The first prompt and the target knowledge document are input into a preset LLM to generate corresponding multiple contexts.

3. The RAG evaluation data generation method according to claim 2, characterized in that: After inputting the first prompt and the target knowledge document into a preset LLM to generate a corresponding plurality of contexts, the method further includes: A high-similarity negative sample and a low-similarity negative sample of each context are calculated.

4. The RAG evaluation data generation method according to claim 3, characterized in that: The calculation to obtain the high-similarity negative samples and the low-similarity negative samples of each context includes: Convert each of the context's texts into a vector; Calculate the cosine similarity between each context vector and other context vectors; According to the similarity results, high-similarity negative samples and low-similarity negative samples of each context are determined.

5. The RAG evaluation data generation method according to claim 4, characterized in that: After determining the high-similarity negative samples and the low-similarity negative samples of each context according to the similarity results, the method further includes: The Kendall rank correlation coefficient algorithm is used to quantitatively analyze the correlation between the high-similarity negative samples and the low-similarity negative samples of each context.

6. The RAG evaluation data generation method according to claim 5, characterized in that: The text of each context is converted into a vector, including: The preset BGE-large-model is used to convert the text of each context into a vector.

7. The RAG evaluation data generation method according to any one of claims 3 to 6, characterized in that: The step of designing a third prompt based on the generated query, inputting the third prompt, the query and the context into the LLM, and generating a corresponding answer further includes: Based on the query, the answer, the context, and the high-similarity negative samples and the low-similarity negative samples corresponding to the context, the performance of the target RAG system is evaluated.

8. A RAG evaluation data generating device, characterized in that: include: A context generation module, used to design a first prompt based on the collected target knowledge document, input the first prompt and the target knowledge document into a preset LLM, and generate a corresponding context; A query generation module, used for designing a second prompt based on the generated context, inputting the second prompt and the context into the LLM, and generating a corresponding query; The answer generation module is used to design a third prompt based on the generated query, input the third prompt, the query and the context into the LLM, and generate a corresponding answer.

9. A computer device, characterized in that: The computer device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the RAG evaluation data generating method according to any one of claims 1 to 7 can be implemented.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the RAG evaluation data generating method according to any one of claims 1 to 7 can be implemented.

Images