Reinforcement learning based rag system evaluation dataset automatic synthesis method and apparatus

Abstract

The application relates to a method and device for automatically synthesizing an evaluation dataset of an RAG system based on reinforcement learning, wherein the method comprises the following steps: acquiring knowledge base data of the RAG system, and extracting keywords and related documents of a target vertical field from the knowledge base data; synthesizing an initial question and answer pair by using a preset prompt word template according to the keywords and the documents; constructing a reward of the initial question and answer pair; determining training data of an initial strategy model according to the keywords, the initial question and answer pair, the reward and parameters of the initial strategy model; training the initial strategy model based on the training data to obtain a target strategy model; and generating target question and answer pairs meeting the reward based on the keywords and the documents by using the target strategy model. By the application, the problem that the cost of synthesizing an evaluation dataset is high, the efficiency is low and the effectiveness of evaluating an RAG system is affected in the related art is solved, and the synthesis cost can be reduced and the synthesis efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a method and device for automatically synthesizing an evaluation dataset of an RAG system based on reinforcement learning. BACKGROUND

[0002] In a wide range of application scenarios of RAG systems, more RAG systems need to adapt to professional knowledge bases for vertical field scenarios. Since the current evaluation dataset of the RAG system is mainly derived from public general question and answer datasets, this approach is costly and inefficient, and the data quality completely depends on the knowledge level and experience of individual experts, which leads to the fact that these data cannot well meet the evaluation of RAG systems in vertical fields. Therefore, in the vertical field scenario, the scarcity of professional evaluation question and answer data and the lack of field specialization of the evaluation data limit the effectiveness of the evaluation of the RAG system.

[0003] There is currently no effective solution to the problem of high cost and low efficiency of evaluation data synthesis, which affects the effectiveness of the evaluation of the RAG system. SUMMARY

[0004] The present application provides a method and device for automatically synthesizing an evaluation dataset of an RAG system based on reinforcement learning to solve the problem of high cost and low efficiency of evaluation data synthesis in the related art, which affects the effectiveness of the evaluation of the RAG system.

[0005] In the first aspect, the present application provides a method for automatically synthesizing an evaluation dataset of an RAG system based on reinforcement learning, comprising:

[0006] Obtaining knowledge base data of the RAG system, and extracting keywords and related documents of a target vertical field from the knowledge base data;

[0007] According to the keywords and the documents, an initial question and answer pair for evaluating the RAG system is synthesized using a preset prompt word template;

[0008] Constructing a reward for the initial question and answer pair;

[0009] According to the keywords, the initial question and answer pair, the reward, and the parameters of an initial strategy model, training data of the initial strategy model is determined, and the initial strategy model is trained based on the training data to obtain a target strategy model;

[0010] Using the target strategy model, a target question and answer pair satisfying the reward is generated in batches based on the keywords and the documents.

[0011] In some embodiments, keywords and related documents of a target vertical field are extracted from the knowledge base data, including:

[0012] The knowledge base data is preprocessed to obtain structured data.

[0013] A keyword extraction model is used to extract keywords and related documents of a target vertical field from the structured data.

[0014] In some embodiments, the preprocessing includes denoising processing, uniform data format processing, and chunking processing.

[0015] The keyword extraction model is an LLM model.

[0016] In some embodiments, the method further includes:

[0017] The documents are used as candidate documents, and a relevance score between the candidate documents and the keywords is calculated.

[0018] According to the relevance score, all the candidate documents are sorted.

[0019] According to the relevance score, all the candidate documents are sorted.

[0020] In some embodiments, the reward of the initial question-answer pair is constructed, including:

[0021] Based on an evaluation model, a relevance reward in the reward of the initial question-answer pair is constructed.

[0022] Based on a forced policy model, a format reward in the reward of the initial question-answer pair is constructed.

[0023] In some embodiments, based on the training data, the initial policy model is trained to obtain a target policy model, including:

[0024] Based on the training data, the parameters of the initial policy model are iteratively updated with a preset target function as the optimization target until a convergence condition is met to obtain a target policy model.

[0025] In the updating process, the expected total reward of the advantage function in the target function is maximized, and the divergence between the new policy model and the old policy model is controlled through the KL divergence constraint.

[0026] In some embodiments, using the target policy model, target question-answer pairs satisfying the reward are batch generated based on the keywords and the documents, including:

[0027] The target policy model is used to batch generate target question and answer pairs meeting the rewards based on the keywords and the documents.

[0028] The target question and answer pair meeting the correlation reward and the format reward in the rewards is screened out from the candidate initial question and answer pairs.

[0029] In a second aspect, an embodiment of the present application provides an automatic synthesis device of a reinforcement learning-based RAG system evaluation dataset, comprising an acquisition module, a synthesis module, a construction module, a training module and a processing module.

[0030] The acquisition module is configured to acquire knowledge base data of the RAG system, and extract keywords and related documents of a target vertical field from the knowledge base data.

[0031] The synthesis module is configured to synthesize initial question and answer pairs for evaluating the RAG system according to the keywords and the documents and by using a preset prompt word template.

[0032] The construction module is configured to construct rewards of the initial question and answer pairs.

[0033] The training module is configured to determine training data of an initial policy model according to the keywords, the initial question and answer pairs, the rewards and parameters of the initial policy model, and train the initial policy model based on the training data to obtain a target policy model.

[0034] The processing module is configured to batch generate target question and answer pairs meeting the rewards based on the keywords and the documents by using the target policy model.

[0035] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the reinforcement learning-based RAG system evaluation dataset automatic synthesis method of the first aspect when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, and the computer program is executable by a processor to implement the reinforcement learning-based RAG system evaluation dataset automatic synthesis method of the first aspect.

[0037] Compared with the related art, the reinforcement learning-based RAG system evaluation dataset automatic synthesis method and device provided in the embodiment solves the problems of high cost and low efficiency of evaluation dataset synthesis in the related art, affects the effectiveness of RAG system evaluation, and solves the problems of high cost and low efficiency of evaluation dataset synthesis in the related art, affects the effectiveness of RAG system evaluation, and solves the problems of high cost and low efficiency of evaluation dataset synthesis in the related art. The initial question and answer pair synthesis process is conditionally constrained by the extracted keywords and related documents, to ensure that the synthesized initial question and answer pair is closely related to the knowledge base content; and the constructed reward is used to evaluate the quality of the initial question and answer pair, the generation ability of the initial strategy model is continuously optimized through the reinforcement learning method, and the target strategy model is obtained; finally, the automatic synthesis of the high-quality evaluation dataset is realized based on the strategy model, thereby reducing the synthesis cost and improving the synthesis efficiency.

[0038] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0039] The drawings described herein are intended to provide further understanding of the present application, form a part of the present application, and are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0040] Figure 1 is a hardware structure block diagram of a terminal device for the reinforcement learning-based RAG system evaluation dataset automatic synthesis method provided by an embodiment of the present application;

[0041] Figure 2 is a flowchart of the reinforcement learning-based RAG system evaluation dataset automatic synthesis method provided by an embodiment of the present application;

[0042] Figure 3 is a flowchart of step S210;

[0043] Figure 4 is a flowchart of document screening provided by an embodiment of the present application;

[0044] Figure 5 is a flowchart of step S250;

[0045] Figure 6is a structural block diagram of an automatic synthesis device of a reinforcement learning based RAG system evaluation dataset provided by an embodiment of the present application.

[0046] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 210, acquisition module; 220, synthesis module; 230, construction module; 240, training module; 250, processing module. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and explained below in conjunction with the drawings and embodiments.

[0048] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the general meaning understood by a person with ordinary skill in the art to which the present application belongs. In the present application, "one", "a", "an", "the", "these" and similar words do not represent a quantitative limitation, and they can be singular or plural. In the present application, the terms "include", "contain", "have" and any variants thereof have the purpose of covering non-exclusive inclusion; for example, a process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connected", "connected", "coupled" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. In the present application, "multiple" means two or more. The association between the associated objects is described by "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. In general, the character " / " represents an "or" relationship between the objects before and after it. In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0049] The method embodiments provided in the present embodiment can be executed in a terminal, a computer or a similar computing device. For example, the method embodiments are executed on a terminal, Figure 1 is a hardware structure block diagram of a terminal of the reinforcement learning based RAG system evaluation dataset automatic synthesis method of the present embodiment. As shown in Figure 1 the terminal can include one or more (CPU) central processing units, memories, buses, input and output devices, and the like. Figure 1The terminal shown in FIG. 1 includes only one processor 102 and a memory 104 for storing data, wherein the processor 102 can include, but is not limited to, a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA). The terminal can also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the terminal. For example, the terminal can include more or less components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1. Figure 1 For example, the terminal can include more or less components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1. Figure 1 For example, the terminal can include more or less components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1.

[0050] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as the computer program corresponding to the method of automatic synthesis of RAG system evaluation data set based on reinforcement learning in the present embodiment. The processor 102 can execute various functional applications and data processing by running the computer programs stored in the memory 104, i.e., implement the method described above. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0051] The transmission device 106 is used to receive or send data via a network. The network includes a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (NIC), which can be connected to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

[0052] It should be noted that the retrieval augmented generation (RAG) system is a system combining retrieval technology and language generation model. In order to effectively evaluate the RAG system, especially when some modules are abnormal, a special evaluation data set is usually needed. These evaluation data sets contain various high-quality question and answer pairs. However, in specific fields, the existing open-source question and answer data sets lack domain-related professional terms and cannot fully verify the document retrieval and answer generation capabilities of the RAG system in a specific field. Building such an evaluation data set for a specific field requires a lot of resources and the participation of domain experts, which limits the professional evaluation question and answer data scarcity and the lack of evaluation data field expertise in the vertical field scenario, affecting the effectiveness of evaluating the RAG system.

[0053] To solve the above problems, an automatic synthesis method of RAG system evaluation data set based on reinforcement learning is provided in the embodiment, Figure 2 The flowchart of the automatic synthesis method of RAG system evaluation data set based on reinforcement learning in the embodiment is shown in Figure 2 As shown in the figure, the flow includes the following steps:

[0054] Step S210, acquiring the knowledge base data of the RAG system, and extracting the keywords and related documents of the target vertical field from the knowledge base data;

[0055] Step S220, synthesizing an initial question and answer pair according to the keywords and documents using a preset prompt word template;

[0056] Step S230, constructing the reward of the initial question and answer pair;

[0057] Step S240, determining the training data of the initial strategy model according to the keywords, the initial question and answer pair, the reward, and the parameters of the initial strategy model; and training the initial strategy model based on the training data to obtain a target strategy model;

[0058] Step S250, using the target strategy model to batch generate target question and answer pairs that meet the reward based on the keywords and documents.

[0059] Since the RAG systems of different vertical fields each have their own specific knowledge base data, in the embodiment, all knowledge base data is first acquired, and then the keywords and related documents corresponding to the target vertical field are extracted from the knowledge base data. For example, for the heart disease diagnosis and treatment field, the extracted keywords include "high blood pressure", "antihypertensive drugs", "diuretics", and "combined medication". The related documents refer to the documents where these keywords are located or related to the keywords.

[0060] Next, the keywords and the document are input into a preset prompt template to output an initial question and answer pair for evaluating the RAG system. Since the keywords and the document are related to the target vertical field, the data synthesis process is conditionally constrained in the form of keywords, so as to ensure that the synthesized initial question and answer pair is closely related to the content of the knowledge base. In addition, the extraction of the domain keywords also helps to capture the long-tail distribution of professional terms in the knowledge base, thereby improving the diversity and field coverage of the evaluation question and answer pair data. The prompt template includes but is not limited to an unstructured prompt template and a structured prompt template, and is not limited thereto.

[0061] Since the reward is the source of the training signal, it determines the optimization direction of the reinforcement learning. In order to train the initial strategy model to synthesize high-quality question and answer pairs, a corresponding reward is constructed for each initial question and answer pair in the embodiment. The reward can be divided into a relevance reward and a format reward. The relevance reward is used to verify whether the answer synthesized by the strategy model is related to the question and answer pair. The format reward is used to ensure the correctness of the format of the synthesized question and answer pair.

[0062] After obtaining the keywords, the initial question and answer pair, and the reward, the training data can be collected in combination with the parameters of the initial strategy model. The initial strategy model is an untrained strategy model, and the target strategy model is a trained strategy model. For example, the training data . Wherein, θ refers to the parameters of the initial strategy model; is the input of the initial strategy model, which is the keywords corresponding to the target vertical field extracted from the knowledge base; is the output of the initial strategy model, which is the synthesized initial question and answer pair ; the reward of the initial question and answer pair can be the sum of the relevance reward and the format reward. In particular, for each input keyword, a set of output question and answer pairs is sampled from the original strategy model, and the parameters of the initial strategy model are optimized by a related optimization algorithm (such as the GRPO algorithm), so as to complete the training of the initial strategy model and obtain the target strategy model capable of generating high-quality question and answer pairs.

[0063] Finally, the target strategy model is used to batch generate target question and answer pairs satisfying the reward based on the keywords and the document; the dual mechanism of the target strategy model and the reward is used to only take the target question and answer pairs satisfying the reward as the data set for evaluating the RAG system, so as to complete the quality screening of the question and answer pairs.

[0064] In related technologies, the evaluation datasets of RAG systems are mainly derived from publicly available general question-and-answer datasets. However, this approach is costly and inefficient, and the data quality depends entirely on the knowledge and experience of individual experts, which means that these datasets cannot adequately meet the evaluation requirements of RAG systems in vertical domains. Therefore, in vertical domain scenarios, the scarcity of professional evaluation question-and-answer data and the lack of professionalism in the evaluation data domain affect the effectiveness of evaluating RAG systems. In this embodiment, knowledge base data from the RAG system is acquired, and keywords and related documents in the target vertical domain are extracted from the knowledge base data. Initial question-and-answer pairs are synthesized using preset prompt word templates based on the keywords and documents. Rewards for the initial question-and-answer pairs are constructed. Training data for the initial strategy model is determined based on the keywords, initial question-and-answer pairs, rewards, and parameters of the initial strategy model. The initial strategy model is then trained using the training data to obtain the target strategy model. Using the target strategy model, target question-and-answer pairs that meet the rewards are generated in batches based on keywords and documents. This solves the problems of high cost and low efficiency in synthesizing evaluation datasets in related technologies, which affect the effectiveness of evaluating RAG systems. The extracted keywords and related documents are used to constrain the synthesis process of the initial question-and-answer pairs, ensuring that the synthesized initial question-and-answer pairs are closely related to the knowledge base content. The quality of the initial question-and-answer pairs is evaluated using the constructed rewards. The generation capability of the initial strategy model is continuously optimized through reinforcement learning methods to obtain the target strategy model. Finally, the strategy model is used as the basis for the automatic synthesis of high-quality evaluation datasets, thereby reducing synthesis costs and improving synthesis efficiency.

[0065] The steps described above are explained in detail below:

[0066] In some of these embodiments, such as Figure 3 As shown, step S210, which involves extracting keywords and related documents from the knowledge base data for the target vertical domain, includes the following steps:

[0067] Step S211: Preprocess the knowledge base data to obtain structured data;

[0068] Step S212: Use the keyword extraction model to extract keywords and related documents for the target vertical domain from the structured data.

[0069] Specifically, the knowledge base data of a RAG system typically contains knowledge in various formats and modalities, covering structured, semi-structured, and unstructured documents. These documents are often massive in scale and contain a large amount of redundant and irrelevant information. Directly inputting unprocessed documents into the keyword extraction model will reduce the quality of the initial question-answer pairs.

[0070] In order to combine the content of the relevant vertical fields in the knowledge base, the knowledge base data needs to be preprocessed to obtain structured data. Among them, preprocessing includes denoising processing, unified data format processing and cutting processing. Through preprocessing, the original document of the knowledge base data is converted into a structured document set , as the source of initial question and answer pair synthesis to support the synthesis of high-quality initial question and answer pairs. Among them, is the i-th cleaned and cut text segment.

[0071] Since the RAG systems of different vertical fields have their own specific knowledge base data, in order to synthesize evaluation test questions that are highly related to a specific vertical field and representative, by capturing the domain terms and key concepts in the knowledge base data, the synthesis process of the initial question and answer pair is conditionally constrained in the form of keywords, so as to ensure that the synthesized initial question and answer pair is closely related to the knowledge base content. In addition, the extraction of keywords also helps to capture the long-tail distribution of professional terms in the knowledge base, and improves the diversity and field coverage of the target question and answer pair.

[0072] Then, given the preprocessed structured data, i.e. the structured document set , the keywords and related documents of the target vertical field are extracted from the structured data using the keyword extraction model.

[0073] Further, the keyword extraction model can be an LLM model, which sets the task description and provides the related document context for the LLM model, guiding the LLM model to extract the core keyword set from the structured document set and obtain the related documents. Its process can be formally represented as:

[0074] ;

[0075] Among them, represents the i-th extracted keyword, LLM represents a large language model, represents a prompt word template for keyword extraction.

[0076] In some embodiments thereof, as shown in Figure 4 , the reinforcement learning-based RAG system evaluation dataset automatic synthesis method further includes the following steps:

[0077] Step S410, taking the document as a candidate document, calculating the relevance score between the candidate document and the keyword;

[0078] Step S420, ranking all candidate documents according to the relevance score;

[0079] Step S430, selecting the top-ranked candidate documents from the ranking result according to a preset number, and taking the selected candidate documents as new documents.

[0080] Specifically, the document obtained in step S210 is taken as a candidate document, and the relevance score between each candidate document and the keyword is calculated using an information retrieval algorithm such as BM25. Then, all candidate documents are sorted according to the relevance score; the top k documents are selected from the sorted results to form a new document, which can be represented as Drelated.

[0081] Through the above document screening mechanism, the context information input in the subsequent initial question and answer pair generation stage is ensured to have higher accuracy and relevance, thereby improving the quality of the subsequent generated target question and answer pair; at the same time, the redundancy information and noise interference of the knowledge base document can be further reduced, and the input context can be avoided to be too long to exceed the maximum input length limit of the LLM model.

[0082] In some embodiments, step S230 of constructing the reward of the initial question and answer pair includes the following steps:

[0083] Step S231, based on the evaluation model, constructs the relevance reward in the reward of the initial question and answer pair;

[0084] Step S232, based on the forced policy model, constructs the format reward in the reward of the initial question and answer pair.

[0085] Specifically, the relevance reward is used to verify whether the answer synthesized by the strategy model is related to the question and answer pair. For example: if it is related, return reward 1, if it is not related, return reward -1. Further, the evaluation model can be an evaluator large language model (Evaluator LLM), and the evaluation is performed using the Evaluator LLM; the output of the evaluation model is parsed into a binary label, as shown in the following formula:

[0086]

[0087] The format reward is used to ensure the correctness of the synthesized question and answer pair. For example: if the format is correct, return reward 1, if the format is incorrect, return reward -1. Further, the forced policy model can be <question><\Question>and <answer>The forced policy model outputs the correct format, as shown in the following formula:

[0088] ;

[0089] In this embodiment, relevance rewards and format rewards are set to determine the optimization direction of reinforcement learning.

[0090] In some embodiments, step S240, which involves training the initial policy model based on training data to obtain the target policy model, includes the following steps:

[0091] Step S241: Based on the training data, with the preset objective function as the optimization objective, iteratively update the parameters of the initial policy model until the convergence condition is met, and obtain the target policy model.

[0092] Step S242: During the update process, maximize the expectation of the advantage function in the objective function; and ensure that the deviation between the new policy model and the old policy model is controllable through KL divergence constraints.

[0093] Specifically, the training data collected through sampling The training samples are obtained; the process is as follows: for each domain's keywords... Using the current initial policy model (with parameters denoted as θ) and related new documentation. Generate an initial question-and-answer pair. Then, the reward for the initial question-answer pair (the sum of the relevance reward and the format reward) is calculated. This yields a set of training samples. Repeating the above process allows you to sample all the training samples.

[0094] Then, based on these training samples, the GRPO algorithm is used for optimization. The parameters of the initial policy model are iteratively updated with a predefined objective function as the optimization objective until the convergence condition is met, resulting in the target policy model. Furthermore, during the update process, the expected total reward of the advantage function in the objective function is maximized; and KL divergence constraints are used to ensure that the deviation between the new policy model and the old policy model is controllable.

[0095] The objective function is expressed as follows:

[0096] ;

[0097] In the formula, S represents the set of input training samples for the initial policy model. The set of training samples represents the first... 1 training sample; G represents the number of candidate outputs generated by the policy model; Represents the new strategy model; Represents the old strategy model; represents a reference policy model; represents a advantage function; clip(·) represents a clipping function; represents a clipping coefficient, specifically, it can be 0.1; β represents a KL divergence penalty term coefficient; represents a KL divergence.

[0098] wherein, the expression of the KL divergence constraint is:

[0099] ;

[0100] In the formula, represents the probability of the reference policy model generating the i th output question and answer pair under the condition that the i th training sample is given as input; represents the probability of the new policy model currently being optimized generating the i th output question and answer pair under the condition that the same i th training sample is given as input.

[0101] Through the embodiment, the relative advantage is directly calculated by using the reward distribution of the group action, the calculation overhead is reduced, and the calculation efficiency is improved; and by using the designed target function and the KL divergence constraint, the stable performance improvement of the policy model is maintained in fine-tuning, so that the target policy model capable of outputting high-quality question and answer pairs is obtained.

[0102] In some of the embodiments, as shown in Figure 5 , the step S250 of generating target question and answer pairs meeting the reward based on the keyword and the document batch by using the target policy model comprises the following steps:

[0103] Step S251, generating candidate initial question and answer pairs based on the keyword and the document batch by using the target policy model;

[0104] Step S252, screening target question and answer pairs meeting the relevance reward and the format reward in the reward from the candidate initial question and answer pairs.

[0105] Specifically, the target policy model already has the ability to generate high-quality evaluation question and answer pairs. Therefore, the candidate initial question and answer pairs are generated in batches based on the keyword and the document extracted in step S210 by using the target policy model; of course, in order to further improve the quality of the candidate initial question and answer pairs, the patent document can be a new document screened.

[0106] Then, for each candidate initial question and answer pair, a quality screening is performed, and a relevance reward and a format reward of each candidate initial question and answer pair are calculated; only when both the relevance reward and the format reward are satisfied, the candidate initial question and answer pair is considered as the final target question and answer pair, so as to obtain a high-quality evaluation dataset.

[0107] Through the embodiment, the quality of the evaluation dataset is maximally guaranteed, and the construction efficiency of the evaluation dataset is improved.

[0108] The embodiment will be described and explained below through preferred embodiments.

[0109] The RAG system evaluation dataset automatic synthesis method based on reinforcement learning of the preferred embodiment includes the following steps:

[0110] In step S1, knowledge base data of the RAG system is obtained, and the knowledge base data is preprocessed to obtain structured data.

[0111] For example, the knowledge base data includes the following types of documents: 1. Unstructured: a medical research PDF format paper on "the latest progress of hypertension treatment"; 2. Semi-structured: an online clinical guideline for heart disease in HTML format; 3. Structured: a database CSV file containing basic information, evaluation, and medication records of the rescued. The original knowledge base data is preprocessed; for example, the text is extracted from the PDF and the header and footer are removed, the HTML is parsed and the tags are removed, and the missing values in the CSV are processed. The data in different formats is unified into a pure text format. For structured and semi-structured data, keywords can be extracted and organized into easy-to-understand text paragraphs. Then the text in the unified format is cut into blocks. Limited by the context window of the subsequent large language model and to ensure that each text block contains relatively complete semantic information, the text is divided according to a fixed block size (for example, 500 words or 1000 characters), and the division is performed at the boundaries of sentences or paragraphs.

[0112] Therefore, the original diversified knowledge base data is converted into a standardized structured document set composed of a large number of text blocks ; wherein each is a cleaned and cut text segment.

[0113] In step S2, a keyword extraction model is used to extract keywords and related documents of the target vertical field from the structured data.

[0114] For example, the target vertical field is the heart disease diagnosis and treatment field; then in order to generate target question and answer pairs highly related to the heart disease diagnosis and treatment field, keywords representing the core concepts and terminologies of the field need to be extracted. Some or all document blocks are input into the LLM model, and the prompt word template , get the preliminary set of keywords of the domain For example, for the document block containing the hypertension guideline, the extracted keywords can include: "hypertension", "antihypertensive drugs", "diuretics", and "combination therapy", etc.

[0115] Step S3, screen new documents; after getting the set of keywords of the domain, the most relevant new documents to the specific keywords can be screened from the entire preprocessed document set to provide accurate context information for the subsequent question-answer pair generation.

[0116] The documents in the document set are taken as candidate documents, and the relevance score between each candidate document and the keyword is calculated using information retrieval algorithms such as BM25. Then, all candidate documents are sorted according to the relevance score; the top 5 documents are selected from the sorted results to form new documents, which can be represented as Drelated.

[0117] For example: for the keyword "side effects of ACE inhibitors", the new documents screened by the BM25 algorithm can be: literature abstracts about adverse reactions in ACE inhibitor clinical trials; part of the instructions for use about the side effects of ACE inhibitors; clinical notes containing rescuer's discussion about the cough symptoms of the rescued person after taking ACE inhibitors.

[0118] Step S4, according to the keyword and the document, use the preset prompt word template to synthesize the initial question-answer pair;

[0119] The prompt word template can be: "Please focus on the topic of [specific domain keyword], and use the relevant document set [screened relevant document set] as reference documents to synthesize a question about the reference document set, and provide a clear and accurate answer. Please ensure that the question and answer are highly relevant to [specific domain keyword]. Please output the question-answer pair in the following format: <question>[<Question> What is the user's question?] <answer>[Detailed answer to this question]<\Answer>”.

[0120] The specific formula can be formally defined as:

[0121] ;

[0122] wherein, represents the initial question-answer pair synthesized by the policy model (Qwen2.5-7B policy model), represents the prompt word template used for initial question-answer pair synthesis, is the keyword, is the new document after screening.

[0123] Step S5, build the reward of the initial question-answer pair; for each initial question-answer pair generated by the policy model, the relevance reward and the format reward will be built.

[0124] wherein, the relevance reward is: using an independent evaluation model (Evaluator LLM), which can be another large language model, to judge whether the generated answer Answer is related to the question Question, and whether the answer can be found in the new document . Take the question, answer and new document as input to the Evaluator LLM; and set the prompt to output a binary judgment (for example, "relevant" or "irrelevant"). For example, the prompt can be: "Please judge whether the following answer can answer the corresponding question, and whether it can be supported from the provided document. Answer'relevant' or 'irrelevant'. Question: [Question] Answer: [Answer] Document: [D related Content]". If the Evaluator LLM judges "relevant", the relevance reward Relevance_Reward=1 is given; if it is judged "irrelevant", the relevance reward Relevance_Reward=-1 is given.

[0125] wherein, the format reward is: check whether the generated initial question-answer pair strictly follows <question>... and <answer>the format of the...<\Answer> is correct. If the format is correct, a format reward Format_Reward = 1 is given; if the format is incorrect (e.g., the label is missing, does not match, or is redundant), a format reward Format_Reward = -1 is given.

[0126] wherein the total reward Reward of the initial question-answer pair is the sum of the relevance reward and the format reward.

[0127] Step S6, training the initial strategy model to obtain a target strategy model.

[0128] In the collected training data Next, for the keywords of each field , using the current initial strategy model (the parameters are denoted as θ) and the related new documents , an initial question-answer pair is generated . Then, the reward (the sum of the relevance reward and the format reward) of the initial question-answer pair is calculated. In this way, a set of training samples is obtained. The above process is repeated to sample all the training samples.

[0129] Then, based on these training samples, the GRPO algorithm is used for optimization, with a preset target function as the optimization objective, to iteratively update the parameters of the initial strategy model until the convergence condition is met, to obtain the target strategy model. During the updating process, the expected total reward of the advantage function in the target function is maximized; and through the KL divergence constraint, the controllable degree of deviation of the new strategy model from the old strategy model is ensured.

[0130] Step S7, quality screening. The target strategy model is used to batch generate candidate initial question-answer pairs based on the extracted keywords and documents; of course, in order to further improve the quality of the candidate initial question-answer pairs, the documents of the patents can be new documents that have been screened.

[0131] Next, for each candidate initial question-answer pair, the relevance reward and the format reward of each candidate initial question-answer pair are calculated according to the reward of step 5; only when both the relevance reward and the format reward meet the requirements, the candidate initial question-answer pair is considered as the final target question-answer pair, and other subsequent initial question-answer pairs are discarded, so as to obtain a high-quality evaluation data set.

[0132] Through the preferred embodiment, a high-quality evaluation dataset is synthesized for a specific vertical field, ensuring that the synthesized question-answer pairs have good field characteristics and can be widely applied in RAG systems in various industries, improving the diversity of evaluation scenarios; and using a language model to automatically synthesize evaluation questions reduces the cost of manual annotation and improves data construction efficiency; in addition, using a deep reinforcement learning strategy for training can control costs compared to directly calling the api of a closed-source model and outperforms general large language models in data enhancement tasks.

[0133] It should be noted that the steps shown in the above flow or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0134] In this embodiment, an automatic synthesis device for a reinforcement learning-based RAG system evaluation dataset is also provided, which is used to implement the above embodiments and preferred embodiments, which have been described and will not be repeated. The terms "module", "unit", "sub-unit" and the like used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware implementation is also possible and contemplated.

[0135] Figure 6 is a structural block diagram of the reinforcement learning-based RAG system evaluation dataset automatic synthesis device of the present embodiment, as Figure 6 shown, the device comprises an acquisition module 210, a synthesis module 220, a construction module 230, a training module 240 and a processing module 250;

[0136] The acquisition module 210 is configured to acquire the knowledge base data of the RAG system, and extract the keywords and related documents of the target vertical field from the knowledge base data;

[0137] The synthesis module 220 is configured to synthesize an initial question-answer pair using a preset prompt word template according to the keywords and the documents;

[0138] The construction module 230 is configured to construct a reward of the initial question-answer pair;

[0139] The training module 240 is configured to determine training data of an initial strategy model according to the keywords, the initial question-answer pair, the reward and parameters of the initial strategy model; and train the initial strategy model based on the training data to obtain a target strategy model;

[0140] The processing module 250 is configured to use the target strategy model to batch generate target question-answer pairs satisfying the reward based on the keywords and the documents.

[0141] By the above device, the problems of high cost and low efficiency of evaluating the data set in the related art, which affect the effectiveness of the RAG system, are solved. The extracted keywords and related documents are used to conditionally constrain the synthesis process of the initial question-answer pair, to ensure that the synthesized initial question-answer pair is closely related to the knowledge base content. The initial question-answer pair is evaluated in terms of the constructed reward, and the generation ability of the initial strategy model is continuously optimized through the reinforcement learning method to obtain a target strategy model. Finally, the automatic synthesis of a high-quality evaluation data set is realized based on the strategy model, thereby reducing the synthesis cost and improving the synthesis efficiency.

[0142] In some embodiments, the acquisition module 210 is further configured to preprocess the knowledge base data to obtain structured data.

[0143] The keyword extraction model is used to extract keywords and related documents of the target vertical field from the structured data.

[0144] In some embodiments, the preprocessing includes denoising processing, unified data format processing, and chunking processing.

[0145] The keyword extraction model is an LLM model.

[0146] In some embodiments, the RAG system evaluation data set automatic synthesis device based on reinforcement learning includes a screening module.

[0147] The screening module is configured to calculate the relevance score between the candidate documents and the keywords.

[0148] According to the relevance score, all candidate documents are sorted.

[0149] According to the relevance score, all candidate documents are sorted.

[0150] In some embodiments, the construction module 230 is further configured to construct the relevance reward in the reward of the initial question-answer pair based on the evaluation model.

[0151] Based on the forced strategy model, the format reward in the reward of the initial question-answer pair is constructed.

[0152] In some embodiments, the training module 240 is further configured to update the parameters of the initial strategy model based on the training data, with a preset target function as an optimization target, until a convergence condition is met, to obtain a target strategy model.

[0153] In the updating process, the expectation total reward of the advantage function in the target function is maximized, and the deviation degree of the new policy model from the old policy model is controlled through the KL divergence constraint.

[0154] In some embodiments, the processing module 250 is further configured to generate candidate initial question-answer pairs based on the keywords and the document batch using the target policy model.

[0155] The target question-answer pair that meets the relevance reward and the format reward in the reward is selected from the candidate initial question-answer pairs.

[0156] It should be noted that each of the above modules can be a functional module or a program module, which can be implemented by software or hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor, or each of the above modules can be located in different processors in any combination.

[0157] In this embodiment, a computer device is also provided, which includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the above method embodiments.

[0158] Optionally, the computer device can further include a transmission device and an input and output device, wherein the transmission device is connected with the processor, and the input and output device is connected with the processor.

[0159] Optionally, in this embodiment, the processor can be configured to execute the following steps through the computer program:

[0160] S11, knowledge base data of the RAG system is obtained, and keywords and related documents of a target vertical field are extracted from the knowledge base data;

[0161] S12, initial question-answer pairs are synthesized using a preset prompt word template according to the keywords and the documents;

[0162] S13, rewards of the initial question-answer pairs are constructed;

[0163] S14, training data of an initial policy model is determined according to the keywords, the initial question-answer pairs, the rewards, and parameters of the initial policy model, and the initial policy model is trained based on the training data to obtain a target policy model;

[0164] S15, target question-answer pairs that meet the rewards are generated based on the keywords and the document batch using the target policy model.

[0165] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, which will not be described herein again.

[0166] In addition, in combination with the automatic synthesis method of the RAG system evaluation dataset based on reinforcement learning provided in the above-mentioned embodiments, a storage medium can also be provided to implement the same in the present embodiment. The storage medium has a computer program stored thereon; the computer program is executed by a processor to implement any one of the automatic synthesis methods of the RAG system evaluation dataset based on reinforcement learning in the above-mentioned embodiments.

[0167] It should be noted that the information and data involved in the present application are all information and data authorized by the user or fully authorized by all parties, which will be used legally.

[0168] It should be understood that the specific embodiments described herein are intended to explain, not limit, the application. According to the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0169] Obviously, the drawings are only some examples or embodiments of the present application, and those of ordinary skill in the art can also apply the present application to other similar situations according to the drawings without creative labor. In addition, it can be understood that although the work done in the development process may be complex and long, some design, manufacture or production changes made by those of ordinary skill in the art according to the technical content disclosed in the present application are only routine technical means and should not be regarded as insufficient disclosure of the present application.

[0170] The term "embodiment" in the present application means that the specific features, structures or characteristics described in combination with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean independence or alternative to other embodiments. It can be clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0171] The above-described embodiments only express several implementation manners of the present application, which are described in detail and specifically, but should not be understood as limitations on the scope of patent protection. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.< / answer> < / question> < / answer> < / question> < / answer> < / question> ​

Claims

1. A method for automatic synthesis of a reinforcement learning-based RAG system evaluation dataset, comprising the steps of: The method comprises the following steps: obtaining knowledge base data of the RAG system, and extracting keywords and related documents of a target vertical field from the knowledge base data; synthesizing an initial question and answer pair for evaluating the RAG system according to the keywords and the documents by using a preset prompt template; constructing a reward of the initial question and answer pair; the reward comprises a relevance reward and a format reward; the relevance reward is used for verifying whether the answer synthesized by a strategy model is related to the question and answer pair; and the format reward is used for ensuring the correctness of the format of the synthesized question and answer pair; determining training data of an initial strategy model according to the keywords, the initial question and answer pair, the reward, and parameters of the initial strategy model; training the initial strategy model based on the training data to obtain a target strategy model; generating target question and answer pairs satisfying the reward in batches based on the keywords and the documents by using the target strategy model.

2. The method of claim 1, wherein the method further comprises: The method for extracting keywords and related documents of a target vertical field from the knowledge base data comprises the following steps: preprocessing the knowledge base data to obtain structured data; extracting keywords and related documents of a target vertical field from the structured data by using a keyword extraction model.

3. The method of claim 2, wherein the method further comprises: The preprocessing comprises denoising processing, unified data format processing, and chunking processing; the keyword extraction model is an LLM model.

4. The method of claim 1, wherein the method further comprises: The method further comprises the following steps: taking the documents as candidate documents, and calculating a relevance score between the candidate documents and the keywords; sorting all the candidate documents according to the relevance scores; selecting top-ranking candidate documents from the sorting result according to a preset number, and taking the selected candidate documents as new documents.

5. The method of claim 1, wherein the method further comprises: The method for constructing the reward of the initial question and answer pair comprises the following steps: constructing the relevance reward in the reward of the initial question and answer pair based on an evaluation model; constructing the format reward in the reward of the initial question and answer pair based on a forced strategy model.

6. The method of claim 1, wherein the method further comprises: The method for training the initial strategy model based on the training data to obtain a target strategy model comprises the following steps: iteratively updating parameters of the initial strategy model based on a preset target function as an optimization target until a convergence condition is met, so as to obtain the target strategy model; in the updating process, the expectation total reward of an advantage function in the target function is maximized; and a KL divergence constraint is used to ensure that the deviation degree of a new strategy model from an old strategy model is controllable.

7. The reinforcement learning based RAG system evaluation dataset automatic synthesis method according to any one of claims 1 to 6, characterized in that, The method for generating target question and answer pairs satisfying the reward in batches based on the keywords and the documents by using the target strategy model comprises the following steps: generating candidate initial question and answer pairs in batches based on the keywords and the documents by using the target strategy model; selecting target question and answer pairs satisfying the relevance reward and the format reward in the reward from the candidate initial question and answer pairs.

8. A device for automatic synthesis of a reinforcement learning-based RAG system evaluation dataset, characterized by, The method comprises the following steps: an obtaining module, a synthesizing module, a constructing module, a training module, and a processing module; the obtaining module is used for obtaining knowledge base data of the RAG system, and extracting keywords and related documents of a target vertical field from the knowledge base data; The synthesis module is configured to synthesize an initial question-answer pair for evaluating the RAG system according to the keyword and the document by using a preset prompt word template. The construction module is configured to construct a reward of the initial question-answer pair, the reward including a relevance reward and a format reward, the relevance reward being used to verify whether a response synthesized by a policy model is related to the question-answer pair, and the format reward being used to ensure that the synthesized question-answer pair is correct in format. The training module is configured to determine training data of the initial policy model according to the keyword, the initial question-answer pair, the reward, and parameters of the initial policy model. And train the initial policy model based on the training data to obtain a target policy model. The processing module is configured to generate a target question-answer pair satisfying the reward based on the keyword and the document by using the target policy model. 9.A computer device, comprising a memory and a processor, and characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute the steps of the method for automatically synthesizing a reinforcement learning-based RAG system evaluation dataset according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method for automatically synthesizing a reinforcement learning-based RAG system evaluation dataset according to any one of claims 1 to 7.

Images