RAG adaptive evidence refining method based on submodule optimization

Abstract

The invention provides an RAG adaptive evidence refining method based on submodule optimization, and belongs to the technical field of artificial intelligence. Collecting data to construct a knowledge base, and generating a candidate document set according to the knowledge base inquired and retrieved by a user; designing an instruction and calling a DeepSeek model to evaluate the answer contribution degree of the candidate document, calculating the relevancy between the query and the candidate document and the information coverage degree score of the optimal document to the candidate document, and constructing a sub-module objective function; calculating a marginal gain threshold value, iteratively calculating optimal documents with marginal gains greater than the threshold value, and generating an optimal document set; and splitting the optimal document set according to the paragraph level, generating a candidate paragraph set, calculating the redundancy score of the optimal paragraph set, constructing a sub-module objective function added with redundancy penalty by multiplexing the method, and performing iterative calculation to generate a final optimal evidence paragraph subset. The method effectively solves the problems of over-high redundancy, over-low information coverage and the like caused by a traditional processing method, so that the final model reasoning performance is improved.

Description

Technical Field

[0001] This invention proposes a RAG adaptive evidence refinement method based on sub-modulus optimization, which belongs to the field of artificial intelligence technology. Background Technology

[0002] In recent years, large language models (LLMs) have made significant progress in the field of natural language processing, particularly excelling in tasks such as content creation, intelligent dialogue, and knowledge-based question answering. However, their capabilities also have inherent limitations: the knowledge boundary of LLMs is constrained by the update time of the training data, making it difficult for them to acquire the latest information; at the same time, the "illusion" phenomenon is also prevalent, meaning that the model may generate seemingly reasonable but actually inaccurate or completely wrong content. To address these challenges, retrieval-enhanced generation (RAG) technology has emerged. This technique optimizes LLMs by integrating external knowledge sources, enabling them to utilize up-to-date information from real-time or domain-specific sources to generate more accurate answers. This approach significantly improves the accuracy and timeliness of the model's responses and effectively reduces the risk of illusion.

[0003] In typical RAG systems, the retrieval engine usually returns multiple complete documents or fragments relevant to the user's query. However, directly providing these long documents as context to an LLM (Local Level Analyzer) presents several problems: the LLM's context length is limited, unable to accommodate excessive content; the LLM tends to focus on information appearing earlier in the document, potentially ignoring later information containing crucial evidence during reasoning; and redundant and irrelevant information within the documents dilutes key evidence, interfering with the LLM's reasoning process. To address these challenges, researchers have explored various strategies, such as reordering, context compression, and direct LLM extraction. These methods significantly improve the quality and reliability of the generated answers through secondary processing of the retrieved evidence documents.

[0004] Despite these advancements, several issues remain: reordering essentially only calculates the relevance between the question and the document and adjusts the document order accordingly, without directly processing the overall document content, thus only achieving localized optimization; context compression methods inevitably suffer from information loss during compression, and due to the typically one-time and global nature of the process, it is difficult to accurately retain key evidence while reducing content; while methods that write prompts for LLM to extract directly are highly effective, they are limited by response time and token costs, and lack theoretical guarantees. Summary of the Invention

[0005] Objective: To address the shortcomings of existing technologies, this invention proposes an adaptive evidence refinement method for RAG (Research and Acquisition) based on sub-modulus optimization. By treating the optimization of retrieved documents as an optimal subset selection problem, a sub-modulus objective function is constructed. This function comprehensively considers relevance, information coverage, answer contribution, and redundancy. A lazy greedy algorithm is employed to iteratively optimize at both the document and paragraph levels. Adaptive stopping is achieved by setting a marginal gain threshold, rather than the traditional method of fixedly returning k elements, ensuring the output of the optimal evidence subset. This method provides efficient and accurate evidence support for RAG systems, improving the quality of generated answers.

[0006] To achieve the above technical objectives, the RAG adaptive evidence refinement method based on sub-modulus optimization of the present invention comprises the following steps:

[0007] Step 1: Collect relevant documents from the coal mine safety website, clean the data, and build a retrieval knowledge base;

[0008] Step 2: Construct a candidate document pool from the retrieved documents. Calculate the relevance score between candidate documents and the query, and the information coverage score between the selected documents and candidate documents based on the embedding model. Then, call the DeepSeek model to evaluate the contribution score of the candidate document answers and construct the sub-model objective function.

[0009] Step 3: Traverse the document candidate pool, calculate and obtain the candidate document with the largest score that satisfies the submodulus objective function, and add it to the selection set of documents;

[0010] Step 4: Calculate the marginal gain threshold based on the initial marginal gain score, use the threshold as the stopping condition to iteratively calculate the highest marginal gain score using a lazy greedy algorithm, extract the candidate document index corresponding to the high score, add it to the selected documents to generate the optimal document subset;

[0011] Step 5: Write rule expressions to segment the optimal document subset by paragraph level, generating an evidence paragraph candidate pool. Calculate the relevance score between candidate paragraphs and the query, the information coverage score between the selected paragraphs and candidate paragraphs, call the DeepSeek model to evaluate the contribution score of the candidate paragraphs to the answer, and introduce a redundancy penalty function to construct a new sub-model objective function;

[0012] Step 6: Repeat the operation of step 4 for the new function until adaptive termination, and output the final optimal subset of evidence paragraphs.

[0013] Furthermore, relevant documents, including technical reports and regulatory documents, were obtained from the coal mine safety website via web crawler. The original documents underwent data cleaning and preprocessing: useless information and noise were removed using regular expressions; duplicate content was eliminated using hash algorithms; and text normalization was performed using NLP tools. Vector representations were generated using the bge-large-zh-v1.5 embedding model, and a vector index was constructed and stored using Milvus.

[0014] Furthermore, the retrieved documents are treated as a whole to form a document candidate pool, denoted as . The first layer, which optimizes at the document-level granularity, aims to find an optimal subset of documents.

[0015] Based on the embedding model bge-large-zh-v1.5, retrieve query q and each document D. i ∈V d Embedding vector:

[0016] e q =Embed(q),e i =Embed(D i )

[0017] Calculate query q and candidate document D i The correlation score r between them qi The cosine similarity is used for calculation:

[0018] r i =cos(e q ,e i )

[0019] Calculate candidate document D i With selected document D j The information coverage score sim(i,j) between them is calculated using cosine similarity:

[0020] sim(i,j)=cos(e i ,e j )

[0021] Call the DeepSeek model API, write a prompt word template, and let the large model evaluate the contribution score of the current document as evidence to the query answer, limited to [0,1]. Obtain and parse the returned JSON format information as the answer utility weight:

[0022] w i =LLMScore(q,D i )

[0023] The goal of document-level optimization is to maximize information coverage while ensuring that candidate documents are highly relevant to the query, and to consider answer contribution to avoid ignoring documents with low relevance but containing key evidence. A sub-modulus objective function F1(S) is constructed, and the saturation coverage score of the selection set is calculated to comprehensively evaluate relevance, answer contribution, and information coverage.

[0024]

[0025] Where V represents the document candidate pool, which contains all retrieved documents; The min function selects a subset of documents that are currently selected; min limits the selection of content that has high information coverage but is not very relevant to the query.

[0026] Furthermore, compared to the ordinary greedy "full recalculation," the lazy greedy approach only recalculates the marginal gain of a few candidate documents that "currently appear to have the most potential" at each step, while temporarily using the cached results from the previous round for the gain values ​​of the remaining documents. This can reduce the computational load by an order of magnitude without sacrificing optimality. Therefore, this invention utilizes the lazy greedy algorithm to calculate the marginal gain Δ1 of the new document:

[0027]

[0028] Initialize document subset Calculate the value that maximizes the score of the submodulus objective function and add its corresponding document to the empty subset S. Then, iterate through the marginal gain formula to sequentially obtain the optimal candidate document index j that meets the conditions and add it to S.

[0029] j = argmax(Δ1(j∣S)<τ)

[0030] Here, τ is the stopping threshold of the marginal gain. To avoid stopping too early, leading to insufficient information coverage, or stopping too late, leading to excessively high document similarity in the subset, τ is set to 5% of the initial marginal gain value. At this point, the information coverage has approached saturation, and updating S finally yields the optimal subset S of documents with an adaptive number. d .

[0031] Furthermore, regular expressions are written to split the subset obtained in the first stage into paragraph-level segments and add them to the paragraph-level candidate pool:

[0032]

[0033] For V p Each paragraph in the text is recalculated as r. i , sim(i,j), w i A redundancy penalty term is introduced to optimize at a finer-grained paragraph level while maximizing the removal of redundant paragraphs, thereby obtaining the optimal subset S of paragraphs. pConstruct a sub-module objective function F2(S) to comprehensively evaluate the relevance of each paragraph, the contribution of the answer, the information coverage, and the redundancy:

[0034]

[0035] in, The redundancy penalty term is calculated by summing the pairwise cosine similarities within the current optimal segment subset, where μ is the penalty weight.

[0036] Further, the marginal gain Δ2 of the new paragraph is calculated using the lazy greedy algorithm:

[0037] Δ2(j∣S)=F2(S∪{j})-F2(S)

[0038] Initialize paragraph subset Calculate the value that maximizes the submodulus objective function score and add its corresponding document to an empty subset S. Then, iterate through the marginal gain formula, successively adding the optimal candidate paragraph j that satisfies j = argmaxΔ2(j|S) to S. The iteration automatically stops when Δ2(j|S) < τ, at which point the gain score has approached saturation. Update S to ultimately obtain the optimal subset S of evidence paragraphs that satisfies high relevance, high information coverage, high answer contribution, and minimal redundancy. p .

[0039] In summary, this invention offers the following advantages: It optimizes submodules at both the document and paragraph levels, significantly reducing computational complexity and time costs compared to direct paragraph-level optimization, thus achieving a dual improvement in efficiency and effectiveness. Furthermore, this method comprehensively considers multiple dimensions such as relevance, information coverage, answer contribution, and redundancy, ensuring high-quality information retention while significantly simplifying evidence documents. Moreover, by introducing a dynamic threshold mechanism, the function adaptively returns the optimal number of subsets, outperforming traditional methods that fixedly return k documents. The employed lazy greedy algorithm provides an approximate guarantee of 0.632 for the submodule function, thus ensuring theoretical optimization performance under any circumstances. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of a RAG adaptive evidence refinement method based on sub-modulus optimization according to the present invention. Detailed Implementation

[0041] The embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0042] like Figure 1As shown, this invention discloses a RAG adaptive evidence refinement method based on submodule optimization, with the following steps: collecting domain documents and constructing a knowledge base; placing the retrieved documents as a whole into a candidate pool, calculating the relevance between documents and queries and the coverage between documents based on an embedding model, calling LLM to evaluate the contribution of the answer, and constructing a submodule objective function; iteratively selecting the document with the highest objective function score using a lazy greedy algorithm, calculating the marginal gain, and setting a threshold to adaptively generate a subset of documents with optimal relevance and coverage; splitting the selected documents into paragraphs to form a paragraph candidate pool, constructing a submodule objective function with added redundancy penalty, and iteratively selecting the paragraph with the largest marginal gain using a lazy greedy algorithm until the gain is below the threshold, ensuring high relevance and minimizing redundancy, and generating the optimal paragraph subset.

[0043] Specifically, the following steps are included:

[0044] Step 1: Obtain relevant documents, including technical reports and regulatory documents, from the coal mine safety website using a web crawler. Perform data cleaning and preprocessing on the raw documents: remove useless information and noise using regular expressions; eliminate duplicate content using hash algorithms; and perform text normalization using NLP tools. Generate vector representations using the bge-large-zh-v1.5 embedding model, and construct and store vector indexes using Milvus.

[0045] Step 2: Calculate the relevance and information coverage scores between individual documents, evaluate the document's answer contribution score using a large model, and introduce sub-model optimization to construct the sub-model function. The retrieved documents are then used as a whole to form a document candidate pool, denoted as... The first layer, which optimizes at the document-level granularity, aims to find an optimal subset of documents.

[0046] Based on the embedding model bge-large-zh-v1.5, retrieve query q and each document D. i ∈V d Embedding vector:

[0047] e q =Embed(q),e i =Embed(D i )

[0048] Calculate query q and candidate document D i The correlation score r between them qi The cosine similarity is used for calculation:

[0049] r i =cos(e q ,e i )

[0050] Calculate candidate document Di With selected document D j The information coverage score sim(i,j) between them is calculated using cosine similarity:

[0051] sim(i,j)=cos(e i ,e j )

[0052] Call the DeepSeek model API, write a prompt word template, and let the large model evaluate the contribution score of the current document as evidence to the query answer, limited to [0,1]. Obtain and parse the returned JSON format information as the answer utility weight:

[0053] w i =LLMScore(q,D i )

[0054] The goal of document-level optimization is to maximize information coverage while ensuring that candidate documents are highly relevant to the query, and to consider answer contribution to avoid ignoring documents with low relevance but containing key evidence. A sub-modulus objective function F1(s) is constructed, and the saturation coverage score of the selection set is calculated to comprehensively evaluate relevance, answer contribution, and information coverage.

[0055]

[0056] Where V represents the document candidate pool, which contains all retrieved documents; The min function selects a subset of documents that are currently selected; min limits the selection of content that has high information coverage but is not very relevant to the query.

[0057] Step 3: Initialize document subset Calculate the value that maximizes the score of the submodulus objective function:

[0058] j = argmaxF1(S)

[0059] Obtain the corresponding document index and add it to the empty subset S.

[0060] Step 4: Calculate the maximum marginal gain for each new document added based on the calculated submodulus target score, and calculate the minimum threshold to allow the function to adaptively stop iteration to obtain the optimal subset of evidence documents. The marginal gain Δ1 for new documents is calculated using a lazy greedy algorithm.

[0061]

[0062] Then, iterate through the marginal gain formula, sequentially obtaining the optimal candidate document index j that meets the conditions and adding it to S:

[0063] j = argmax(Δ1(j∣S)<τ)

[0064] Here, τ is the stopping threshold of the marginal gain. To avoid stopping too early, which would lead to insufficient information coverage, or stopping too late, which would lead to excessively high document similarity in the subset, τ is set to 5% of the initial marginal gain value, at which point the information coverage has already approached saturation. Updating S finally yields the optimal subset S of documents with an adaptive number. d .

[0065] Step 5: Split the best subset of documents into paragraphs, calculate the relevance and information coverage scores between paragraphs, evaluate the answer contribution scores of each paragraph using a large model, and introduce sub-model optimization at the paragraph level to construct the sub-model function. Write regular expressions to split the subset obtained in the first stage at the paragraph level and add it to the paragraph-level candidate pool:

[0066]

[0067] For V p Each paragraph in the text is recalculated as r. i , sim(i,j), w i A redundancy penalty term is introduced to optimize at a finer-grained paragraph level while maximizing the removal of redundant paragraphs, thereby obtaining the optimal subset S of paragraphs. p Construct a sub-module objective function F2(S) to comprehensively evaluate the relevance of each paragraph, the contribution of the answer, the information coverage, and the redundancy:

[0068]

[0069] in, The redundancy penalty term is calculated by summing the pairwise cosine similarities within the current optimal segment subset, where μ is the penalty weight.

[0070] Step 6: Calculate the maximum marginal gain for each new paragraph added based on the calculated submodulus target score, and calculate the minimum threshold to allow the function to adaptively stop iteration to obtain the optimal subset of evidence paragraphs. Calculate the marginal gain Δ2 of the new paragraph using a lazy greedy algorithm.

[0071] Δ2(j∣S)=F2(S∪{j})-F2(S)

[0072] Initialize paragraph subset Calculate the value that maximizes the submodulus objective function score and add its corresponding document to an empty subset S. Then, iterate through the marginal gain formula, successively adding the optimal candidate paragraph j that satisfies j = argmaxΔ2(j|S) to S. The iteration automatically stops when Δ2(j|S) < τ, at which point the gain score has approached saturation. Update S to ultimately obtain the optimal subset S of evidence paragraphs that satisfies high relevance, high information coverage, high answer contribution, and minimal redundancy. p .

[0073] To demonstrate the effectiveness of this invention, a comparative experiment was conducted with commonly used document retrieval and processing methods. The retrieval method used a BGE dense search engine, the knowledge base constructed in step 1 was used, the default retrieval returned 20 documents, and DeepSeek was used as the inference model.

[0074] This method primarily compares the processing capabilities of evidence documents under the same retrieval results, focusing on the overall quality of the processed evidence documents and the accuracy of the inference answers. Among five methods, this method is compared with document compression methods for the relevance and completeness of the final evidence; with re-ranking methods for the number of processed tokens and the accuracy of the final inference; and with methods that directly call large models for extraction for overall time consumption and the accuracy of the final inference. The results show that this method outperforms in multiple metrics and significantly reduces the number of tokens transmitted to the large model while maintaining high-quality evidence documents.

[0075] The experimental results are shown in Table 1:

[0076] Table 1 Comparison of experimental results between this method and the comparative method.

[0077]

[0078] Among them, MRR measures the ability to rank positive samples first, NDCG measures the ranking quality of search results, taking into account the relevance score and position of relevant documents, Tokens represents the number of document tokens after processing, F1 measures the accuracy of the generated answer, Oringal means no processing, LLMLingua means processing with a context compression tool, Bge-reranker-large means re-ranking, and DeepSeek Extract means direct processing via the model API.

[0079] The experimental results show that this method achieves optimal results across most metrics. By extracting the optimal subset of evidence, this method can significantly reduce document redundancy and context length while avoiding a large amount of critical information, thereby improving the model's reasoning ability.

Claims

1. A RAG adaptive evidence refinement method based on submodular optimization, characterized in that, The steps are as follows: Step 1, collect field documents from the coal mine safety network, and clean the data to build a retrieval knowledge base; Step 2, build the retrieved documents into a candidate document pool, calculate the relevance score of the candidate documents and the query, the information coverage score of the selected documents and the candidate documents, call the DeepSeek model to evaluate the answer contribution score of the candidate documents, and build a sub-module objective function; Step 3, traverse the document candidate pool, calculate the candidate document that meets the maximum score of the sub-module objective function, and add it to the selected document; Step 4, calculate the marginal gain threshold value according to the initial marginal gain score, use the lazy greedy algorithm to iteratively calculate the highest marginal gain score with the threshold value as the stopping condition, extract the candidate document index corresponding to the high score, and add it to the selected document to generate the optimal document subset; Step 5, write a rule expression to split the optimal document subset by paragraph level, generate an evidence paragraph candidate pool, calculate the relevance score of the candidate paragraph and the query, the information coverage score of the selected paragraph and the candidate paragraph, call the DeepSeek model to evaluate the answer contribution score of the candidate paragraph, and introduce a redundancy penalty function to build a new sub-module objective function; Step 6, repeat the operation of step 4 for the new function until the adaptive stop, and output the final optimal evidence paragraph subset.

2. The RAG adaptive evidence refinement method based on submodular optimization of claim 1, wherein, Through network crawler, get the field related documents from the coal mine safety network, including technical reports, regulations and other documents; data cleaning and preprocessing of original documents: remove useless information and noise through regular expression; use hash algorithm to remove duplicate content; use NLP tools for text normalization; use bge-large-zh-v1.5 embedding model to generate vector representation, use Milvus to build vector index and store.

3. The RAG adaptive evidence refinement method based on submodular optimization of claim 1, wherein, The retrieved documents are constituted as a whole to form a document candidate pool, denoted as based on an embedding model bge-large-zh-v1.5 to obtain an embedding vector for the query q and each document D i ∈ V d of the document D e q = Embed(q), e i = Embed(D i ) Calculate query q and candidate document D i The correlation score r between them qi The cosine similarity is used for calculation: r i = cos(e q , e i ) Computing the information coverage score sim(i,j) between the candidate document D i and the selected document D j is computed using the cosine similarity: sim(i,j) = cos(e i ,e j ) Call the DeepSeek model API, write a prompt word template, let the large model evaluate the answer contribution score of the current document as evidence to the query, limited to [0,1]. Get and parse the returned Json format information as the answer utility weight: w i = LLMScore(q, D i ) Build a sub-module objective function F1(S), calculate the saturation coverage score of the selected set, and comprehensively evaluate the relevance, answer contribution, and information coverage: where V denotes the pool of document candidates, containing all retrieved documents, For the currently selected subset of documents, the min processing restriction function selects those with high information coverage but little query relevance.

4. The RAG adaptive evidence refinement method based on submodular optimization of claim 3, wherein, Use the idea of lazy greedy algorithm to calculate the marginal gain of new documents Δ1: Initializing the subset of documents The value satisfying the maximum sub-module objective function score is calculated, and the corresponding document is added to the empty subset S. Then, the marginal gain formula is iterated to add the optimal candidate document index j satisfying the condition to S in turn: j=argmax(Δ1(j∣S)<τ) Wherein τ is the stop threshold of marginal gain, τ takes 5% of the initial marginal gain value to avoid the information coverage being insufficient due to the iteration stopping too early and the subset document similarity being too high due to the iteration stopping too late; at this time, the information coverage has tended to be saturated, and the updating of S finally obtains the adaptive number of optimal document subsets S d .

5. The RAG adaptive evidence refinement method based on submodular optimization of claim 1, wherein, Write a regular expression to split the subset obtained in the first stage by paragraph level, and add it to the paragraph level candidate pool: For each paragraph in V p , recalculate r i , sim(i, j), w i ; introduce a redundancy penalty term to maximize the removal of redundant paragraphs while optimizing at a more granular paragraph level to obtain the optimal paragraph subset S p ; construct a sub-module objective function F2(S) that comprehensively evaluates the relevance of each paragraph, the contribution of the answer, the coverage of the information, and the redundancy: wherein, is a redundancy penalty term, calculated by summing the pairwise cosine similarities within the current optimal subset of passages, and μ is a penalty weight.

6. The RAG adaptive evidence refinement method based on submodular optimization of claim 5, wherein, Use the idea of lazy greedy algorithm to calculate the marginal gain of new paragraphs Δ2: Δ2(j∣S)=F2(S∪{j})-F2(S) Initialize the subset of passages The value satisfying the maximum sub-module target function score is calculated, and the corresponding document is added to the empty subset S. Then the marginal gain formula is iterated to obtain the optimal candidate passage j that satisfies j = argmaxΔ2(j|S) to join S. When Δ2(j|S) < τ, the loop iteration is automatically stopped, at which time the gain score has tended to saturation, and S is updated to obtain the optimal evidence passage subset S p .

Images