A Large Language Model (RAG) Method and System Based on Fine-Grained Text Retrieval Optimization

Abstract

This invention relates to a large language model (RAG) method and system based on fine-grained retrieval text optimization. The method includes obtaining original question-answer pairs, retrieving several relevant texts using a semantic vector-based retrieval system, constructing a fine-grained retrieval text optimization model, which includes a text quality assessment module, a text optimization module, and a reinforcement learning agent. The text quality assessment module models the information content and distribution structure of the retrieved text, while the text optimization module selects corresponding optimization strategy templates from four types of strategy action signals: competition, backoff, cooperation, and complementarity. The reinforcement learning agent streamlines the optimization process, dynamically selecting and executing the optimal optimization action from the text optimization module to train the fine-grained retrieval text optimization model. The trained fine-grained retrieval text optimization model is then used for retrieval enhancement generation tasks. This invention, through intelligent decision-making text optimization, facilitates the achievement of higher quality and more interpretable and robust retrieval enhancement generation.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to the RAG method and system based on fine-grained text retrieval optimization. Background Technology

[0002] Retrieval-Augmented Generation (RAG) aims to combine the knowledge of external retrieval text with the generative capabilities of large language models to address issues such as "outdated knowledge" and "factual errors" in models. Its applications are widespread, encompassing fields like intelligent question answering, document generation, knowledge graph question answering, and customer service dialogue. However, due to the uncertainty of retrieval text quality and the limitations of large language models in utilizing unstructured text, RAG technology still faces significant challenges in practical implementation. Retrieval text is often affected by corpus noise, retrieval recall bias, and loose text representation, making it difficult for models to accurately extract effective information. Furthermore, traditional RAG often employs a simple process of "directly concatenating retrieval text + generation," without targeted optimization of the retrieval text, further exacerbating redundancy and logical inconsistencies in the generated results, and even leading to factual inconsistencies, severely limiting the performance ceiling of RAG. Existing RAG optimization methods mainly focus on two directions: "retrieval optimization" and "generation strategy adjustment," attempting to alleviate the above problems by improving the quality of retrieval text or refining the generation logic.

[0003] For methods focused on retrieval optimization, the core idea is to improve the relevance between the retrieved text and the query by refining the retrieval algorithm or strategy. Researchers often employ techniques such as semantic vector retrieval, hybrid retrieval (keywords + semantics), and multi-stage retrieval (initial screening + fine ranking) to optimize the "initial quality" of the retrieved text. For example, common semantic retrieval methods based on Sentence-BERT map the query and text to a unified vector space through a pre-trained semantic encoder, using cosine similarity to filter highly relevant text; some methods also introduce a retrieval result re-ranking module (such as Cross-Encoder) to further filter low-relevance text. However, these methods only focus on "filtering high-quality text" and do not address the problem that "high-quality text may still contain information redundancy and scattered evidence"—even if highly relevant text is retrieved, it may contain a large number of sentences unrelated to the answer, or key evidence may be scattered across multiple sentences, causing the model to "fail to grasp the key points" or "lack logical coherence" during generation.

[0004] For methods that focus on adjusting generation strategies, the emphasis is on improving how large language models utilize retrieved text, rather than optimizing the retrieved text itself. For example, some methods employ strategies such as "segmented injection of retrieved text" and "attention-guided generation" to try to improve the model's utilization of retrieved information; other studies have proposed a "retrieval-generation joint optimization" framework, which uses reinforcement learning to allow the model to dynamically adjust its retrieval behavior during the generation process. However, these methods are essentially "passively adapting" to the retrieved text. If the retrieved text itself has problems such as information redundancy and fragmented evidence, adjusting the generation strategy alone cannot fundamentally solve these problems. Moreover, most methods do not consider the need for "different optimization strategies to be adapted to retrieved text of different qualities," and use a uniform processing logic for "low-information-noise text," "high-redundancy text," and "text with scattered evidence," resulting in insufficient robustness of the model in complex scenarios.

[0005] However, existing methods, whether optimizing the retrieval engine or adjusting the generation strategy, have not broken through the limitation of "heavy retrieval / generation, light text optimization": they either only focus on "obtaining high-quality retrieval text" while neglecting the optimization of the text's internal structure and information distribution; or they only passively adapt to the retrieval text during the generation stage, failing to actively perform fine-grained modifications to the retrieval text to match the model's generation requirements. This limitation leads to three core problems for existing RAG systems: first, information redundancy and dilution of key information, with irrelevant sentences in the retrieval text occupying the model's input space, resulting in the dilution of key evidence; second, evidence fragmentation and broken reasoning chains, with core evidence supporting the answer scattered across multiple sentences or texts, making it difficult for the model to integrate them into a coherent reasoning chain; and third, poor scenario adaptability, as the processing strategy cannot be dynamically adjusted according to the quality of the retrieval text, resulting in large fluctuations in generation performance under diverse text scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a large language model (RAG) method and system based on fine-grained text retrieval optimization, which can help achieve higher quality and more interpretable and robust retrieval enhancement generation.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a large language model (RAG) method based on fine-grained text retrieval optimization, comprising the following steps: Step S1: Obtain the original question-answer pair, which contains a question and multiple answers, and retrieve several relevant texts based on the question using a semantic vector-based retrieval tool; Step S2: Construct a fine-grained text retrieval optimization model, which includes a text quality assessment module, a text optimization module, and a reinforcement learning agent; The text quality assessment module is used to model the information content and distribution structure of the retrieved text, and to convert the retrieved text into a state vector as the decision input of the reinforcement learning agent. The text optimization module selects the corresponding optimization strategy template from four types of strategy action signals—competition, backsliding, cooperation, and complementarity—based on the text quality results output by the text quality assessment module, and performs optimization processing on the retrieved text. The reinforcement learning agent formalizes the optimization process using Markov Decision Process (MDP). Based on the state vector output by the text quality assessment module, it dynamically selects and executes the optimal optimization action in the text optimization module to maximize the expected cumulative reward and train the fine-grained retrieval text optimization model. Step S3 uses the trained fine-grained retrieval text optimization model for the retrieval enhancement generation task.

[0008] Further, in step 1), for the obtained original question-answer pairs, a semantic vector-based retrieval tool is used to retrieve relevant text from a specific corpus based on the questions. The retrieved text is then combined with the original question-answer pairs to form a data sample. The set of retrieved texts can be represented as: .

[0009] Furthermore, in step 2), the implementation method of the fine-grained text retrieval optimization model is as follows: A) The retrieved text and corresponding query are sent to the text quality assessment module for processing. The retrieved text is initialized and encoded by a pre-trained semantic encoder. The point mutual information sequence between each sentence and the query is calculated. The temporal dependency relationship of the sequence is modeled using an LSTM encoder. The potential distribution of the point mutual information sequence in the temporal feature space is modeled by combining a variational autoencoder (VAE). Distribution features and statistical features are extracted to construct a state vector for decision-making. B) The agent's policy action signal and the text to be optimized are fed into the text optimization module. Based on the policy action signal output by the reinforcement learning agent, the module calls the optimization policy template corresponding to the policy action signal from the preset unified policy template library. The competitive strategy performs semantic analysis on the text by calling a large language model to filter out sentences with low information content and rewrite them. The fallback strategy calls the knowledge completion interface to supplement the missing information in the output of the competitive strategy. The cooperative strategy integrates scattered evidence into a coherent chain of evidence. The complementary strategy rewrites the chain of evidence into a structured reasoning chain. C) The state vector, the policy action signal of the text optimization module, and the policy candidate set formed after action masking are fed into the reinforcement learning agent. The policy candidate set includes four basic policy actions: competition, backoff, cooperation, and complementarity. The agent formalizes the optimization process using a Markov decision process and trains the policy network of the reinforcement learning agent using generalized advantage estimation and a truncated surrogate loss function. The reward is obtained by weighting multiple text quality indicators, and the policy network is optimized with the goal of maximizing the expected cumulative reward, so that the agent learns the ability to dynamically execute the optimal optimization action based on the state vector. D) During the training process, a step-by-step training strategy is adopted. First, supervised fine-tuning is used to enable the text optimization module to accurately execute various strategies. Then, reinforcement learning is used to optimize the agent's strategy selection logic end-to-end.

[0010] Furthermore, in step A), the specific implementation method of the text quality assessment module is as follows: First, the text quality assessment module accepts a set of retrieved text paragraphs and sentences. and corresponding queries As input, where Indicates the first element in the search text One sentence; To quantify the amount of information a sentence carries relative to the query, each sentence is calculated. relative query Point mutual information, point mutual information It can be represented as: in, For large models based on given sentences Time query The conditional probability estimate, Queries generated for large models The marginal probability estimate, after obtaining the point mutual information of all sentences relative to the query, is formed by assembling all the point mutual information into a variable-length information sequence, which can be expressed as follows: To capture the temporal features of the sequence, the variable-length information sequence is encoded using a Long Short-Term Memory (LSTM) network, outputting a fixed-dimensional hidden state representation. It can be represented as: The hidden state h is input into a variational autoencoder (VAE) for distribution modeling, and the encoder of the VAE outputs latent variables. mean vector ,satisfy: The output of the VAE decoder reconstructing the information sequence is: After reconstructing the information content sequence, the distribution feature set MI-Distribution of VAE is extracted, which can be formally represented as: in, It outputs latent variables The mean vector, The latent semantic vector represents the overall pattern of the text in the mutual information space; KL divergence measures the degree of difference between the latent distribution and the standard normal distribution. This represents the reconstruction error, used to characterize the reliability of the reconstruction. Simultaneously calculate the statistical feature set MI-Stats of the point mutual information sequence: in, It represents the average mutual information value, reflecting the overall information content level; The maximum mutual information value reflects the strength of the strongest information segment in the text. Mutual information variance is used to assess the balance of information distribution. The upper quartile represents the concentration of highly informative regions. The ratio of the position of the maximum value indicates the relative position of key information in the text, which helps to determine the concentration of information; the ratio of information content in the first and second halves measures the difference in the distribution of information in different parts of the text.

[0011] Finally, the text quality assessment module concatenates the distribution features and statistical features into a state vector s: The comma "," represents a vector concatenation operation, and this state vector is used for policy decisions by the reinforcement learning agent.

[0012] Furthermore, in step B), the specific implementation method of the text optimization module is as follows: First, the text optimization module receives the policy action signals output by the reinforcement learning agent. and the search text to be optimized As input, the query corresponding to the search text is also obtained. ; Then, based on the strategy action signal 'a', the corresponding optimized strategy template is retrieved from the preset strategy template library. The optimization strategy template includes fields with a unified structure: strategy task description, problem placeholder {Q}, text set placeholder {D}, and output format {...}. }; Additionally, it includes two optional fields for different strategies: a placeholder for the original text {O} and a placeholder for the intermediate result output {M}; After filling in the corresponding fields, a complete natural language instruction is generated. The task descriptions for the competitive strategy are as follows: "For a specific question, select the most relevant statements from the text set that can be used to answer the question as intermediate outputs; then, integrate these statements into a coherent text according to their original order in the text set." The task description for the backtracking strategy is as follows: "To answer the question, select sentences from the original text that can help the text set respond to the question; use these sentences along with relevant sentences from the text set as intermediate outputs; then, integrate them into a coherent and fluent text according to the original order." The task description for the collaborative strategy is as follows: "Conduct a comprehensive analysis of the given text set to accurately identify and extract core evidence supporting the answer to the question; then, rewrite the core meaning of the core evidence to present it in concise language." The task description for the complementary strategy is as follows: "Based on multiple pieces of evidence provided, construct a complete reasoning chain through logical connections and deductions; then, transform the reasoning chain into a coherent and fluent text and output it." Subsequently, the instruction is input into the large language model after fine-tuning the text optimization task to perform text optimization.

[0013] Furthermore, in step C), the specific implementation method of the reinforcement learning agent is as follows: First, the reinforcement learning agent receives the state vector s output by the text quality assessment module and the policy candidate set A of the text optimization module formed after action masking. The action mask is a concrete expression of predefined policy execution rules, used to constrain the temporal dependencies of policy action calls in the policy candidate set. The design rules of the action mask include: the execution of the fallback policy depends on the execution of the competing policy; and the execution of the complementary policy depends on the execution of the cooperative policy. and strategy candidate set The specific definitions are as follows: in, Corresponding competitive strategies Corresponding rollback strategy, Corresponding cooperation strategy, Corresponding complementary strategies This indicates that the strategy is currently executable. This indicates that the strategy cannot be executed because the dependency rule was not met; These correspond to the strategies of competition, retreat, cooperation, and complementarity, respectively. Secondly, the agent formalizes the text optimization process using a Markov Decision Process (MDP). This requires clearly defining the five elements of Markov decision-making: decision period, system state, action set, transition probability, and reward function. The decision period is the maximum number of steps in the policy execution sequence; the system state is the state vector output by the text quality assessment module. The action set is the policy candidate set A after filtering by action mask; the transition probability is in the first... Step state Next action Then, transferred to the first Step state The conditional probability distribution; the reward function is a weighted calculation of immediate rewards based on three levels of text generation quality indicators, which quantifies the strategy optimization effect. The specific reward function... It can be represented as: in, This is the optimized search text. It is the golden answer. It is word-level precision. It is word-level recall. This is the first level of reward, used to measure the matching degree of long sequences. This is the second-level reward, used to measure 4-gram matching accuracy. This is the third level of reward, used to measure lexical-level matching. , , These are adjustable weight parameters, initially set to 0.4, 0.3, and 0.3. Subsequently, the policy network and core components of the text optimization agent are defined, including the policy network. A two-layer fully connected network is used, with the input layer dimension being the state vector dimension, the hidden layer dimension being 256, and the output layer dimension being the size of the action space. The activation function is... The network output is the probability distribution of each possible strategy in the current state s. ; Generalized advantage estimation is used to calculate the advantage value, which quantifies the greater reward obtained by the current action compared to the average action. Specifically, this can be manifested as: in, For temporal difference error, V(s) is the state-value network. These are the preset GAR parameters; Using a truncated surrogate loss function as the objective function for policy updates helps prevent excessively large policy update magnitudes from causing training instability. This can be expressed as: in, For policy network parameters, The old parameters before the update. It is a predefined cutoff coefficient. This represents the expectation of the trajectory at step t; Finally, the selection logic of the text optimization agent is clarified: for the input state vector s, the output action space is... The probability of choosing each strategy And select the strategy with the highest probability. As the optimal action in the current optimization step, the strategy selection probability is dynamically adjusted to maximize the expected cumulative return. This ensures that the text optimization effect is adapted to search text of different qualities.

[0014] Furthermore, in step D), the specific implementation method of the step-by-step training strategy is as follows: First, a supervised fine-tuning phase combined with LoRA lightweight fine-tuning technology is used to train the text optimization module, enabling it to accurately execute various strategy templates. First, we need to construct a fine-tuning dataset, building a supervised dataset containing a quadruple set of "query-retrieval text-optimized results". , The specific definitions are as follows: Where Q represents the query. { } is the collection of text to be retrieved. These are texts to be optimized under different strategies. It is the result of optimization; The text to be optimized in the competitive strategy is the original search text. The optimization result is achieved by calculating the cosine similarity between each sentence and the concatenated answer to the question, retaining the first 60% of sentences, and rewriting them in their original order using a large model. This can be represented as follows: The text to be optimized using the fallback strategy is based on the competitive strategy, with 20% to 40% of the competitive strategy result sentences removed. Then, it is polished and rewritten by the larger model. The optimized result is the result of the competitive strategy optimization, which can be represented as follows: The text to be optimized in the collaborative strategy is the original search text. The optimization result is generated by instructing a high-performance large model to generate and extract the reasons supporting the answer from each document, which serve as key evidence. This can be represented in the following form: The text to be optimized for complementary strategies is key evidence for multiple cooperative strategies. The optimization result is an evidence causal chain generated by a high-performance large model through instructions, so that the large model can better understand and utilize it. It can be represented in the following form: After generating the optimization results, the average mutual information of the text before and after optimization is calculated, and training samples with an improvement of more than 50% are retained for fine-tuning of the text optimizer. The text optimizer is fine-tuned using the LoRA method, with a pre-trained large language model as the backbone model, and trained using a preset learning rate, training epochs, and batch size. After training, the low-rank adaptive weight parameters generated by the LoRA fine-tuning method are combined with the basic model parameters of the pre-trained large language model to obtain the fine-tuned text optimization module. Based on a predefined reinforcement learning framework, the agent is trained in an end-to-end manner. The fine-tuned text optimization module is used as the "policy executor". It receives the text state vector from the text state evaluator and the policy candidate set obtained after action masking, and collects trajectory data containing "state-action-reward-next state". The policy network is optimized using the PPO algorithm and the surrogate loss function. Parameters are updated after collecting a predetermined number of trajectories, with a predetermined batch size and number of training epochs, and the learning rate is decayed according to a predetermined decay rate. The state value network is simultaneously updated to optimize the advantage value. The accuracy of the estimate.

[0015] A retrieval enhancement generation system based on fine-grained retrieval text optimization includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps of the method described above.

[0016] The beneficial effects of this invention are as follows: By constructing a fine-grained text retrieval optimization framework, this invention collaboratively models the information distribution structure, text optimization strategies, and strategy selection mechanisms of the retrieved text, effectively uncovering the deep relationships between information content, semantic associations, and structural organization within the retrieved text. Through a text quality assessment module based on point mutual information sequences, combined with LSTM and variational autoencoders, the information distribution characteristics of the retrieved text are modeled, characterizing the information structure of the text from both statistical and latent distribution perspectives, providing accurate state representations for subsequent strategy decisions. Furthermore, through explicit modeling of four types of text optimization strategies—competition, backoff, cooperation, and complementarity—and combining a unified strategy template library to achieve differentiated text reconstruction, the system can perform targeted content filtering, information completion, evidence integration, and inference chain construction for retrieved texts of different qualities and structures. Simultaneously, a reinforcement learning agent based on Markov decision processes is introduced. Through dynamic strategy selection and joint optimization using multi-index reward functions, the text optimization process can adaptively adjust the strategy execution order according to the text state, thereby reducing redundancy and noise while ensuring information integrity. Through the synergistic effect of the above mechanisms, this invention can significantly improve the accuracy, structuring degree and reasoning consistency of evidence utilization in the retrieval enhancement generation process, and systematically improve the generation quality, robustness and interpretability of large language models in knowledge-intensive question answering tasks. Attached Figure Description

[0017] Figure 1 ; Detailed Implementation

[0018] The invention will now be further described with reference to the accompanying drawings.

[0019] Please see Figure 1 This invention provides an embodiment: a large language model (RAG) method based on fine-grained text retrieval optimization, comprising the following steps: Step S1: Obtain the original question-answer pairs, which contain questions and multiple answers. Then, use a semantic vector-based retrieval tool to retrieve several relevant texts based on the questions. Combine these with the original question-answer pairs to form an enhanced data sample. This step constructs an initial sample library to be optimized by associating and combining the retrieved texts with the original question-answer pairs. Its core value lies in providing the computational foundation for the subsequent quality assessment module and providing an initial decision-making scenario for the reinforcement learning agent.

[0020] Step S2: Construct a fine-grained text retrieval optimization model, which includes a text quality assessment module, a text optimization module, and a reinforcement learning agent; The text quality assessment module models the information content and distribution structure of the retrieved text, transforming it into a state vector that serves as the decision input for the reinforcement learning agent. By modeling the latent distribution of point mutual information sequences, the text quality assessment module can deeply mine the characteristics of the information content distribution of the retrieved text. This allows the system to keenly perceive subtle differences in text quality, thereby providing the agent with a more representative basis for decision-making.

[0021] The text optimization module, based on the text quality results output by the text quality assessment module, selects corresponding optimization strategy templates from four types of strategy action signals: competition, backoff, cooperation, and complementarity, and performs optimization processing on the retrieved text. The four preset optimization strategies achieve targeted text transformation. The competition strategy filters noise through rewriting; the backoff strategy completes missing information; the cooperation strategy integrates scattered evidence; and the complementarity strategy constructs a logical reasoning chain. This multi-strategy linkage ensures that the retrieved text can be accurately transformed into high-quality generated material.

[0022] The reinforcement learning agent formally optimizes the process using a Markov Decision Process (MDP). Based on the state vector output by the text quality assessment module, it dynamically selects and executes the optimal optimization action from the text optimization module to maximize the expected cumulative reward, thereby training the fine-grained retrieval text optimization model. By introducing an action masking mechanism, the sequential logic of policy execution is explicitly constrained, reducing the search space and improving training efficiency. The agent aims to achieve long-term cumulative rewards and learns to dynamically select the optimal action combination based on text quality, ensuring the system maintains strong robustness even in complex and ever-changing retrieval scenarios.

[0023] Step S3 applies the trained fine-grained retrieval text optimization model to the retrieval enhancement generation task. This step addresses the pain points of traditional methods, such as information redundancy, evidence fragmentation, and logical breaks, by outputting intelligently optimized retrieval text. Ultimately, this enables the large language model to more accurately utilize external knowledge during the generation stage, significantly improving the factual consistency and interpretability of the responses.

[0024] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, in step 1), for the obtained original question-answer pairs, a semantic vector-based retrieval tool is used to retrieve relevant texts from a specific corpus based on the questions. The retrieved texts are then combined with the original question-answer pairs to form a data sample. The set of retrieved texts can be represented as follows: .

[0025] Please continue reading. Figure 1As shown, in one embodiment of the present invention, in step 2), the implementation method of the fine-grained retrieval text optimization model is as follows: A) The retrieved text and corresponding query are sent to the text quality assessment module for processing. The retrieved text is initialized and encoded by a pre-trained semantic encoder. The point mutual information sequence between each sentence and the query is calculated. The temporal dependency relationship of the sequence is modeled using an LSTM encoder. The potential distribution of the point mutual information sequence in the temporal feature space is modeled by combining a variational autoencoder (VAE). Distribution features and statistical features are extracted to construct a state vector for decision-making. B) The agent's policy action signal and the text to be optimized are fed into the text optimization module. Based on the policy action signal output by the reinforcement learning agent, the module calls the optimization policy template corresponding to the policy action signal (competition, backsliding, cooperation, or complementarity) from a preset unified policy template library. The competition strategy performs semantic analysis on the text by calling a large language model to filter out sentences with low information content and rewrite them. The backsliding strategy calls a knowledge completion interface to supplement the missing information in the output of the competition strategy. The cooperation strategy integrates scattered evidence into a coherent chain of evidence. The complementarity strategy rewrites the chain of evidence into a structured reasoning chain. C) The state vector, the policy action signal of the text optimization module, and the policy candidate set formed after action masking are fed into the reinforcement learning agent. The policy candidate set includes four basic policy actions: competition, backoff, cooperation, and complementarity. The agent formalizes the optimization process using a Markov decision process and trains the policy network of the reinforcement learning agent using generalized advantage estimation and a truncated surrogate loss function. The reward is obtained by weighting multiple text quality indicators, and the policy network is optimized with the goal of maximizing the expected cumulative reward, so that the agent learns the ability to dynamically execute the optimal optimization action based on the state vector. D) During the training process, a step-by-step training strategy is adopted. First, supervised fine-tuning is used to enable the text optimization module to accurately execute various strategies. Then, reinforcement learning is used to optimize the agent's strategy selection logic end-to-end.

[0026] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, the specific implementation method of the text quality assessment module in step A) is as follows: First, the text quality assessment module accepts a set of retrieved text paragraphs and sentences. and corresponding queries As input, where Indicates the first element in the search text One sentence; To quantify the amount of information a sentence carries relative to the query, each sentence is calculated. relative query Point mutual information, point mutual information It can be represented as: in, For large models based on given sentences Time query The conditional probability estimate, Queries generated for large models The marginal probability estimate, after obtaining the point mutual information of all sentences relative to the query, is formed by assembling all the point mutual information into a variable-length information sequence, which can be expressed as follows: To capture the temporal features of the sequence, the variable-length information sequence is encoded using a Long Short-Term Memory (LSTM) network, outputting a fixed-dimensional hidden state representation. It can be represented as: The hidden state h is input into a variational autoencoder (VAE) for distribution modeling, and the encoder of the VAE outputs latent variables. mean vector ,satisfy: The output of the VAE decoder reconstructing the information sequence is: After reconstructing the information content sequence, the distribution feature set MI-Distribution of VAE is extracted, which can be formally represented as: in, It outputs latent variables The mean vector, The latent semantic vector represents the overall pattern of the text in the mutual information space; KL divergence measures the degree of difference between the latent distribution and the standard normal distribution. The reconstruction error is represented by the value of the value, which characterizes the reliability of the reconstruction. The Variational Autoencoder (VAE) is a complete model architecture, consisting of an encoder and a decoder. The VAE encoder is the first part of the VAE, responsible for mapping the hidden states to the mean and variance of the latent space. The VAE decoder is the second part of the VAE, responsible for reconstructing the original sequence from the latent variable $z$.

[0027] Simultaneously calculate the statistical feature set MI-Stats of the point mutual information sequence: in, It represents the average mutual information value, reflecting the overall information content level; The maximum mutual information value reflects the strength of the strongest information segment in the text. Mutual information variance is used to assess the balance of information distribution. The upper quartile represents the concentration of highly informative regions. The ratio of the position of the maximum value indicates the relative position of key information in the text, which helps to determine the concentration of information; the ratio of information content in the first and second halves measures the difference in the distribution of information in different parts of the text.

[0028] Finally, the text quality assessment module concatenates the distribution features and statistical features into a state vector s: The comma "," represents a vector concatenation operation, and this state vector is used for policy decisions by the reinforcement learning agent.

[0029] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, the specific implementation method of the text optimization module in step B) is as follows: First, the text optimization module receives the policy action signals output by the reinforcement learning agent. and the search text to be optimized As input, the query corresponding to the search text is also obtained. ; Then, based on the strategy action signal 'a', the corresponding optimized strategy template is retrieved from the preset strategy template library. The optimization strategy template includes fields with a unified structure: strategy task description, problem placeholder {Q}, text set placeholder {D}, and output format {...}. Additionally, it includes two optional fields for different strategies: a placeholder for the original text {O} and a placeholder for the intermediate result output {M}. Filling in the corresponding fields generates a complete natural language instruction and strategy template. The format is as follows: The task descriptions for the competitive strategy are as follows: "For a specific question, select the most relevant statements from the text set that can be used to answer the question as intermediate outputs; then, integrate these statements into a coherent text according to their original order in the text set." The task description for the backtracking strategy is as follows: "To answer the question, select sentences from the original text that can help the text set respond to the question; use these sentences along with relevant sentences from the text set as intermediate outputs; then, integrate them into a coherent and fluent text according to the original order." The task description for the collaborative strategy is as follows: "Conduct a comprehensive analysis of the given text set to accurately identify and extract core evidence supporting the answer to the question; then, rewrite the core meaning of the core evidence to present it in concise language." The task description for the complementary strategy is as follows: "Based on multiple pieces of evidence provided, construct a complete reasoning chain through logical connections and deductions; then, transform the reasoning chain into a coherent and fluent text and output it." Subsequently, the instruction is input into the large language model after fine-tuning the text optimization task to perform text optimization.

[0030] Please continue reading. Figure 1 As shown, in one embodiment of the present invention, in step C), the specific implementation method of the reinforcement learning agent is as follows: First, the reinforcement learning agent receives the state vector s output by the text quality assessment module and the policy candidate set A of the text optimization module formed after action masking. The action mask is a concrete expression of predefined policy execution rules, used to constrain the temporal dependencies of policy action calls in the policy candidate set. The design rules of the action mask include: the execution of the fallback policy depends on the execution of the competing policy; and the execution of the complementary policy depends on the execution of the cooperative policy. and strategy candidate set The specific definitions are as follows: in, Corresponding competitive strategies Corresponding rollback strategy, Corresponding cooperation strategy, Corresponding complementary strategies This indicates that the strategy is currently executable. This indicates that the strategy cannot be executed because the dependency rule was not met; These correspond to competition, backoff, cooperation, and complementarity strategies, respectively. The constraints mentioned above refer to the eligibility of all candidate strategies to be invoked in the action space. The action mask is a vector with the same dimension as the strategy set A, where each bit corresponds to a competition, backoff, cooperation, or complementarity strategy, respectively. By setting the bit of a strategy that does not satisfy the dependency rule (e.g., backoff depends on competition) to 0 (unexecutable), dynamic constraints are implemented on the eligibility of all candidate strategies to be invoked at a specific decision moment.

[0031] Secondly, the agent formalizes the text optimization process using a Markov Decision Process (MDP). This requires clearly defining the five elements of Markov decision-making: decision period, system state, action set, transition probability, and reward function. The decision period is the maximum number of steps in the policy execution sequence; the system state is the state vector output by the text quality assessment module. The action set is the policy candidate set A after filtering by action mask; the transition probability is in the first... Step state Next action Then, transferred to the first Step state The conditional probability distribution; the reward function is a weighted calculation of immediate rewards based on three levels of text generation quality indicators, which quantifies the strategy optimization effect. The specific reward function... It can be represented as: in, This is the optimized search text. It is the golden answer. It is word-level precision. It is word-level recall. This is the first level of reward, used to measure the matching degree of long sequences. This is the second-level reward, used to measure 4-gram matching accuracy. This is the third level of reward, used to measure lexical-level matching. , , These are adjustable weight parameters, initially set to 0.4, 0.3, and 0.3. Subsequently, the policy network and core components of the text optimization agent are defined, including the policy network. A two-layer fully connected network is used, with the input layer dimension being the state vector dimension, the hidden layer dimension being 256, and the output layer dimension being the size of the action space. The activation function is... The network output is the probability distribution of each possible strategy in the current state s. ; Generalized advantage estimation is used to calculate the advantage value, which quantifies the greater reward obtained by the current action compared to the average action. Specifically, this can be manifested as: in, For temporal difference error, V(s) is the state-value network. These are the preset GAR parameters; Using a truncated surrogate loss function as the objective function for policy updates helps prevent excessively large policy update magnitudes from causing training instability. This can be expressed as: in, For policy network parameters, The old parameters before the update. It is a predefined cutoff coefficient. This represents the expectation of the trajectory at step t; Finally, the selection logic of the text optimization agent is clarified: for the input state vector s, the output action space is... The probability of choosing each strategy And select the strategy with the highest probability. As the optimal action in the current optimization step, the strategy selection probability is dynamically adjusted to maximize the expected cumulative return. This ensures that the text optimization effect is adapted to search text of different qualities.

[0032] Furthermore, in step D), the specific implementation method of the step-by-step training strategy is as follows: First, a supervised fine-tuning phase combined with LoRA lightweight fine-tuning technology is used to train the text optimization module, enabling it to accurately execute various strategy templates. First, we need to construct a fine-tuning dataset, building a supervised dataset containing a quadruple set of "query-retrieval text-optimized results". , The specific definitions are as follows: Where Q represents the query. { } is the collection of text to be retrieved. These are texts to be optimized under different strategies. It is the result of optimization; The text to be optimized in the competitive strategy is the original search text. The optimization result is achieved by calculating the cosine similarity between each sentence and the concatenated answer to the question, retaining the first 60% of sentences, and rewriting them in their original order using a large model. This can be represented as follows: The text to be optimized using the fallback strategy is based on the competitive strategy, with 20% to 40% of the competitive strategy result sentences removed. Then, it is polished and rewritten by the larger model. The optimized result is the result of the competitive strategy optimization, which can be represented as follows: The text to be optimized in the collaborative strategy is the original search text. The optimization result is generated by instructing a high-performance large model to generate and extract the reasons supporting the answer from each document, which serve as key evidence. This can be represented in the following form: The text to be optimized for complementary strategies is key evidence for multiple cooperative strategies. The optimization result is an evidence causal chain generated by a high-performance large model through instructions, so that the large model can better understand and utilize it. It can be represented in the following form: After generating the optimization results, the average mutual information of the text before and after optimization is calculated, and training samples with an improvement of more than 50% are retained for fine-tuning of the text optimizer. The text optimizer is fine-tuned using the LoRA method, with a pre-trained large language model as the backbone model, trained with a preset learning rate, training epochs, and batch size. After training, the low-rank adaptive weight parameters generated by the LoRA fine-tuning method are combined with the base model parameters of the pre-trained large language model to obtain the fine-tuned text optimization module. The low-rank adaptive weight parameters generated by the LoRA fine-tuning method are incremental weight parameters introduced additionally into the base model after low-rank decomposition. The base model parameters of the pre-trained large language model refer to the original parameters of the pre-trained large language model before LoRA fine-tuning.

[0033] Based on a predefined reinforcement learning framework, the agent is trained in an end-to-end manner. The fine-tuned text optimization module is used as the "policy executor". It receives the text state vector from the text state evaluator and the policy candidate set obtained after action masking, and collects trajectory data containing "state-action-reward-next state". The policy network is optimized using the PPO algorithm and the surrogate loss function. Parameters are updated after collecting a predetermined number of trajectories, with a predetermined batch size and number of training epochs, and the learning rate is decayed according to a predetermined decay rate. The state value network is simultaneously updated to optimize the advantage value. The accuracy of the estimate.

[0034] A retrieval enhancement generation system based on fine-grained retrieval text optimization includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps of the method described above.

[0035] This invention operates on the following principle: By constructing a closed-loop framework of "text quality assessment - dynamic strategy selection - text optimization execution," it breaks through the limitations of existing Retrieval Augmentation (RAG) algorithms. First, the text quality assessment module quantifies the information content and distribution characteristics of the retrieved text. Then, a reinforcement learning agent dynamically selects four optimization strategies—competition, backsliding, cooperation, and complementarity—based on the text quality. Finally, the text optimization module specifically modifies the retrieved text. This "proactive optimization of retrieved text" approach addresses the quality problem of retrieved text at its source. Simultaneously, the dynamic strategy adaptation mechanism enhances the model's robustness across different text scenarios, ultimately achieving higher-quality, more factually consistent, and interpretable retrieval enhancement generation.

[0036] The above description is only a preferred embodiment of the present invention and should not be construed as a limitation of this application. All equivalent changes and modifications made in accordance with the scope of the patent application of the present invention should be included in the scope of the present invention.

Claims

1. A large language model (RAG) method based on fine-grained text retrieval optimization, characterized in that, Includes the following steps: Step S1: Obtain the original question-answer pair, which contains a question and multiple answers, and retrieve several relevant texts based on the question using a semantic vector-based retrieval tool; Step S2: Construct a fine-grained text retrieval optimization model, which includes a text quality assessment module, a text optimization module, and a reinforcement learning agent; The text quality assessment module is used to model the information content and distribution structure of the retrieved text, and to convert the retrieved text into a state vector as the decision input of the reinforcement learning agent. The text optimization module selects the corresponding optimization strategy template from four types of strategy action signals—competition, backsliding, cooperation, and complementarity—based on the text quality results output by the text quality assessment module, and performs optimization processing on the retrieved text. The reinforcement learning agent formalizes the optimization process using a Markov Decision Process (MDP). Based on the state vector output by the text quality assessment module, it dynamically selects and executes the optimal optimization action in the text optimization module to maximize the expected cumulative reward and train the fine-grained retrieval text optimization model. Step S3 uses the trained fine-grained retrieval text optimization model for the retrieval enhancement generation task.

2. The Large Language Model (RAG) method based on fine-grained text retrieval optimization according to claim 1, characterized in that, In step 1), for the obtained original question-answer pairs, a semantic vector-based retrieval tool is used to retrieve relevant text from a specific corpus based on the questions. The retrieved text is then combined with the original question-answer pairs to form a data sample. The set of retrieved texts can be represented as follows: 。 3. The Large Language Model (RAG) method based on fine-grained text retrieval optimization according to claim 1, characterized in that, In step 2), the implementation method of the fine-grained text retrieval optimization model is as follows: A) The retrieved text and corresponding query are sent to the text quality assessment module for processing. The retrieved text is initialized and encoded by a pre-trained semantic encoder. The point mutual information sequence between each sentence and the query is calculated. The temporal dependency relationship of the sequence is modeled using an LSTM encoder. The potential distribution of the point mutual information sequence in the temporal feature space is modeled by combining a variational autoencoder (VAE). Distribution features and statistical features are extracted to construct a state vector for decision-making. B) The agent's policy action signal and the text to be optimized are fed into the text optimization module. Based on the policy action signal output by the reinforcement learning agent, the module calls the optimization policy template corresponding to the policy action signal from the preset unified policy template library. The competitive strategy performs semantic analysis on the text by calling a large language model to filter out sentences with low information content and rewrite them. The fallback strategy calls the knowledge completion interface to supplement the missing information in the output of the competitive strategy. The cooperative strategy integrates scattered evidence into a coherent chain of evidence. The complementary strategy rewrites the chain of evidence into a structured reasoning chain. C) The state vector, the policy action signal of the text optimization module, and the policy candidate set formed after action masking are fed into the reinforcement learning agent. The policy candidate set includes four basic policy actions: competition, backoff, cooperation, and complementarity. The agent formalizes the optimization process using a Markov decision process and trains the policy network of the reinforcement learning agent using generalized advantage estimation and a truncated surrogate loss function. The reward is obtained by weighting multiple text quality indicators, and the policy network is optimized with the goal of maximizing the expected cumulative reward, so that the agent learns the ability to dynamically execute the optimal optimization action based on the state vector. D) During the training process, a step-by-step training strategy is adopted. First, supervised fine-tuning is used to enable the text optimization module to accurately execute various strategies. Then, reinforcement learning is used to optimize the agent's strategy selection logic end-to-end.

4. The Large Language Model (RAG) method based on fine-grained text retrieval optimization according to claim 3, characterized in that, In step A), the specific implementation method of the text quality assessment module is as follows: First, the text quality assessment module accepts a set of retrieved text paragraphs and sentences. and corresponding queries As input, where Indicates the first element in the search text One sentence; To quantify the amount of information a sentence carries relative to the query, each sentence is calculated. relative query Point mutual information, point mutual information It can be represented as: ； in, For large models based on given sentences Time query The conditional probability estimate, Queries generated for large models The marginal probability estimate, after obtaining the point mutual information of all sentences relative to the query, is formed by assembling all the point mutual information into a variable-length information sequence, which can be expressed as follows: ； To capture the temporal features of the sequence, the variable-length information sequence is encoded using a Long Short-Term Memory (LSTM) network, outputting a fixed-dimensional hidden state representation. It can be represented as: ； The hidden state h is input into a variational autoencoder (VAE) for distribution modeling, and the encoder of the VAE outputs latent variables. mean vector Sum of variance vectors ,satisfy: ； The output of the VAE decoder reconstructing the information sequence is: ； After reconstructing the information sequence, the distribution feature set MI-Distribution of the variational autoencoder (VAE) is extracted, which can be formally represented as: ； in, It outputs latent variables The mean vector, The latent semantic vector represents the overall pattern of the text in the mutual information space; KL divergence measures the degree of difference between the latent distribution and the standard normal distribution. This represents the reconstruction error, used to characterize the reliability of the reconstruction. Simultaneously calculate the statistical feature set MI-Stats of the point mutual information sequence: ； in, It represents the average mutual information value, reflecting the overall information content level; The maximum mutual information value reflects the strength of the strongest information segment in the text. Mutual information variance is used to assess the balance of information distribution. The upper quartile represents the concentration of highly informative regions. The ratio of the position of the maximum value indicates the relative position of key information in the text, which helps to determine the concentration of information; the ratio of information content in the first and second halves measures the difference in the distribution of information in different parts of the text. Finally, the text quality assessment module concatenates the distribution features and statistical features into a state vector s: ； Where "," represents the vector concatenation operation, and the state vector is used for policy decision-making by the reinforcement learning agent.

5. The Large Language Model (RAG) method based on fine-grained text retrieval optimization according to claim 3, characterized in that, In step B), the specific implementation method of the text optimization module is as follows: First, the text optimization module receives the policy action signals output by the reinforcement learning agent. ； and the search text to be optimized As input, the query corresponding to the search text is also obtained. ; Then, based on the strategy action signal 'a', the corresponding optimized strategy template is retrieved from the preset strategy template library. The optimization strategy template includes fields with a unified structure: strategy task description, problem placeholder {Q}, text set placeholder {D}, and output format {...}. The algorithm also includes two optional fields for different strategies: a placeholder for the original text {O} and a placeholder for the intermediate output {M}. After filling in the corresponding fields, a complete natural language instruction is generated. The task description for the competitive strategy is "For a specific question, select the statements that are most relevant to the question and can be used to answer the question from the text set as intermediate outputs; then, integrate these statements into a coherent text according to their original order in the text set." The task description for the backtracking strategy is "To answer the question, select sentences from the original text that can help the text set answer the question; use these sentences together with relevant sentences in the text set as intermediate outputs; then, integrate them into a coherent and fluent text according to the original order of the text." The task description for the collaborative strategy is "To perform a comprehensive analysis of the given text set, accurately identify and extract the core evidence supporting the answer to the question." Subsequently, the core meaning of the core evidence is presented in concise language through rewriting; the task description of the complementary strategy is "based on the provided multiple pieces of evidence, construct a complete reasoning chain through logical association and deduction; then, transform the reasoning chain into a coherent and fluent text and output it"; Subsequently, the instruction is input into the large language model after fine-tuning the text optimization task to perform text optimization.

6. The Large Language Model (RAG) method based on fine-grained text retrieval optimization according to claim 3, characterized in that, In step C), the specific implementation method of the reinforcement learning agent is as follows: First, the reinforcement learning agent receives the state vector s output by the text quality assessment module and the policy candidate set A of the text optimization module formed after action masking. The action mask is a concrete expression of predefined policy execution rules, used to constrain the temporal dependencies of policy action calls in the policy candidate set. The design rules of the action mask include: the execution of the fallback policy depends on the execution of the competing policy; and the execution of the complementary policy depends on the execution of the cooperative policy. and strategy candidate set The specific definitions are as follows: ； ； in, Corresponding competitive strategies Corresponding rollback strategy, Corresponding cooperation strategy, Corresponding complementary strategies This indicates that the strategy is currently executable. This indicates that the strategy cannot be executed because the dependency rule was not met; These correspond to the strategies of competition, retreat, cooperation, and complementarity, respectively. Secondly, the agent formalizes the text optimization process using a Markov Decision Process (MDP). This requires clearly defining the five elements of Markov decision-making: decision period, system state, action set, transition probability, and reward function. The decision period is the maximum number of steps in the policy execution sequence; the system state is the state vector output by the text quality assessment module. The action set is the policy candidate set A after filtering by action mask; the transition probability is in the first... Step state Next action Then, transferred to the first Step state The conditional probability distribution; the reward function is a weighted calculation of immediate rewards based on three levels of text generation quality indicators, which quantifies the strategy optimization effect. The specific reward function... It can be represented as: ； ； ； ； in, This is the optimized search text. It is the golden answer. It is word-level precision. It is word-level recall. This is the first level of reward, used to measure the matching degree of long sequences. This is the second-level reward, used to measure 4-gram matching accuracy. This is the third level of reward, used to measure lexical-level matching. , , These are adjustable weight parameters, initially set to 0.4, 0.3, and 0.

3. Subsequently, the policy network and core components of the text optimization agent are defined, including the policy network. A two-layer fully connected network is used, with the input layer dimension being the state vector dimension, the hidden layer dimension being 256, and the output layer dimension being the size of the action space. The activation function is... The network output is the probability distribution of each possible strategy in the current state s. ; Generalized advantage estimation is used to calculate the advantage value, which quantifies the greater reward obtained by the current action compared to the average action. Specifically, this can be manifested as: ； ； in, For temporal difference error, V(s) is the state-value network. These are the preset GAR parameters; Using a truncated surrogate loss function as the objective function for policy updates avoids excessively large policy update magnitudes that could lead to training instability. This can be expressed as: ； in, For policy network parameters, The old parameters before the update. It is a predefined cutoff coefficient. This represents the expectation of the trajectory at step t; Finally, the selection logic of the text optimization agent is clarified: for the input state vector s, the output action space is... The probability of choosing each strategy And choose the strategy with the highest probability. As the optimal action in the current optimization step, the strategy selection probability is dynamically adjusted to maximize the expected cumulative return. This ensures that the text optimization effect is adapted to search text of different qualities.

7. The Large Language Model (RAG) method based on fine-grained text retrieval optimization according to claim 3, characterized in that, In step D), the specific implementation method of the step-by-step training strategy is as follows: First, a supervised fine-tuning phase combined with LoRA lightweight fine-tuning technology is used to train the text optimization module, enabling it to accurately execute various strategy templates. First, we need to construct a fine-tuning dataset, building a supervised dataset containing a quadruple set of "query-retrieval text-optimized results". , The specific definitions are as follows: ； Where Q represents the query. { } is the collection of text to be retrieved. These are texts to be optimized under different strategies. It is the result of optimization; The text to be optimized in the competitive strategy is the original search text. The optimization result is achieved by calculating the cosine similarity between each sentence and the concatenated answer to the question, retaining the first 60% of sentences, and rewriting them in their original order using a large model. This can be represented as follows: ； ； The text to be optimized using the fallback strategy is based on the competitive strategy, with 20% to 40% of the competitive strategy result sentences removed. Then, it is polished and rewritten by the larger model. The optimized result is the result of the competitive strategy optimization, which can be represented as follows: ； ； The text to be optimized in the collaborative strategy is the original search text. The optimization result is generated by instructing a high-performance large model to generate and extract the reasons supporting the answer from each document, which serve as key evidence. This can be represented in the following form: ； ； The text to be optimized for complementary strategies is key evidence for multiple cooperative strategies. The optimization result is an evidence causal chain generated by a high-performance large model through instructions, so that the large model can better understand and utilize it. It can be represented in the following form: ； ； After generating the optimization results, the average mutual information of the text before and after optimization is calculated, and training samples with an improvement of more than 50% are retained for fine-tuning of the text optimizer. The text optimizer is fine-tuned using the LoRA method, with a pre-trained large language model as the backbone model, and trained using a preset learning rate, training epochs, and batch size. After training, the low-rank adaptive weight parameters generated by the LoRA fine-tuning method are combined with the basic model parameters of the pre-trained large language model to obtain the fine-tuned text optimization module. Based on a predefined reinforcement learning framework, the agent is trained in an end-to-end manner. The fine-tuned text optimization module is used as the "policy executor". It receives the text state vector from the text state evaluator and the policy candidate set obtained after action masking, and collects trajectory data containing "state-action-reward-next state". The policy network is optimized using the PPO algorithm and the surrogate loss function. Parameters are updated after collecting a predetermined number of trajectories, with a predetermined batch size and number of training epochs, and the learning rate is decayed according to a predetermined decay rate. The state value network is simultaneously updated to optimize the advantage value. The accuracy of the estimate.

8. A retrieval enhancement generation system based on fine-grained retrieval text optimization, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-7.

Images