Retrieval enhancement generation reasoning method for reinforcement learning and thinking chain based on rules

Abstract

The invention relates to a rule-based reinforcement learning and thinking chain retrieval enhancement generation reasoning method. The method comprises the following steps of: preprocessing data; model training; and performing actual reasoning by adopting the trained model. The method has the beneficial effects that the complex multi-step reasoning capability of a large model is remarkably improved; knowledge integration is carried out, and the efficiency of retrieval enhancement generation (RAG) is optimized; the training cost is reduced; the stability of the strategy is enhanced; dynamic decision support and logic specification constraints are enhanced; the method can perfectly accord with verification requirements for multi-step reasoning ability, and can better adapt to knowledge and language styles in specific fields; the problem that a traditional RAG retrieval result is disjointed with an inference chain is solved, and the capacity of the model for processing complex and multi-step inference tasks is improved.

Description

Technical Field

[0001] This invention belongs to the field of machine learning technology, and in particular relates to a rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method. Background Technology

[0002] Reinforcement learning (RLHF) is an important branch of machine learning, its core being learning optimal decision-making strategies through interaction with the environment. Early applications of RLHF included robot path planning and game AI. AlphaGo's victory over humans in 2016 spurred an explosion in RLHF's popularity, making it a hot technology in the field of artificial intelligence. With the increasing application of RLHF in complex scenarios, traditional algorithms face new challenges in training efficiency, alignment with human intent, and multi-objective balance. To address these challenges, researchers have proposed various innovative optimization frameworks. For example, OpenAI's Proximal Policy Optimization (PPO) algorithm, proposed in 2017, ensures training stability by limiting the magnitude of policy updates. In 2023, the Stanford team proposed the Direct Preference Optimization (DPO) method, which directly optimizes policies based on human preference data, bypassing the reward model modeling step in traditional RLHF. In 2025, DeepSeek's Group Relative Policy Optimization (GRPO) algorithm optimizes policies through relative comparisons of samples within a group, reducing training costs and computational complexity.

[0003] Large language modeling techniques can be traced back to early explorations in sequence modeling. Although recurrent neural networks (RNNs) first achieved text sequence modeling, they struggled to capture long-range semantic dependencies. The Transformer architecture proposed by Vaswani et al. achieved global context modeling through a self-attention mechanism.

[0004] Attention mechanisms are the cornerstone of large language models, efficiently capturing complex dependencies in language. Many advanced large models, such as Qwen and DeepSeek, are based on the Transformer architecture. The Transformer architecture significantly enhances the model's capabilities in long text understanding, multi-step reasoning, and context awareness through self-attention mechanisms. Self-attention allows the model to assign different weights to each element in the input sequence, thereby identifying key information. In rule-based reinforcement learning, attention mechanisms help the model more effectively identify important features in the environmental state, thus making more accurate decisions. For example, DeepSeek-R1's reinforcement learning framework, optimized with attention mechanisms, better simulates the human "insight" process. For Chain of Thought (CoT), attention mechanisms enable the model to progressively focus on and integrate relevant information during reasoning, forming a coherent logical chain. This not only enhances the model's interpretability but also improves its accuracy in solving complex problems. The Qwen model has demonstrated excellent performance in solving mathematical problems and scientific problems. In Retrieval Augmentation (RAG), attention mechanisms are used to connect pre-trained language models with external knowledge bases, enabling effective utilization of rich contextual information. The RAG model supplements the input data by querying relevant documents, and then uses an attention mechanism to integrate the retrieved information with the original input, thereby generating a more accurate and content-rich answer.

[0005] Existing proximal policy optimization (PPO) algorithms face computational and storage challenges with large-scale models. PPO algorithms require constructing a value network commensurate with the policy model size to evaluate the advantage function, leading to significant memory consumption and computational costs. Furthermore, PPO algorithms can cause drastic changes in the policy distribution during policy updates, affecting training stability. To address this, DeepSeek proposes an innovative reinforcement learning algorithm—Group Relative Policy Optimization (GRPO). GRPO reduces training resource consumption by directly using the average reward of multiple sampled outputs as a baseline. GRPO optimizes the policy model by maximizing the following objective function:

[0006]

[0007] Where, π θ and These are the current policy model and the old policy model, respectively. q and O are derived from the problem dataset and the old policy model. The problem and output of mid-sampling, hyperparameters ∈ used to stabilize the training process, π ref This is the reference model, where β is the coefficient of the KL penalty term. It is based on the relative advantage estimation of within-group rewards, and the clip function limits the update magnitude to prevent policy mutation. GRPO estimates the baseline by directly using the output of the reward model, avoiding the need to train complex value functions. Furthermore, GRPO simplifies the training process by directly incorporating the KL divergence between the policy model and the reference model into the loss function for regularization.

[0008] However, existing technologies have the following drawbacks:

[0009] (1) Limited reasoning ability of large models: For example, the 7B series basic models deployed on consumer-grade graphics cards (such as Qwen7B-instruct) are fragile when dealing with complex multi-step reasoning tasks, and are prone to logical confusion or reasoning jumps, making it difficult to cope with problem scenarios that require deep logical chains.

[0010] (2) Insufficient knowledge integration and high time cost of RAG system: Traditional NativeRAG relies on fragmented knowledge bases, the retrieval results are disconnected from the reasoning chain, and relying solely on the knowledge inside the large model is prone to illusion problems, resulting in insufficient accuracy for complex reasoning tasks; RAG involves complex steps such as vector embedding, hybrid retrieval and reordering, which inevitably introduces additional time costs; given that the large model itself is already capable of handling some simple reasoning tasks, in practical applications, the system needs to make a trade-off between efficiency and accuracy to decide whether to enable the RAG mechanism;

[0011] (3) Inefficient reinforcement learning algorithms: The classic PPO algorithm needs to build a value network that matches the scale of the policy model, which leads to high training costs, large memory consumption, and unstable policy updates in large-scale scenarios.

[0012] (4) Limitations of attention mechanism: In the semantic understanding of long texts and multi-step reasoning, the existing model's attention weight allocation mechanism is difficult to accurately capture key information nodes, which affects the coherence of reasoning.

[0013] (5) Lack of dynamic decision support: The existing system lacks a real-time reasoning verification mechanism based on rule constraints. It cannot ensure that each step of reasoning conforms to the preset logical specifications by dynamically adjusting the rule weights, resulting in logical gaps and easily causing the results to deviate from expectations. Summary of the Invention

[0014] The purpose of this invention is to overcome the shortcomings of the prior art and provide a rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method.

[0015] This rule-based reinforcement learning and thought chain-based retrieval-enhanced generative reasoning method includes the following steps:

[0016] Step 1: Data Preprocessing: The GSM8K dataset is used as the benchmark test set. The GSM8K dataset contains approximately 8,800 high-quality elementary school math word problems (7,473 in the training set and 1,319 in the test set). Each problem requires 2 to 8 steps to solve. The Qwen32B model is used to generate CoT data through the CoT inference process on the original question-answer data (original question-answer pairs). A portion of the CoT data is used as training data, and the remaining portion of the CoT data is used as knowledge storage generated by retrieval enhancement and stored in the Milvus vector database.

[0017] Step 2: Train the model;

[0018] Step 2.1: Use 4-bit quantization technology to load the basic model (4-bit quantization technology can improve computational efficiency while reducing memory usage);

[0019] Step 2.2: Use LoRA technology to perform lightweight fine-tuning on the base model (so that it can better adapt to the knowledge and language style of the specific domain), and introduce a low-rank matrix into the fine-tuned base model (the low-rank matrix can effectively reduce the number of training parameters, speed up the convergence speed of the LoRA-tuned base model, and prevent the model from overfitting).

[0020] Step 2.3: Use an optimization algorithm to update the model parameters after introducing the low-rank matrix;

[0021] Step 2.4: Construct a two-layer reward function, which includes a format reward and a result reward. The format reward refers to the standardization of the output format, and the result reward refers to the accuracy of the final answer. The two-layer reward function is used to calculate the reward for the model after updating the model parameters.

[0022] Step 2.5: Use the GRPO algorithm to iteratively calibrate the output of the model after constructing the two-layer reward function and update the model's policy gradient; through the above model training process, the obtained model can more effectively learn the logical reasoning ability required to solve elementary school math word problems, laying a solid foundation for the subsequent reasoning stage; and save the model checkpoint, which is the trained model file;

[0023] Step 3: Perform actual reasoning using the trained model;

[0024] Step 3.1: Make a dialogue question-and-answer request, create or restore the dialogue, extract the question, and load the adapter of the pre-trained model corresponding to the domain of the question; use the pre-trained model to first understand and analyze the input mathematical application problem, and calculate the confidence level of the initial answer;

[0025] Step 3.2: Based on the confidence level of the initial response, output the corresponding response:

[0026] If the confidence level of the initial response is higher than the set threshold (the set threshold is obtained based on result parameter tuning), the initial response is directly output; if the confidence level of the initial response does not exceed the set threshold, search enhancement generation (RAG) is performed to generate the corresponding response output.

[0027] As a preferred option, the Qwen2.5-7B-Instruct model is used as the base model during model training.

[0028] Preferably, the AdamW algorithm is used when training the model.

[0029] Preferably, in step 2.5, the specific method for iteratively calibrating the output of the model after constructing the two-layer reward function using the GRPO algorithm during model training is as follows:

[0030]

[0031] In the above formula, Ri is the reward score, and the higher the score, the better the learning effect. Ri is also the target that needs to be optimized in the GRPO reinforcement learning process; Sf is the format reward, and Sr is the outcome reward; α and β are the coefficients of the format reward Sf and the outcome reward Sr, respectively; clip is a function used to limit the update magnitude. Importance sampling ratio; A i For the content of the reply, A i It is usually a token sequence; π θ For the current model, For the old strategy model, o i This represents the i-th output extracted from the old model; ∈ is the pruning threshold. The values ​​of α and β are related to the target attribute, which refers to, for example, the type of question, such as whether it is for simple mathematical reasoning or complex multi-step reasoning. When answering simple mathematical reasoning questions, α = 0.2 and β = 0.8, the format reward is low and the result reward is high, with more emphasis on the correctness of the result. When answering complex multi-step reasoning questions, α = 0.5 and β = 0.5, the format reward is high and the result reward is moderate, requiring a balance between format guidance and result accuracy.

[0032] As a preferred method, the specific calculation method for the confidence level of the initial response in step 3.1 is as follows:

[0033] Multiple candidate response sequences are generated using a trained model, and the probability of each token is retained. The token refers to the smallest unit in the text, the smallest unit refers to the candidate response sequence, and the token probability refers to the probability value of the smallest unit in the text appearing in a specific context. The output score generated by the trained model includes the unnormalized log probabilities (logits) of each decoding step. The softmax function is used to convert the logits into a probability distribution. The probability value corresponding to the position of each generated token in the probability distribution is obtained.

[0034] The key steps are weighted (to highlight the impact of important reasoning nodes). The key steps include equality, addition, subtraction, multiplication, division, causal inference and transitional inference (i.e. "=", "+", "-", "×", "÷", "so" and "therefore"). The key steps mean that the step plays a key role in the actual reasoning process of the trained model.

[0035] The probability of the token is calculated using a geometric mean:

[0036]

[0037] In the above formula, geom_mean refers to the result of calculating the geometric mean of the token probabilities, n refers to the total number of tokens, i is the index of the current token, and w i p refers to the weight applied to the key step corresponding to the i-th token. i This refers to the probability value of the i-th token appearing in a specific context;

[0038] A low-probability penalty is applied (the percentage of tokens with a probability lower than a set value in the initial response is counted; if this percentage exceeds a certain threshold, the initial response is considered to have uncertainty and the confidence level of the initial response needs to be reduced). The low-probability penalty is a double penalty: in the current inference step, when the percentage of tokens with a probability value lower than 30% exceeds 20% of the total number of tokens, geom_mean = geom_mean * 0.6.

[0039] Perform critical error detection (check whether there are obvious logical errors or content deviations in the candidate's initial response; if so, significantly reduce the confidence of the initial response). If a critical step error occurs, apply an error detection penalty: adjust the weight of the critical step to 0 to 0.1.

[0040] A weighted geometric average is calculated based on the weights applied to the key steps corresponding to the token, the token probability, whether a low-probability penalty is triggered, and whether an error detection penalty is triggered, to obtain the confidence score of the initial response and output it. The specific calculation method for the confidence score of the initial response is as follows:

[0041]

[0042] In the above formula, Confidence refers to the degree of confidence in the response; p refers to the weight applied to the key step corresponding to the i-th token; i I refers to the probability value of the i-th token appearing in a specific context (i.e., the token probability); low_prob This refers to whether a low-probability penalty is triggered, I low_prob The value of I is 0 or 1; error This refers to whether a low-probability penalty is triggered, I error The value can be 0 or 1.

[0043] Preferably, in step 3.2, when the confidence level of the initial response does not exceed the set threshold, the specific method for performing row retrieval enhancement generation (RAG) and generating the corresponding response output is as follows:

[0044] The extracted questions, high-quality CoT data based on the questions, and the corresponding knowledge base vectors are embedded into the Milvus vector database. Similar question-answer pairs with high-quality CoT data are retrieved from the Milvus vector database (question-answer pairs with high-quality CoT data can help the model learn further and improve its reasoning ability). These retrieved question-answer pairs with CoT data are integrated into the context of the initial answer. The trained model is then used for multiple rounds of actual reasoning to gradually generate detailed answer steps. The context of these detailed answer steps is integrated, and the prompts are enhanced. The integrated results are output to the user in a clear and accurate manner. This demonstrates the model's powerful ability in multi-step reasoning by efficiently and accurately solving mathematical application problems in the GSM8K dataset, providing a solid foundation and clear direction for further model optimization and improved reasoning performance.

[0045] Preferably, the confidence threshold for the initial response is set to 0.9.

[0046] The beneficial effects of this invention are:

[0047] This invention significantly improves the complex multi-step reasoning capabilities of large models: by optimizing strategy training through the GRPO algorithm (combining a two-layer reward function and relative advantage estimation), and by combining 4-bit quantization, LoRA lightweight fine-tuning, and other techniques, the model's ability to capture deep logical chains is enhanced; at the same time, the reliability of reasoning is dynamically evaluated based on the confidence calculation of token probability (including key step weighting, low probability penalty, and error detection), effectively solving the problems of logical confusion and reasoning jumps that traditional large models are prone to in complex multi-step reasoning, and improving the accuracy and coherence of solving complex problems.

[0048] This invention integrates knowledge and optimizes the efficiency of Retrieval Augmentation (RAG): it integrates the Milvus vector database to achieve efficient similarity retrieval, and combines a high-quality CoT data fusion mechanism to solve the shortcomings of traditional NativeRAG, such as fragmented knowledge and disconnect between retrieval and reasoning; it dynamically triggers RAG through confidence thresholds (only enabled when confidence is low), which reduces additional computational costs while ensuring accuracy, balances efficiency and effectiveness, avoids the knowledge illusion problem within large models, and improves the success rate of complex reasoning tasks.

[0049] This invention reduces training costs and enhances policy stability: it employs 4-bit quantization to reduce memory usage, LoRA lightweight fine-tuning to reduce the number of parameters and accelerate convergence, and the AdamW optimizer to improve policy update stability; the GRPO algorithm replaces the traditional value network with population relative advantage estimation, saving additional training costs, and combines KL divergence constraints to regularize policy distribution, effectively solving the problems of high training costs and drastic policy update fluctuations in large-scale models of the PPO algorithm, thus improving training efficiency and model robustness.

[0050] This invention also enhances dynamic decision support and logical norm constraints: by using a two-layer reward function (format reward function + result reward function) to guide the model to balance output norms and answer accuracy, and by combining the GRPO policy constraint mechanism, rule-based dynamic decision support is achieved; the confidence calculation incorporates key step weighting and error detection penalties to verify in real time whether the reasoning process conforms to logical norms (such as key nodes such as "=" and "therefore"), avoiding logical breaks, ensuring that the reasoning results strictly conform to the preset norms, and improving output reliability.

[0051] This invention perfectly meets the verification requirements for multi-step reasoning capabilities and can better adapt to the knowledge and language style of specific domains; the model can dynamically select the optimal reasoning path during the reasoning process; it improves the problem of the disconnect between traditional RAG retrieval results and reasoning chains, and enhances the model's ability to handle complex, multi-step reasoning tasks; it can simultaneously calculate the confidence level of the answer when the large model generates the answer for the first time. Attached Figure Description

[0052] Figure 1 The flowchart shows the rule-based reinforcement learning and mind chain retrieval enhancement generative reasoning method of the present invention.

[0053] Figure 2 Flowchart for calculating the confidence level of the initial response;

[0054] Figure 3 The left side shows an example of the original dataset, and the right side shows an example of data containing high-quality CoT inference. Detailed Implementation

[0055] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0056] As one example, such as Figures 1 to 3 As shown, a rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method includes the following steps:

[0057] Step 1: Data Preprocessing: The publicly available citation intent classification dataset GSM8K is used as the benchmark test set. The GSM8K dataset contains approximately 8,800 high-quality elementary school math word problems (7,473 in the training set and 1,319 in the test set). Each problem requires 2 to 8 steps to solve. The original question-answer data (original question-answer pairs) is generated through the CoT inference process using the Qwen32B model to obtain CoT data. A portion of the CoT data is used as training data, and the remaining portion of the CoT data is used as knowledge storage generated by retrieval enhancement and stored in the Milvus vector database.

[0058] Step 2: Select 10 training data points for model training (in this embodiment, only 10 data points were used for training, achieving an accuracy level comparable to training with the full dataset); the parameter settings for model training are as follows: 5 epochs, batch size set to 8, maximum sentence length set to 256, AdamW optimizer used, and learning rate set to 5e. -5 .

[0059] Step 2.1: Use 4-bit quantization technology to load the basic model (i.e., Qwen2.5-7B-Instruct model) (4-bit quantization technology can improve computational efficiency while reducing video memory usage);

[0060] Step 2.2: Use LoRA technology to perform lightweight fine-tuning on the base model (so that it can better adapt to the knowledge and language style of the specific domain), and introduce a low-rank matrix into the fine-tuned base model (the low-rank matrix can effectively reduce the number of training parameters, speed up the convergence speed of the LoRA-tuned base model, and prevent the model from overfitting).

[0061] Step 2.3: Use the AdamW optimization algorithm to update the model parameters after introducing the low-rank matrix;

[0062] Step 2.4: Construct a two-layer reward function, which includes a format reward and a result reward. The format reward refers to the standardization of the output format, and the result reward refers to the accuracy of the final answer. The two-layer reward function is used to calculate the reward for the model after updating the model parameters.

[0063] Step 2.5: The GRPO algorithm is used to iteratively calibrate the output of the model after constructing the two-layer reward function and update the model's policy gradient. Through the above model training process, the obtained model can more effectively learn the logical reasoning ability required to solve elementary school math word problems, laying a solid foundation for the subsequent reasoning stage. Model checkpoints are saved; these checkpoints are the trained model files. The specific method for iteratively calibrating the output of the model after constructing the two-layer reward function using the GRPO algorithm is as follows:

[0064]

[0065] In the above formula, Ri is the reward score, and the higher the score, the better the learning effect. Ri is also the target that needs to be optimized in the GRPO reinforcement learning process; Sf is the format reward, and Sr is the outcome reward; α and β are the coefficients of the format reward Sf and the outcome reward Sr, respectively; clip is a function used to limit the update magnitude. Importance sampling ratio; A i For the content of the reply, A i It is usually a token sequence; π θ For the current model, For the old strategy model, o i This represents the i-th output extracted from the old model; ∈ is the pruning threshold. The values ​​of α and β are related to the target attribute, which refers to, for example, the type of question, such as whether it is for simple mathematical reasoning or complex multi-step reasoning. When answering simple mathematical reasoning questions, α = 0.2 and β = 0.8, with a lower format reward and a higher result reward, focusing more on the correctness of the result. When answering complex multi-step reasoning questions, α = 0.5 and β = 0.5, with a higher format reward and a moderate result reward, requiring a balance between format guidance and result accuracy.

[0066] Step 3: Perform actual reasoning using the trained model;

[0067] Step 3.1: Make a dialogue question-and-answer request, create or restore the dialogue, extract the question, and load the adapter of the pre-trained model corresponding to the domain of the question; select 2000 question-and-answer pairs containing CoT to construct a knowledge base, and use the pre-trained model to first understand and analyze the input mathematical application problem, and calculate the confidence of the initial answer; the specific calculation method for the confidence of the initial answer is as follows:

[0068] Multiple candidate response sequences are generated using a trained model, and the probability of each token is retained. The token refers to the smallest unit in the text, the smallest unit refers to the candidate response sequence, and the token probability refers to the probability value of the smallest unit in the text appearing in a specific context. The output score generated by the trained model includes the unnormalized log probabilities (logits) of each decoding step. The softmax function is used to convert the logits into a probability distribution. The probability value corresponding to the position of each generated token in the probability distribution is obtained.

[0069] The key steps are weighted (to highlight the impact of important reasoning nodes). The key steps include equality, addition, subtraction, multiplication, division, causal inference and transitional inference (i.e. "=", "+", "-", "×", "÷", "so" and "therefore"). The key steps mean that the step plays a key role in the actual reasoning process of the trained model.

[0070] The probability of the token is calculated using a geometric mean:

[0071]

[0072] In the above formula, geom_mean refers to the result of calculating the geometric mean of the token probabilities, n refers to the total number of tokens, i is the index of the current token, and w i p refers to the weight applied to the key step corresponding to the i-th token. i This refers to the probability value of the i-th token appearing in a specific context;

[0073] A low-probability penalty is applied (the percentage of tokens with a probability lower than a set value in the initial response is counted; if this percentage exceeds a certain threshold, the initial response is considered to have uncertainty and the confidence level of the initial response needs to be reduced). The low-probability penalty is a double penalty: in the current inference step, when the percentage of tokens with a probability value lower than 30% exceeds 20% of the total number of tokens, geom_mean = geom_mean * 0.6.

[0074] A Roberta classification pipeline is used for critical error detection (checking whether there are obvious logical errors or content deviations in the candidate's initial response; if so, the confidence of the initial response is significantly reduced). If the return is directly "Error", it is judged as a reasoning error. If a critical step error occurs, an error detection penalty is applied: the weight of the critical step is adjusted to 0 to 0.1.

[0075] A weighted geometric average is calculated based on the weights applied to the key steps corresponding to the token, the token probability, whether a low-probability penalty is triggered, and whether an error detection penalty is triggered, to obtain the confidence score of the initial response and output it. The specific calculation method for the confidence score of the initial response is as follows:

[0076]

[0077] In the above formula, Confidence refers to the degree of confidence in the response; p refers to the weight applied to the key step corresponding to the i-th token; i I refers to the probability value of the i-th token appearing in a specific context (i.e., the token probability); low_prob This refers to whether a low-probability penalty is triggered, I low_prob The value of I is 0 or 1; error This refers to whether a low-probability penalty is triggered, I error The value can be 0 or 1.

[0078] Step 3.2: Based on the confidence level of the initial response, output the corresponding response:

[0079] If the confidence level of the initial answer is higher than a set threshold of 0.9 (the set threshold is obtained based on parameter tuning of the results), the initial answer is directly output. If the confidence level of the initial answer does not exceed the set threshold, Retrieval Augmentation (RAG) is performed to generate the corresponding answer output: the extracted question, high-quality CoT data based on the question, and the knowledge base vector corresponding to the question are embedded into the Milvus vector database. Similar question-answer pairs with high-quality CoT data are retrieved from the Milvus vector database (question-answer pairs with high-quality CoT data can help the model learn further and improve the model's reasoning ability). The retrieved question-answer pairs with CoT data are integrated into the context of the initial answer. Then, the trained model is used for multiple rounds of actual reasoning to gradually generate detailed answer steps. The context of the detailed answer steps is integrated and the prompts are enhanced. The integrated result is output in a clear and accurate manner and presented to the user. This allows for efficient and accurate solving of mathematical application problems in the GSM8K dataset, fully demonstrating the powerful ability of this model in multi-step reasoning. This provides a solid foundation and clarifies the direction for further model optimization and improvement of reasoning performance.

[0080] The accuracy of the proposed method on the test set (GSM8K dataset) was compared with that of the Qwen2.5-7B-Instruct baseline model trained on the entire training set (i.e., 7473 data points). The comparison results are shown in Table 1 below. The proposed method only used 10 data points for training, and compared with training the entire dataset using the baseline model, the proposed method achieved better results.

[0081] Table 1. Accuracy comparison results between the present invention and the Qwen2.5-7B-Instruct baseline model trained on the full training set.

[0082]

[0083] Ablation experiments were conducted, and the reinforcement learning, COT, and RAG modules were removed sequentially. The results were compared with the method of this invention. The results of the ablation experiments are shown in Table 2 below. It can be seen that after removing the RL, COT, and RAG modules sequentially, the effect of each step gradually decreased, which emphasizes the necessity of each step of this invention.

[0084] Table 2 Ablation Experiment Results

[0085]

[0086]

Claims

1. A rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method, characterized in that, Includes the following steps: Step 1, Data Preprocessing: The GSM8K dataset is used as the benchmark test set; The original question-and-answer data is processed by CoT inference to generate CoT data. A portion of the CoT data is used as training data, and the remaining portion of the CoT data is stored in the Milvus vector database as knowledge base generated by retrieval enhancement. Step 2: Train the model; Step 2.1: Use 4-bit quantization technology to load the basic model; Step 2.2: Fine-tune the basic model using LoRA technology, and introduce a low-rank matrix into the fine-tuned basic model; Step 2.3: Use an optimization algorithm to update the model parameters after introducing the low-rank matrix; Step 2.4: Construct a two-layer reward function, which includes a format reward and a result reward; use the two-layer reward function to calculate the reward for the model after updating the model parameters; Step 2.5: Iteratively calibrate the output of the model after constructing the two-layer reward function, update the model's policy gradient, and save the model checkpoints; Step 3: Perform actual reasoning using the trained model; Step 3.1: Make a dialogue question-and-answer request, create or restore the dialogue, extract the question, and load the adapter of the pre-trained model corresponding to the domain of the question; use the pre-trained model to first understand and analyze the input mathematical application problem, and calculate the confidence level of the initial answer; Step 3.2: Based on the confidence level of the initial response, output the corresponding response: If the confidence level of the initial response is higher than the set threshold, the initial response is output directly. If the confidence level of the initial response does not exceed the set threshold, then search enhancement generation is performed to generate the corresponding response output.

2. The rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method according to claim 1, characterized in that: When training the model, the Qwen2.5-7B-Instruct model was used as the base model.

3. The rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method according to claim 2, characterized in that, The optimization algorithm used in training the model is the AdamW algorithm.

4. The rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method according to claim 3, characterized in that, In step 2.5, the specific method for iteratively calibrating the output of the model after constructing the two-layer reward function during model training is as follows: In the above formula, Ri is the reward score, Sf is the format reward, and Sr is the result reward; α and β are the coefficients of the format reward Sf and the result reward Sr, respectively; clip is a function used to limit the update magnitude. Importance sampling ratio; A i The content of the reply; π θ For the current model, For the old strategy model, o i This represents the i-th output extracted from the old model; ∈ is the pruning threshold, and the values ​​of α and β are related to the target attribute.

5. The rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method according to claim 4, characterized in that, The specific calculation method for the confidence level of the initial response in step 3.1 is as follows: Multiple candidate response sequences are generated using a trained model, and the probability of each token is retained. The token refers to the smallest unit in the text, the smallest unit refers to the candidate response sequence, and the token probability refers to the probability value of the smallest unit in the text appearing in a specific context. The output score generated by the trained model includes the unnormalized log probabilities (logits) of each decoding step. The softmax function is used to convert the logits into a probability distribution. The probability value corresponding to the position of each generated token in the probability distribution is obtained. The key steps are weighted, including equality, addition, subtraction, multiplication, division, causal inference, and transition inference; The probability of the token is calculated using a geometric mean: In the above formula, geom_mean refers to the result of calculating the geometric mean of the token probabilities, n refers to the total number of tokens, i is the index of the current token, and w i p refers to the weight applied to the key step corresponding to the i-th token. i This refers to the probability value of the i-th token appearing in a specific context; A low-probability penalty is applied, which is a double penalty: in the current inference step, when the proportion of tokens with a probability value below 30% exceeds 20% of the total number of tokens, geom_mean = geom_mean * 0.6; Perform critical error detection. If a critical step error occurs, apply an error detection penalty: adjust the weight of the critical step to 0 to 0.

1. A weighted geometric average is calculated based on the weights applied to the key steps corresponding to the token, the token probability, whether a low-probability penalty is triggered, and whether an error detection penalty is triggered, to obtain the confidence score of the initial response and output it. The specific calculation method for the confidence score of the initial response is as follows: In the above formula, Confidence refers to the degree of confidence in the response; p refers to the weight applied to the key step corresponding to the i-th token; i This refers to the probability value of the i-th token appearing in a specific context; I low_prob Indicates whether a low-probability penalty is triggered, I low_prob The value of I is 0 or 1; error Indicates whether a low-probability penalty is triggered, I error The value can be 0 or 1.

6. The rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method according to claim 5, characterized in that, In step 3.2, when the confidence level of the initial response does not exceed the set threshold, the specific method for performing row retrieval enhancement generation and generating the corresponding response output is as follows: The extracted question, the CoT data based on the question, and the knowledge base vector corresponding to the question are embedded into the Milvus vector database. Similar question-answer pairs with CoT data are retrieved from the Milvus vector database. The retrieved question-answer pairs with CoT data are fused into the context of the initial answer. Then, the trained model is used to perform multiple rounds of actual reasoning to generate detailed answer steps. The context of the detailed answer steps is integrated and the prompts are enhanced. The integrated result is then output.

7. The rule-based reinforcement learning and thought chain-based retrieval enhancement generative reasoning method according to claim 1, characterized in that: The confidence threshold for the initial response is set at 0.9.

Images