A method and apparatus for fine-tuning off-policy reinforcement learning for large language models
By employing a phased training and hybrid policy constraint mechanism, combined with KL-Convex policy constraints, the problems of low sample utilization and training instability in the reinforcement fine-tuning of large language models are solved, achieving efficient and stable inference performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2026-03-24
- Publication Date
- 2026-07-31
AI Technical Summary
Existing reinforcement fine-tuning methods for large language models suffer from problems such as low sample utilization, difficulty in balancing training efficiency and performance, and lack of a dynamic control mechanism for systematic training. In particular, when using off-policy reinforcement learning, sample overhead and training instability are difficult to solve.
By employing a phased training process and a hybrid policy constraint mechanism, the algorithm combines a hybrid policy proximal optimization algorithm with KL-Convex policy constraints, utilizes historical policy data for gradient training, and switches anchor points during the policy reset phase, thereby achieving stable policy updates and efficient training.
It significantly improves sample utilization and inference performance, reduces training overhead, ensures training stability and incremental performance improvement, and balances sample utilization, inference performance, training efficiency and stability.
Smart Images
Figure CN122491349A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a reinforcement learning method, more particularly to an off-policy reinforcement learning fine-tuning method for large language models, and further to an off-policy reinforcement learning fine-tuning apparatus for performing the off-policy reinforcement learning fine-tuning method for large language models. Background Technology
[0002] With the rapid evolution of the scale and performance of large language models, large language models have achieved breakthrough performance in high-order reasoning tasks such as mathematics and code by leveraging human-like slow thinking. Reinforcement Fine-Tuning (RFT) has become the core path to improve their reasoning ability.
[0003] However, existing reinforcement fine-tuning schemes mainly suffer from the following problems: First, the sample utilization rate is extremely low. The current mainstream RFT still uses a single PPO or GRPO equivalent policy (on-policy) algorithm. Each iteration must discard all the data from the previous round and only retain the newly generated answers of the current policy. This causes the sample inference volume to expand synchronously with the length of the inference chain and the model parameters, and the GPU time and energy consumption costs rise sharply. Second, it is difficult to balance training efficiency and final performance. Even though some existing technologies have attempted to introduce off-policy data reuse methods, although these existing technologies can improve sample efficiency, they lack effective policy constraint mechanisms and staged training designs, resulting in training instability or decreased convergence performance. They cannot achieve optimal performance while ensuring high efficiency. Third, there is a lack of a systematic dynamic control mechanism for training. How the distribution shift caused by off-policy affects policy evolution behavior has not been quantitatively analyzed. Existing technologies lack a systematic management of historical policies and a dynamic adjustment mechanism for constraint anchors. Hyperparameter tuning and stability control lack theoretical basis.
[0004] Therefore, there is an urgent need for a large-scale model reinforcement learning fine-tuning method that can balance sample utilization, inference performance, training efficiency, and stability, so as to simultaneously overcome technical bottlenecks such as sample overhead, performance limits, and training efficiency. Summary of the Invention
[0005] The technical problem this invention aims to solve is to provide a policy-free reinforcement learning fine-tuning method for large language models. This method aims to simultaneously overcome technical bottlenecks such as sample overhead, performance limits, and training efficiency through staged training steps and hybrid policies and their constraint mechanisms, thereby achieving a balance between sample utilization, inference performance, training efficiency, and stability. Furthermore, this invention provides a policy-free reinforcement learning fine-tuning device for executing this method for large language models.
[0006] To address this, the present invention provides a method for fine-tuning off-policy reinforcement learning for large language models, comprising the following steps:
[0007] Step S1, Model Initialization Phase: Load the large language model to be trained into the memory and initialize it as the baseline model; set the training parameters.
[0008] Step S2, the hybrid policy training phase, firstly, new data is generated based on the current policy, and historical data from the policy set is reused to construct a hybrid training batch; then, using the hybrid training batch, gradient training is performed using the hybrid policy proximal optimization algorithm, and the policy is updated in combination with the KL-Convex policy constraint term of the current step.
[0009] Step S3, Policy Reset Phase: When the training reaches the preset policy reset node, the anchor point of the policy constraint is switched from the baseline model to the currently trained policy model, and the historical policy set is cleared.
[0010] Step S4, same policy training phase: Based on the new data generated by the current policy, the same policy near-policy gradient method is used for training, and the policy is updated by combining the KL-Convex policy constraint term of the current step.
[0011] A further improvement of the present invention is that step S1 includes the following sub-steps:
[0012] Step S101: Load the large language model to be trained into memory and initialize it as a baseline model. ;
[0013] Step S102, set the training batch size Proportion of data from the strategy Update data ratio Historical strategy window size and policy reset node ;
[0014] Step S103: Initialize the current policy The strategy model is the baseline model. and initialize the historical strategy set. It is an empty set.
[0015] A further improvement of the present invention is that step S2 includes the following sub-steps:
[0016] Step S201: Sample problems from the dataset and utilize the current strategy. generate One new reply;
[0017] Step S202, from the mixed distribution Mid-sampling, from the historical strategy set reuse The historical replies, weighted according to the new reply. Form mixed training batches;
[0018] Step S203, change the current strategy The response data and log probability data are stored in the historical strategy set. ,like Then discard the oldest strategy;
[0019] Step S204: Using the hybrid training batches, execute the hybrid strategy proximal optimization algorithm. Subgradient training is performed, and the policy is updated by combining the KL-Convex policy constraints of the current step.
[0020] A further improvement of the present invention is that, in step S204, a hybrid strategy proximal optimization algorithm is used during the training of the first... In the next iteration, its objective function The calculation formula is: ,in, Indicates based on mixed distribution The outer mathematical expectation, For strategy sequence number, ; Indicates by strategy Generated state-action pairs The distribution To act according to this state The expected value of the distribution is calculated by sampling from historical trajectory data; Indicates the importance sampling ratio, Indicates the first Strategy for the next iteration Indicates the first Historical strategy during the next iteration; Indicates the policy network parameters; Indicates the truncation mechanism; Indicates the cutoff coefficient; For policy-based In state Take action below The estimated value of the dominance function.
[0021] A further improvement of the present invention lies in the calculation of the objective function. At that time, the strategy The sampling probability weights are set as follows: and all adopted historical strategies The sum of the total sampling probability weights is set to By adjusting the proportion of data from the strategy The value of this parameter dynamically adjusts the degree to which the hybrid strategy proximal optimization algorithm relies on historical experience.
[0022] A further improvement of the present invention is that, in step S204, according to the formula Calculate the KL-Convex policy constraints for the current step. ,in, Indicates based on the current state The mathematical expectation; Denotes KL divergence, Indicates the current strategy Compared with the benchmark model The differences between them Indicates the current strategy Recent historical strategies The differences between them; For balance coefficient, , This indicates the number of training iterations.
[0023] A further improvement of the present invention is that, in step S3, the training objective function formula is reset through a strategy. Complete the strategy reset phase, in which, Indicates the policy network parameters; Indicates the value network parameters. This represents the loss of the value function in the PPO strategy; Represents the entropy regularization term; , and These represent hyperparameters, This represents the loss coefficient of the additive function. Represents the entropy regularity coefficient. Indicates the constraint coefficient of the KL-Convex policy; Indicates the number of training iterations; This indicates the number of steps required to trigger a policy reset.
[0024] A further improvement of the present invention is that, when At that time, the reference base model will be changed from the initial baseline model. Switch to the current training up to the th Step-by-step strategy model This modifies the reference object of the KL-Convex policy constraint terms and switches the optimization method from the hybrid policy proximal optimization algorithm to the corresponding proximal policy gradient method.
[0025] A further improvement of the present invention is that step S4 includes the following sub-steps:
[0026] Step S401: Sample problems from the dataset and utilize the current strategy. generate One new reply;
[0027] Step S402: Construct a training batch using the new response and train it using the same policy near-end policy gradient method;
[0028] Step S403, according to the switched strategy model And KL-Convex policy constraints, based on the objective function according to The strategy for updating the form in real time.
[0029] The present invention also provides an off-policy reinforcement learning fine-tuning device for large language models, comprising at least one processor and at least one memory, wherein the memory is used to store the large language model, and the processor is used to execute the off-policy reinforcement learning fine-tuning method for large language models as described above.
[0030] Compared with existing technologies, the beneficial effects of this invention are as follows: it supports multiple near-policy gradient methods with the same policy, is applicable to mainstream algorithms such as PPO and GRPO, and has good versatility and scalability; furthermore, in step S2, by reusing historical policy set data in the hybrid policy training phase and combining multiple gradient updates, it significantly improves sample utilization and inference performance, effectively reducing training overhead while ensuring inference capability; on this basis, in step S3, it achieves staged training through a policy reset phase, using off-policy data to accelerate in the early stage and relying on the same policy to ensure convergence in the later stage, thereby effectively improving training efficiency; and during the training process, it uses KL-Convex policy constraints to convexly combine the benchmark model and the most recent historical policy as dual anchor points, and can further dynamically adjust the weights with the number of training steps to achieve a balance between maintaining basic capabilities and policy evolution, effectively ensuring the stability of training and its gradual performance improvement, and well balancing the sample utilization, inference performance, training efficiency, and stability of the overall technical solution. Attached Figure Description
[0031] Figure 1 This is a schematic diagram of the workflow of one embodiment of the present invention;
[0032] Figure 2 This is a comparative diagram verifying the beneficial effects of one embodiment of the present invention;
[0033] Figure 3 This is a performance and efficiency comparison chart of one embodiment of the present invention. Detailed Implementation
[0034] In the description of this invention, the term "several" means one or more; the term "multiple" means two or more; the terms "greater than," "less than," and "exceeding" are all understood to exclude the stated number; and the terms "above," "below," and "within" are all understood to include the stated number. The terms "first," "second," etc., are understood to be used only to distinguish identical or similar technical feature names, and should not be construed as implying / indicating the relative importance of the technical features, the number of technical features, or the sequential relationship between the technical features.
[0035] Before detailing the invention, its application areas and scenarios will be explained. This invention is applicable to the field of large language model technology, particularly for enhancing and fine-tuning the reasoning capabilities of large language models. It is mainly applied in the following scenarios: 1. Enhancing and fine-tuning the reasoning performance of large language models in academic research; 2. Low-cost same-policy iteration and knowledge updating in generative dialogue systems; 3. Efficient reuse of historical data and optimization of computing power in large-scale model training.
[0036] The preferred embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.
[0037] like Figure 1 As shown, this embodiment provides a fine-tuning method for off-policy reinforcement learning for large language models, including the following steps:
[0038] Step S1, Model Initialization Phase: Load the large language model to be trained into the memory and initialize it as the baseline model; set the training parameters.
[0039] Step S2, the hybrid policy training phase, firstly, new data is generated based on the current policy, and historical data from the policy set is reused to construct a hybrid training batch; then, using the hybrid training batch, gradient training is performed using the hybrid policy proximal optimization algorithm, and the policy is updated in combination with the KL-Convex policy constraint term of the current step.
[0040] Step S3, Policy Reset Phase: When the training reaches the preset policy reset node, the anchor point of the policy constraint is switched from the baseline model to the currently trained policy model, and the historical policy set is cleared.
[0041] Step S4, same-policy training phase: Based on the new data generated by the current policy, the same-policy near-end policy gradient method corresponding to the hybrid policy near-end optimization algorithm described in step S2 is used for training, that is, gradient training is performed, and the policy is updated in combination with the KL-Convex policy constraint term of the current step.
[0042] The off-policy reinforcement learning fine-tuning method proposed in this embodiment, also known as Reincarnating Mix-policy Proximal Policy Optimization (R-MPPG), is a method for resetting the hybrid policy proximal policy. The main inventive aspects of this off-policy reinforcement learning fine-tuning method lie in the interaction / cooperation of the Mix-Policy Proximal Policy Gradient (MPPG) algorithm, KL-Convex policy constraints, and a policy reset mechanism, aiming to achieve efficient, stable, and progressive optimization of large language models. In this embodiment, the Mix-Policy Proximal Policy Gradient refers to the Mix-Policy Proximal Policy Gradient (MPPG). The policy reset mechanism refers to the policy reset phase, also known as the policy regeneration mechanism. The same-policy proximal policy gradient method is also called the same-policy algorithm, such as PPO and GRPO.
[0043] Preferably, in this embodiment, the large language model to be trained refers to a pre-trained model or supervised fine-tuning (SFT) model that needs to be fine-tuned by reinforcement learning, such as Qwen-1.5B or Qwen-7B; the memory refers to a hardware storage medium, such as RAM, video memory or hard disk.
[0044] Preferably, step S1 in this embodiment includes the following sub-steps:
[0045] Step S101: Load the large language model to be trained into memory and initialize it as a baseline model. ;
[0046] Step S102, set the training batch size Proportion of data from the strategy Update data ratio Historical strategy window size and policy reset node ;
[0047] Step S103: Initialize the current policy The strategy model is the baseline model. and initialize the historical strategy set. It is an empty set.
[0048] Existing proximal policy optimization methods (PPO, GRPO, etc.) perform well in the fine-tuning of large language models, but their same-policy characteristics lead to extremely low sample utilization efficiency. To address this technical problem, this embodiment, based on a general proximal policy gradient (PPG) architecture (such as PPO or GRPO), draws on research results in off-policy reinforcement learning and innovatively reconstructs the objective function with the support of generalized proximal gradient theory. This extends the improvement limit for same-policy scenarios to off-policy scenarios, enabling the reuse of historical trajectory data while maintaining training stability and improving sample efficiency. Furthermore, this embodiment also achieves the synergistic utilization of historical and current policy data through MPPG introduced in step S2; that is, it uses historical data for off-policy updates and current policy data for same-policy updates.
[0049] Preferably, step S2 in this embodiment includes the following sub-steps:
[0050] Step S201: Sample problems from the dataset and utilize the current strategy. generate One new reply;
[0051] Step S202, from the mixed distribution (Also known as mixed strategy distribution) Sampling from the historical strategy set reuse The historical replies, weighted according to the new reply. The training batches are composed of mixed batches; that is, the proportion of historical responses is... Current strategy The proportion of new responses generated by the same policy proximal policy gradient method is ;
[0052] Step S203, change the current strategy The response data and log probability data (logprob data) are stored in the historical strategy set. ,like Then discard the oldest strategy;
[0053] Step S204: Using the hybrid training batches, execute the hybrid strategy proximal optimization algorithm. Subgradient training is performed, and the policy is updated by combining the KL-Convex policy constraints of the current step.
[0054] Specifically, MPPG in the training... In the next iteration, the current strategy will no longer be used alone. Instead of using the generated data for training, it comes from a mixed distribution. Mid-sampled data, this mixed distribution It also includes the current strategy. and historical strategy The generated trajectory.
[0055] Preferably, in step S204 of this embodiment, a hybrid strategy proximal optimization algorithm is used during the training of the first... In the next iteration, its objective function The calculation formula is: ,in, Indicates based on mixed distribution The outer mathematical expectation, For strategy sequence number, and During training, the most recent The trajectory generated by a historical strategy (when) (The time is the trajectory of the current strategy). Indicates by strategy Generated state-action pairs The distribution To act according to this state The expected value of the distribution is calculated by sampling from historical trajectory data; Indicates the importance sampling ratio, Indicates the first Strategy for the next iteration Indicates the first Historical strategy during the next iteration; Indicates the policy network parameters; Indicates the truncation mechanism; This represents the cutoff factor, which can be preset to 0.2. In practical applications, it can be adjusted according to the actual situation and needs. For policy-based In state Take action below The advantage function estimate is used to measure how well or poorly an action performs relative to the average expected return of the current strategy.
[0056] It is worth noting that, in order to maintain an appropriate proportion of data with the same strategy and avoid instability caused by excessive deviation from the current distribution, thus balancing efficiency and performance, the sampling ratio in this embodiment is determined by parameters. Control. Specifically, parameters. Determines the distribution of mixed strategies Weight allocation: in calculating the objective function At that time, the strategy The sampling probability weights are set as follows: and all adopted historical strategies The sum of the total sampling probability weights is set to By adjusting the proportion of data from the strategy The value of this parameter dynamically adjusts the dependence of the hybrid strategy proximal optimization algorithm on historical experience, thereby achieving a balance between exploring new strategies and utilizing old experience. By default, this embodiment sets the proportion of the strategy data. The default value is 0.4, but it can be adjusted according to actual conditions and needs in practical applications.
[0057] To further improve sample efficiency, this embodiment also introduces an enhanced Update-To-Data Ratio (UTD Ratio) based on MPPG. Specifically, the UTD Ratio is set to a preset parameter for the update data ratio. Then, it will be executed on the same batch of training data obtained from sampling. Sub-gradient updates reduce the reliance on newly generated samples. This parameter can be set and adjusted according to the actual situation. In this embodiment, the default preset value is 2.
[0058] In existing reinforcement fine-tuning methods, a common practice is to use KL constraints to limit the current policy and the baseline model. The deviation from the target policy is designed to maintain basic capabilities. However, this static constraint is too rigid, limiting the model's adaptation to new tasks and knowledge, and making it difficult to cope with the dynamic evolution of policy distribution, thus leading to underoptimization and affecting performance. To address this, this invention proposes KL-Convex Policy Constraint (KLC), which introduces both the base model and the most recent historical policy as anchor points and combines them convexly, thereby dynamically adapting to the policy evolution process while maintaining basic capabilities. This method no longer relies solely on the base model... Instead of using it as the sole anchor point, it simultaneously incorporates the most recent historical strategy. And then perform a convex combination of the two.
[0059] Preferably, this embodiment includes steps S204 and S4 in each training iteration, according to the formula. Calculate the KL-Convex policy constraints for the current step. ,in, Indicates based on the current state The mathematical expectation; Denotes KL divergence, Indicates the current strategy Compared with the benchmark model The differences between them Indicates the current strategy Recent historical strategies The differences between them; The balancing factor is used to weigh the consistency of the base model against the consistency of the recent policy, and it decays dynamically with the number of training steps t: , This indicates the number of training iterations. This embodiment uses this dual-anchoring method, employing KL-Convex policy constraints, to ensure that the policy distribution is consistent with the baseline model. Maintaining close proximity preserves the model's generalizability and prevents the catastrophic loss of core capabilities; on the other hand, it dynamically adapts to recent historical strategies. This allows the model to gradually accumulate and integrate new knowledge while maintaining its stability.
[0060] Step S3 in this embodiment is used to implement policy reincarnation.
[0061] To overcome the bias that may be introduced by the Hybrid Policy Proximal Optimization (MPPG) method during early training using off-policy data, which could affect the final convergence performance, this invention proposes a Policy Reincarnation mechanism. The goal of this mechanism is to seamlessly combine the advantages of off-policy and same-policy methods during training: in the early training phase, the MPPG method fully utilizes historical data to accelerate model performance improvement; in the later phase, it switches to a stable same-policy method to ensure the model has asymptotic convergence capability.
[0062] Specifically, combining hybrid policy proximal optimization and KL-Convex policy constraints, the training process of this invention includes two stages: the first stage ( The initial policy model performs gradient updates for a predetermined number of steps T according to the MPPG algorithm to quickly improve policy performance; the second stage ( During the reset phase (also known as the rebirth phase), two key changes occur in the training settings: the reference base model is changed from the initial baseline model... (Also known as supervised fine-tuning model) Switch to the current training up to the th Step-by-step strategy model This modifies the reference object of the KL constraint terms and switches the optimization method from MPPG to a standard same-policy optimization method, such as PPO or GRPO.
[0063] Under this design, taking PPO as an example, in step S3 of this embodiment, the training objective function formula is reset through a strategy. Complete the strategy reset phase, in which, Indicates the policy network parameters; Indicates the value network parameters. This term represents the loss of the value function in the PPO strategy. If GRPO is chosen as the optimization method for the same strategy, this term does not exist. Represents the entropy regularization term; , and These represent hyperparameters, This represents the loss coefficient of the additive function. Represents the entropy regularity coefficient. Indicates the constraint coefficient of the KL-Convex policy; Indicates the number of training iterations; This indicates the number of steps required to trigger a policy reset.
[0064] Therefore, in this embodiment, when At that time, the reference base model will be changed from the initial baseline model. Switch to the current training up to the th Step-by-step strategy model This modifies the reference object of the KL-Convex policy constraint terms and switches the optimization method from the hybrid policy proximal optimization algorithm to the same policy proximal policy gradient method.
[0065] This embodiment introduces a policy reset mechanism, which fully leverages the efficiency of off-policy methods in the early stages of training, while relying on the stability of same-policy methods in later stages to achieve incremental performance improvement, thus achieving a balance between training efficiency and final performance. Therefore, this embodiment not only avoids the convergence performance degradation problem that may be caused by simple off-policy methods, but also overcomes the bottleneck of low sample utilization in traditional same-policy methods, ultimately forming an efficient technical solution suitable for fine-tuning large-scale language models.
[0066] Preferably, step S4 in this embodiment includes the following sub-steps:
[0067] Step S401: Sample problems from the dataset and utilize the current strategy. generate One new reply;
[0068] Step S402: Construct a training batch using the new response and train it using the same policy near-end policy gradient method (gradient training).
[0069] Step S403, according to the switched strategy model And KL-Convex policy constraints, KL-Convex policy constraints refer to The calculation formula can be found in step S204 above, based on the objective function. according to For details on the time-based update strategy, please refer to step S3.
[0070] Therefore, it is clear that the steps in this embodiment are interactive and related.
[0071] This embodiment is based on the objective function. The pseudocode for its overall implementation process is shown below. By default, the historical data window is set to... =2, the node for policy reset. The default value is 50 or 100. This embodiment uses PPO and GRPO as the same-policy proximal policy gradient methods, which are referred to as algorithms. The models R-MPPG-PPO and R-MPPG-GRPO were obtained through training.
[0072] 1: [Input]: Base model And a proximal policy gradient method with the same policy. (e.g., PPO, GRPO)
[0073] 2: Set the training batch size Proportion of data from strategy UTD ratio (i.e., the ratio of updated data) Historical strategy window size Strategy to reset nodes
[0074] 3: Initialize the model = Historical strategy collection =
[0075] 4: #Phase 1: Hybrid Strategy Proximal Policy Gradient Training
[0076] 5: for steps t = 1,2,3,…,T do
[0077] 6: From the dataset Sampling problem Total (1- Article, using the current strategy And Algorithm Generate new reply
[0078] 7: From historical collections Medium reuse The old reply forms a mixed training batch.
[0079] 8: Change the current strategy The responses and logprob (log probability) data are stored in the historical strategy set. If | |> Then discard the oldest strategy.
[0080] 9: Using this hybrid batch processing method in conjunction with MPPG, the KL-Convex constraints of the current step, and the algorithm. According to the loss function according to Repeated updates in time format common Second-rate
[0081] 10: end for
[0082] 11: # Phase Switching: Policy Reset
[0083] 12: Reset the base reference model from Switch to strategy model And clear the historical strategy set.
[0084] 13: # Phase 2: Reset to gradient training of proximal policies with the same policy
[0085] 14: for steps = +1, +2, +3,… do
[0086] 15: From the dataset Sampling problem common Article, using the current strategy And Algorithm Generate new reply
[0087] 16: Use these new responses to construct training batches and process them according to the algorithm. According to the loss function, the KL-Convex constraint terms of the current step are used in accordance with the loss function. according to Time-based format update
[0088] 17: end for
[0089] In the pseudocode above, lines 1 to 3 implement step S1 of this embodiment; lines 4 to 10 implement step S2; lines 11 to 13 implement step S3; and lines 14 to 17 implement step S4.
[0090] This embodiment also provides an off-policy reinforcement learning fine-tuning device for large language models, including at least one processor and at least one memory. The memory is used to store the large language model, and the processor is used to execute the off-policy reinforcement learning fine-tuning method for large language models as described above.
[0091] Below, we will combine Figure 2 , Figure 3The beneficial effects of this embodiment will be explained in detail using the experimental data in Tables 1 and 2.
[0092] Table 1: Validation Table of Beneficial Effects on Model 1.5B
[0093] Model AIME25 AMC23 MATH500 Minerva Olympiad Average score R1-Distilled-Qwen-1.5B 33.33 43.37 67.40 16.54 27.26 37.58 Open-RS1 23.33 42.17 64.20 16.18 27.11 34.60 Open-RS2 16.67 45.78 65.00 18.38 26.96 34.56 Open-RS3 16.67 44.58 67.60 15.64 25.48 33.99 AdaptThink 13.33 57.83 78.60 23.90 38.07 42.35 II-Thought 26.67 56.63 73.00 23.16 40.89 44.07 FASTCuRL-preview 26.67 60.24 74.20 20.22 32.59 42.78 FASTCuRL-V3 36.67 66.27 84.40 28.67 43.56 51.91 L1-Exact 23.33 71.08 84.00 29.41 44.59 50.48 L1-Max 20.00 69.88 83.00 29.04 46.37 49.66 DeepScaleR 40.00 65.06 83.20 29.04 43.41 52.14 GRPO 36.67 61.45 80.00 25.37 39.70 48.64 R-MPPG-GRPO 33.33 65.06 84.60 26.10 43.55 50.53 PPO 30.00 69.88 84.00 25.74 43.41 50.61 R-MPPG -PPO 36.67 69.88 82.00 30.15 41.78 52.10
[0094] Table 2: Validation Table of Beneficial Effects on the 7B Model
[0095] Model AIME25 AMC23 MATH500 Minerva Olympiad Average score R1-Distilled-Qwen-7B 33.33 68.68 83.80 30.15 44.44 52.08 ReasonFlux-F1 20.00 54.22 77.20 29.04 37.04 43.50 Light-R1 30.00 66.27 87.00 34.56 47.56 53.08 Skywork-OR1-Preview 43.33 63.86 84.40 29.41 46.22 53.44 Polaris 40.00 63.86 87.60 36.40 48.00 55.17 AdaptThink 46.67 75.90 87.60 33.46 50.22 58.77 AceReason-Nemotron 60.00 80.72 89.00 36.40 50.07 63.24 PPO 53.33 78.31 87.00 34.19 48.88 60.34 R-MPPG -PPO 63.33 78.31 90.20 37.50 52.59 64.39
[0096] Depend on Figure 2 , Figure 3 As can be seen from Tables 1 and 2, this embodiment has achieved the following technical effects:
[0097] First, it supports multiple same-policy reinforcement learning algorithms: This embodiment has good versatility and scalability, and is applicable to a variety of mainstream same-policy algorithms, such as PPO and GRPO. Tables 1 and 2 show the accuracy evaluation of five mathematical inference benchmarks. The method R-MPPG in this embodiment can achieve effective improvement when combined with PPO or GRPO.
[0098] Specifically, on the 1.5B model, as shown in Table 1, this embodiment achieves an average score of 50.53 when combining GRPO (R-MPPG-GRPO), which is better than the baseline GRPO score of 48.64; and an average score of 52.10 when combining PPO (R-MPPG-PPO), which is better than the baseline PPO score of 50.61 and surpasses DeepScaleR's score of 52.14. On the 7B model, as shown in Table 2, this embodiment achieves an average score of 64.39 when combining PPO (R-MPPG-PPO), which is significantly better than the baseline PPO score of 60.34 and surpasses AceReason-Nemotron's score of 63.24.
[0099] Secondly, it possesses interpretability and theoretical support: This embodiment, based on the organic combination of same-policy and dissimilar-policy theories in reinforcement learning, proposes a unified optimization objective and constraint method, providing a theoretical guarantee for the convergence and adaptability of the algorithm. This theoretical interpretability enhances the reliability and generalizability of the method.
[0100] Third, efficient data utilization for high-performance, low-cost training: This embodiment leverages MPPG and a high UTD ratio to fully utilize both historical and current policy data, significantly improving data utilization efficiency and overcoming the shortcomings of existing methods with the same policy, such as low sample efficiency and high computational cost. Breakthroughs have also been achieved in both efficiency and performance. Figure 2The performance and efficiency of this embodiment are compared with other publicly available models (reflected by the total amount of sampled data; the smaller the amount, the higher the efficiency). The number of training steps for R-MPPG is indicated in parentheses to compare its performance at different number of steps. Figure 2 As can be seen, R-MPPG consistently ranks in the ideal upper-left region on the efficiency-performance coordinate system when compared with other publicly available models. On the 1.5B model, as shown in Table 1, R-MPPG achieves an average Pass@1 accuracy of 52.10% with only 79,000 inference samples, comparable to the best baseline DeepScaleR, while reducing the required inference data by more than 30 times. On the 7B model, as shown in Table 2, R-MPPG-PPO achieves 63.27% and 64.39% accuracy with only 7,000 and 11,000 inference samples, respectively, surpassing AceReason-Nemotron's 63.24%, which requires over 3,584,000 inference samples, demonstrating a data efficiency advantage of up to 450 times. Therefore, this embodiment significantly reduces training overhead while maintaining inference capabilities.
[0101] Fourth, overcoming the trade-off between efficiency and performance: This embodiment effectively solves the limitations of existing methods in balancing efficiency and performance, significantly reducing computational overhead while maintaining performance. Experimental results, as shown in Table 1, demonstrate that on the 1.5B model, R-MPPG-PPO achieves 50.61% accuracy in 100 steps, surpassing PPO's performance at 900 steps; as shown in Table 2, on the 7B model, R-MPPG-PPO achieves 63.27% accuracy in 50 steps, significantly better than PPO's 60.34% at 200 steps, with a 4-6 times improvement in training speed. This fully demonstrates that this embodiment can achieve superior performance at extremely low computational cost. Figure 3 In this study, the training efficiency of R-MPPG combined with PPO was compared with that of PPO in three dimensions: total inference data volume, number of training steps, and training time. While achieving the same 40% accuracy, R-MPPG combined with PPO was 4-6 times faster than PPO.
[0102] Fifth, improving training stability: Although policy-based methods have data efficiency advantages, they often suffer from training instability or even decreased convergence performance due to distribution differences. This embodiment introduces KL-Convex constraints and a policy reset mechanism to fully utilize historical data to improve efficiency in the early stages while smoothly transitioning to training with the same policy in the later stages, ensuring training stability and gradual performance improvement, thereby achieving a balance between efficiency and final accuracy. During training, this embodiment uses KL-Convex policy constraints to convexly combine the baseline model and the most recent historical policy as dual anchor points, and dynamically adjusts the weights with the number of training steps, thus achieving a balance between maintaining basic capabilities and policy evolution. It demonstrates stable performance improvements on multiple inference tasks such as AIME25, AMC23, MATH500, Minerva, and Olympiad.
[0103] Therefore, it is evident that this embodiment can effectively balance the sample utilization rate, inference performance, training efficiency, and stability of the overall technical solution.
[0104] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A fine-tuning method for off-policy reinforcement learning for large language models, characterized in that, Includes the following steps: Step S1, Model Initialization Phase: Load the large language model to be trained into the memory and initialize it as the baseline model; set the training parameters. Step S2, the hybrid policy training phase, firstly, new data is generated based on the current policy, and historical data from the policy set is reused to construct a hybrid training batch; then, using the hybrid training batch, gradient training is performed using the hybrid policy proximal optimization algorithm, and the policy is updated in combination with the KL-Convex policy constraint term of the current step. Step S3, Policy Reset Phase: When the training reaches the preset policy reset node, the anchor point of the policy constraint is switched from the baseline model to the currently trained policy model, and the historical policy set is cleared. Step S4, same policy training phase: Based on the new data generated by the current policy, the same policy near-policy gradient method is used for training, and the policy is updated by combining the KL-Convex policy constraint term of the current step.
2. The off-policy reinforcement learning fine-tuning method for large language models according to claim 1, characterized in that, Step S1 includes the following sub-steps: Step S101: Load the large language model to be trained into memory and initialize it as a baseline model. ; Step S102, set the training batch size Proportion of data from the strategy Update data ratio Historical strategy window size and policy reset node ; Step S103: Initialize the current policy The strategy model is the baseline model. and initialize the historical strategy set. It is an empty set.
3. The off-policy reinforcement learning fine-tuning method for large language models according to claim 1 or 2, characterized in that, Step S2 includes the following sub-steps: Step S201: Sample problems from the dataset and utilize the current strategy. generate One new reply; Step S202, from the mixed distribution Mid-sampling, from historical strategy set reuse The historical replies, weighted according to the new reply. Form mixed training batches; Step S203, change the current strategy The response data and log probability data are stored in the historical strategy set. ,like Then discard the oldest strategy; Step S204: Using the hybrid training batches, execute the hybrid strategy proximal optimization algorithm. Subgradient training is performed, and the policy is updated by combining the KL-Convex policy constraints of the current step.
4. The off-policy reinforcement learning fine-tuning method for large language models according to claim 3, characterized in that, In step S204, a hybrid strategy proximal optimization algorithm is used during training. In the next iteration, its objective function The calculation formula is: ,in, Indicates based on mixed distribution The outer mathematical expectation, For strategy sequence number, ; Indicates by strategy Generated state-action pairs The distribution To act according to this state The expected value of the distribution is calculated by sampling from historical trajectory data; Indicates the importance sampling ratio, Indicates the first Strategy for the next iteration Indicates the first Historical strategy during the next iteration; Indicates the policy network parameters; Indicates the truncation mechanism; Indicates the cutoff coefficient; For policy-based In state Take action below The estimated value of the dominance function.
5. The off-policy reinforcement learning fine-tuning method for large language models according to claim 4, characterized in that, In calculating the objective function At that time, the strategy The sampling probability weights are set as follows: and all adopted historical strategies The sum of the total sampling probability weights is set to By adjusting the proportion of data from the strategy The value of this parameter dynamically adjusts the degree to which the hybrid strategy proximal optimization algorithm relies on historical experience.
6. The off-policy reinforcement learning fine-tuning method for large language models according to claim 3, characterized in that, In step S204, according to the formula Calculate the KL-Convex policy constraints for the current step. ,in, Indicates based on the current state The mathematical expectation; Denotes KL divergence, Indicates the current strategy Compared with the benchmark model The differences between them Indicates the current strategy Recent historical strategies The differences between them; For balance coefficient, , This indicates the number of training iterations.
7. The off-policy reinforcement learning fine-tuning method for large language models according to claim 4, characterized in that, In step S3, the training objective function formula is reset using a strategy. Complete the strategy reset phase, in which, Indicates the policy network parameters; Indicates the value network parameters. This represents the loss of the value function in the PPO strategy; Represents the entropy regularization term; , and These represent hyperparameters, This represents the loss coefficient of the additive function. Represents the entropy regularity coefficient. Indicates the constraint coefficient of the KL-Convex policy; Indicates the number of training iterations; This indicates the number of steps that trigger a policy reset.
8. The off-policy reinforcement learning fine-tuning method for large language models according to claim 7, characterized in that, when At that time, the reference base model will be changed from the initial baseline model. Switch to the current training up to the th Step-by-step strategy model This modifies the reference object of the KL-Convex policy constraint terms and switches the optimization method from the hybrid policy proximal optimization algorithm to the same policy proximal policy gradient method.
9. The off-policy reinforcement learning fine-tuning method for large language models according to claim 7, characterized in that, Step S4 includes the following sub-steps: Step S401: Sample problems from the dataset and utilize the current strategy. generate One new reply; Step S402: Construct a training batch using the new response and train it using the same policy near-end policy gradient method; Step S403, according to the switched strategy model And KL-Convex policy constraints, based on the objective function according to The strategy for updating the form in real time.
10. A fine-tuning device for off-policy reinforcement learning for large language models, characterized in that, It includes at least one processor and at least one memory, the memory being used to store a large language model, and the processor being used to execute the off-policy reinforcement learning fine-tuning method for a large language model as described in any one of claims 1 to 9.