Unified supervision fine tuning and reinforcement learning training method based on dynamic weight fusion

Abstract

The invention provides a unified supervision fine tuning and reinforcement learning training method based on dynamic weight fusion, and belongs to the field of artificial intelligence. According to the unified supervision fine tuning and reinforcement learning training method based on dynamic weight fusion, knowledge imitation and strategy exploration are balanced based on unified SFT and RL training targets, and therefore the purpose of improving training stability is achieved. Comprising the following implementation steps: step (1), data preparation; step (2), performing double-path parallel processing; step (2.1), an SFT path is established; (2.2) an RL path; step (3), a dynamic weight fusion mechanism; the loss weights of the SFT path and the RL path are dynamically adjusted through the global coefficient mu, and progressive transition from imitation learning to exploration learning is achieved.

Description

Technical Field

[0001] This application proposes a method for training optimization of Large Language Models (LMMs), specifically unifying Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) within the same training framework, which belongs to the field of artificial intelligence. Background Technology

[0002] As AI technology is increasingly applied to various innovative fields both domestically and internationally, numerous training and optimization methods for large language models are emerging.

[0003] Currently, large language model training commonly employs a two-stage "SFT-then-RL" approach, which has become a mainstream training method recognized and practiced by teams in both industry and academia. Implementation methods include...

[0004] Phase 1: Supervised Fine-Tuning (SFT): This involves collecting high-quality expert data, typically containing thousands to millions of (Prompt, Response) pairs, where the Prompt represents user input and the Response represents the standard answer written by the expert. By mimicking the expert data (Prompt-Response pairs), the model learns to imitate the expert's output style, knowledge representation, and reasoning ability, thus establishing basic task capabilities.

[0005] The second stage: Reinforcement Learning (RL): In the reinforcement learning stage, the large model is guided by reward signals to generate answers that are more in line with human preferences, optimize its generation strategy, and improve the usefulness, harmlessness and authenticity of the answers.

[0006] Although the "SFT-then-RL" paradigm has been widely adopted, it still suffers from the following technical drawbacks: 1) Hard switching problem: There is a sudden change in the objective function between the SFT and RL stages, switching directly from minimizing the negative log-likelihood of the response data to policy gradient optimization. This forces the model to readjust to the new optimization objective in the RL stage, causing performance fluctuations; 2) Overfitting risk: The RL stage may overfit the reward signal, ignoring the general knowledge learned in the SFT stage; 3) Training instability: The objectives of SFT and RL conflict, leading to an imbalance between imitation and exploration in the model. That is, SFT encourages imitation of expert behavior, while RL encourages exploration of new policies.

[0007] In view of the above, this patent application is hereby filed. Summary of the Invention

[0008] This application aims to address the problems existing in the prior art by proposing a unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion. It balances knowledge imitation and policy exploration based on unified SFT and RL training objectives, thereby improving training stability.

[0009] To achieve the above design objectives, the unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion, using a dual-path parallel architecture of SFT and RL, implements the unified integration and smooth transition of supervised fine-tuning and reinforcement learning through a dynamic weight fusion mechanism; it includes the following implementation steps:

[0010] Step (1), Data preparation;

[0011] This includes SFT data preparation and RL data preparation, each involving sampling several question-answer pairs from the dataset;

[0012] Step (2): Parallel processing along two paths;

[0013] Step (2.1), SFT path;

[0014] The process includes inputting data to process SFT data. The process involves inputting the SFT data into a pre-trained large language model and calculating the SFT loss of the model's generated response based on minimizing the negative log-likelihood of the response data.

[0015] Step (2.2) RL path;

[0016] The process includes inputting data to process RL data. The processing flow is as follows: for each input prompt, the large language model generates a total of K candidate responses; each candidate response is evaluated using rule-based rewards to calculate a reward value; and reinforcement learning loss is calculated based on the GRPO algorithm.

[0017] Step (3), dynamic weight fusion mechanism;

[0018] By dynamically adjusting the loss weights of the SFT and RL paths using global coefficients μ, a gradual transition from imitation learning to exploratory learning can be achieved.

[0019] Furthermore, the expression for calculating SFT_loss in step (2.1) is as follows:

[0020] ;

[0021] Among them, the weighting function It is based on the model's current confidence level in the token. ; The parameter is Large language models For conditional probability, given the cue word x and all previous real tokens, the large language model predicts the probability of the next real token to dynamically adjust the weight of the loss for each token; its purpose is to adjust the weight of the loss for tokens that the model is already confident about ( (close to 0 or 1) A small value means it's okay if it doesn't learn, preventing model overfitting; for tokens with uncertain models ( (close to 0.5) A large value means that the model should focus on learning it; The parameter is Large language models Let be the conditional probability, given the cue x and all previous real tokens, the probability that the large language model predicts the next real token; log is the natural logarithm of the probability. For each token position t in the standard answer of a data set, calculate its log probability, and then sum them to obtain the total loss of the data set; To perform the above operation on each data i in a batch and sum up the losses, we obtain the total loss of a batch, where B is the number of batches; "-" is a negative sign, representing minimizing the loss function; This represents a normalization operation, which divides the total loss of the batch by the total number of tokens for all standard answers in the batch.

[0022] Furthermore, in step (2.2), the input data only includes the data for the question (Prompt), and the standard answer is only used to calculate the rule-based reward.

[0023] Furthermore, the reward value in step (2.2) is evaluated based on the following rules:

[0024] A positive reward of 1.1 will be given for demonstrating the thought process and providing a correct answer;

[0025] For answers that only provide the correct answer without any thought process, a positive reward of 0.1 will be given.

[0026] A neutral reward of 0.0 will be given for answers that include a thought process but are incorrect.

[0027] Those who neither demonstrate a thought process nor provide an correct answer will be penalized with -1.

[0028] Further, in step (2.2), the reinforcement learning loss is calculated using the following formula:

[0029] ;

[0030] Where B represents the quantity indicated in a batch; The advantage of each candidate response is calculated using the following expression: , The original reward obtained for a candidate response. , The mean and standard deviation of the group rewards. It is a small constant added to ensure numerical stability; token-level importance sampling ratio. This is used to compare the preference of the new strategy and the old strategy for generating the next token; given the prompt word x and the historical tokens that have already been generated. Under the given conditions, the old strategy generates the next token. The probability is The clip mechanism limits the importance sampling rate to a reliable region near the old strategy (i.e., Inside, among which It is a hyperparameter representing the maximum allowable change in a single update; min is the minimum value of the regular reward and the clip reward. Used for summation and normalization operations, it represents the average of the data in the entire batch, where B is the number of prompts in a batch, K is the number of responses generated for each prompt, and t is the number of tokens in each response.

[0031] Furthermore, step (3) includes the following steps:

[0032] Step (3.1), calculation of dynamic weight fusion loss;

[0033] The total loss calculation formula is as follows:

[0034] ;

[0035] in, These are global coefficients in the training framework, used to guide the model's transition from supervised learning to reinforcement learning at a macro level; To enhance the learning loss function, The loss function for supervised learning;

[0036] Step (3.2): Execute the dynamic decay strategy for μ value;

[0037] The μ value gradually decreases from a high value of 0.5 to 1 during training to a low value of 0.5 to 0, thus shifting the training focus from imitation learning to exploratory learning.

[0038] In summary, the unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion proposed in this application has the following advantages and beneficial effects:

[0039] 1. This application unifies the training paradigm by integrating SFT and RL into a unified training framework, thus solving the hard switching problem in the prior art. Specifically, it adopts smooth transition training to eliminate the hard switching between the SFT and RL stages through dynamic decay of the μ value, thereby achieving a natural transition from imitation learning to exploratory learning.

[0040] 2. This application adopts a fine-grained weight control method and introduces a dynamic token-wise weighting function to finely control the influence of SFT data at the token level, thereby achieving the focus of learning "uncertain tokens".

[0041] 3. This application adopts a balanced approach of imitation learning and exploratory learning. In the early stage, the μ value is increased to imitate expert knowledge learning, and in the later stage, the μ value is decreased to enhance exploratory ability, thereby avoiding the risk of overfitting rewards.

[0042] 4. This application significantly improves training stability. The unified training framework can reduce target conflicts, and the token-wise weighting mechanism further suppresses interference from supervised fine-tuning data. Attached Figure Description

[0043] The present invention will now be further described with reference to the following figures.

[0044] Figure 1 This is a schematic diagram of a unified supervised fine-tuning and reinforcement learning training framework based on dynamic weight fusion;

[0045] Figure 2 This is a schematic diagram of the dynamic decay of the μ value; Detailed Implementation

[0046] Example 1: This application proposes a unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion. This method is based on a dual-path parallel architecture and a dynamic weight fusion mechanism to achieve unified integration and smooth transition of supervised fine-tuning (SFT) and reinforcement learning (RL).

[0047] Taking the OpenR1-Math-220k dataset as an example, the following methods are adopted: Figure 1 The unified supervised fine-tuning and reinforcement learning training framework shown includes the following implementation steps:

[0048] Step (1), Data preparation;

[0049] This includes SFT data preparation, which involves sampling 5,000 question-answer pairs (including thought processes and standard answers) from the dataset; and RL data preparation, which involves sampling 20,000 question-answer pairs (including thought processes and standard answers) from the dataset.

[0050] Step (2): Parallel processing along two paths;

[0051] Step (2.1), SFT path; see details. Figure 1 The lower half of the image is shown.

[0052] The SFT path is used to process SFT data question-answer pairs and calculate weighted supervised loss. It includes data input, which processes SFT data containing complete question-answer pairs (Prompt-Response) data.

[0053] In this context, Prompt represents the question or instruction entered by the user, and Response represents the standard answer.

[0054] The processing flow is as follows: input the SFT data into a pre-trained large language model, and calculate the SFT_loss of the model's generated response based on minimizing the negative log-likelihood of the response data.

[0055] ;

[0056] Among them, the weighting function It is based on the model's current confidence level in the token. ; The parameter is Large language models For conditional probability, given the cue word x and all previous real tokens, the large language model predicts the probability of the next real token to dynamically adjust the weight of the loss for each token; its purpose is to adjust the weight of the loss for tokens that the model is already confident about ( (close to 0 or 1) A small value means it's okay if it doesn't learn, preventing model overfitting; for tokens with uncertain models ( (close to 0.5) A large value means that the model should focus on learning it; The parameter is Large language models Let be the conditional probability, given the cue x and all previous real tokens, the probability that the large language model predicts the next real token; log is the natural logarithm of the probability. For each token position t in the standard answer of a data set, calculate its log probability, and then sum them to obtain the total loss of the data set; To perform the above operation on each data i in a batch and sum up the losses, we obtain the total loss of a batch, where B is the number of batches; "-" is a negative sign, representing minimizing the loss function; This represents a normalization operation, which divides the total loss of the batch by the total number of tokens for all standard answers in the batch.

[0057] Step (2.2) RL path, see details. Figure 1 The upper part is shown in the image;

[0058] The RL path is used to generate multiple candidate responses and compute reinforcement learning loss for RL data. It includes data input, which processes RL data containing only the question (Prompt), and the standard answer is only used to compute rule-based rewards. The processing flow is as follows: for each input Prompt, the large language model generates a total of K candidate responses; each candidate response is evaluated using rule-based rewards to compute a reward value.

[0059] The reward value is evaluated based on the following rules:

[0060] 1) Those who demonstrate a thought process and provide a correct answer will receive a positive reward of 1.1.

[0061] 2) For answers that only provide the correct answer without any thought process, a positive reward of 0.1 will be given;

[0062] 3) A neutral reward of 0.0 will be given for answers that include a thought process but are incorrect.

[0063] 4) If there is no thought process and the answer is incorrect, a penalty of -1 will be given.

[0064] The reinforcement learning loss is calculated based on the GRPO algorithm as follows:

[0065] ;

[0066] Where B represents the quantity indicated in a batch; The advantage of each candidate response is calculated using the following expression: , The original reward obtained for a candidate response. , The mean and standard deviation of the group rewards. It is a small constant added to ensure numerical stability; token-level importance sampling ratio. This is used to compare the preference of the new strategy and the old strategy for generating the next token; given the prompt word x and the historical tokens that have already been generated. Under the given conditions, the old strategy generates the next token. The probability is The clip mechanism limits the importance sampling rate to a reliable region near the old strategy (i.e., Inside, among which It is a hyperparameter representing the maximum allowable change in a single update; min is the minimum value of the regular reward and the clip reward. Used for summation and normalization operations, it represents the average of the data in the entire batch, where B is the number of prompts in a batch, K is the number of responses generated for each prompt, and t is the number of tokens in each response;

[0067] Step (3), dynamic weight fusion mechanism;

[0068] By dynamically adjusting the loss weights of the SFT and RL paths using global coefficients μ, a gradual transition from imitation learning to exploratory learning is achieved; including,

[0069] Step (3.1), calculation of dynamic weight fusion loss;

[0070] The total loss calculation formula is as follows:

[0071] ;

[0072] in, These are global coefficients in the training framework, used to guide the model's transition from supervised learning to reinforcement learning at a macro level; To reinforce the learning loss function, the specific calculation process is performed according to step (2.2); To supervise the learning loss function, the specific calculation process is performed according to step (2.1);

[0073] Step (3.2): Execute the dynamic decay strategy for μ value;

[0074] The decay principle is that the μ value gradually decreases from a high value (e.g., 0.9) to a low value (e.g., 0.05) during training, causing the training focus to gradually shift from the SFT path (imitation learning) to the RL path (exploratory learning). The dynamic decay process of the μ value is as follows: Figure 2 As shown.

[0075] As described above, the embodiments given in conjunction with the accompanying drawings are merely preferred solutions for achieving the objectives of this invention. Those skilled in the art can draw inspiration from this and directly derive other alternative structures that conform to the design concept of this invention. Other structural features derived therefrom should also fall within the scope of the solutions described in this invention.

Claims

1. A unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion, characterized in that: Based on a dual-path parallel architecture of SFT and RL, a dynamic weight fusion mechanism is used to implement the unified integration and smooth transition of supervised fine-tuning and reinforcement learning. The implementation steps include the following: Step (1), Data preparation; This includes SFT data preparation and RL data preparation, each involving sampling several question-answer pairs from the dataset; Step (2): Parallel processing along two paths; Step (2.1), SFT path; The process includes inputting data to process SFT data. The process involves inputting the SFT data into a pre-trained large language model and calculating the SFT loss of the model's generated response based on minimizing the negative log-likelihood of the response data. Step (2.2) RL path; The process includes inputting data to process RL data. The processing flow is as follows: for each input prompt, the large language model generates a total of K candidate responses; each candidate response is evaluated using rule-based rewards to calculate a reward value; and reinforcement learning loss is calculated based on the GRPO algorithm. Step (3), dynamic weight fusion mechanism; By dynamically adjusting the loss weights of the SFT and RL paths using global coefficients μ, a gradual transition from imitation learning to exploratory learning can be achieved.

2. The unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion according to claim 1, characterized in that: The expression for calculating SFT_loss in step (2.1) is as follows: ; Among them, the weighting function It is based on the model's current confidence level in the token. ; The parameter is Large language models For conditional probability, given the cue word x and all previous real tokens, the large language model predicts the probability of the next real token to dynamically adjust the weight of the loss for each token; its purpose is to adjust the weight of the loss for tokens that the model is already confident about ( (close to 0 or 1) A small value means it's okay if it doesn't learn, preventing model overfitting; for tokens with uncertain models ( (close to 0.5) A large value means that the model should focus on learning it; The parameter is Large language models Let be the conditional probability, given the cue x and all previous real tokens, the probability that the large language model predicts the next real token; log is the natural logarithm of the probability. For each token position t in the standard answer of a data set, calculate its log probability, and then sum them to obtain the total loss of the data set; To perform the above operation on each data i in a batch and sum up the losses, we obtain the total loss of a batch, where B is the number of batches; "-" is a negative sign, representing minimizing the loss function; This represents a normalization operation, which divides the total loss of the batch by the total number of tokens for all standard answers in the batch.

3. The unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion according to claim 2, characterized in that: In step (2.2), the input data only includes the data for the question (Prompt), and the standard answer is only used to calculate the rule-based reward.

4. The unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion according to claim 2, characterized in that: The reward value in step (2.2) is evaluated based on the following rules: A positive reward of 1.1 will be given for demonstrating the thought process and providing a correct answer; For answers that only provide the correct answer without any thought process, a positive reward of 0.1 will be given. A neutral reward of 0.0 will be given for answers that include a thought process but are incorrect. Those who neither demonstrate a thought process nor provide an correct answer will be penalized with -1.

5. The unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion according to claim 2, characterized in that: In step (2.2), the reinforcement learning loss is calculated using the following formula: ; Where B represents the quantity indicated in a batch; The advantage of each candidate response is calculated using the following expression: , The original reward obtained for a candidate response. , The mean and standard deviation of the group rewards. It is a small constant added to ensure numerical stability; token-level importance sampling ratio. This is used to compare the preference of the new strategy and the old strategy for generating the next token; given the prompt word x and the historical tokens that have already been generated. Under the given conditions, the old strategy generates the next token. The probability is The clip mechanism limits the importance sampling rate to a reliable region near the old strategy (i.e., Inside, among which It is a hyperparameter representing the maximum allowable change in a single update; min is the minimum value of the regular reward and the clip reward. Used for summation and normalization operations, it represents the average of the data in the entire batch, where B is the number of prompts in a batch, K is the number of responses generated for each prompt, and t is the number of tokens in each response.

6. The unified supervised fine-tuning and reinforcement learning training method based on dynamic weight fusion according to claim 1, characterized in that: Step (3) includes the following steps: Step (3.1), calculation of dynamic weight fusion loss; The total loss calculation formula is as follows: ; in, These are global coefficients in the training framework, used to guide the model's transition from supervised learning to reinforcement learning at a macro level; To enhance the learning loss function, The loss function for supervised learning; Step (3.2): Execute the dynamic decay strategy for μ value; The μ value gradually decreases from a high value of 0.5 to 1 during training to a low value of 0.5 to 0, thus shifting the training focus from imitation learning to exploratory learning.

Images