Diversity preserving supervision fine tuning method for reinforcement learning alignment
By redesigning the supervised fine-tuning objective function and combining methods such as entropy regularization and boundary regularization, the diversity of model outputs is maintained, the distribution collapse problem caused by supervised fine-tuning is solved, and the reinforcement learning effect of large language models is improved.
Patent Information
- Application Number
- CN202511435931.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-09
- Publication Date
- 2026-01-09
AI Technical Summary
Existing supervised fine-tuning methods lead to distribution collapse and reduced output diversity in large language models, limiting the exploration space and reasoning ability of subsequent reinforcement learning.
By redesigning the supervised fine-tuning objective function, combining entropy regularization, smoothed cross-entropy, and contrastive or boundary regularization, the diversity of model outputs is maintained, and reinforcement learning optimization is combined with weighted summation for model training.
While maintaining model alignment capabilities, it significantly improves output diversity, expands the solution space of reinforcement learning, and enhances the model's generalization ability and inference efficiency.
Smart Images
Figure CN121303235A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of large language model supervision, in particular to a diversity preserving supervision fine-tuning method for reinforcement learning alignment. BACKGROUND
[0002] Large language models (LLMs) have achieved remarkable success in a wide range of natural language processing tasks, from open-domain dialogue to complex reasoning, and have shown excellent capabilities. An important driving force for this progress comes from the alignment process that makes the pre-trained model conform to human preference behavior, among which the most representative is based on supervised fine-tuning (SFT) and supplemented by reinforcement learning (RL) strategy. This paradigm supports the most advanced reasoning systems including OpenAI's o1 / o3 models and DeepSeek-R1, indicating that post-training strategies are crucial to stimulating advanced reasoning capabilities.
[0003] However, there are still important limitations. Traditional supervised fine-tuning is usually optimized with cross-entropy loss, which often leads to distribution collapse and output diversity decline, thereby limiting the exploration space of subsequent reinforcement learning. Recent studies have further shown that reinforcement learning does not create new reasoning capabilities beyond those obtained by supervised fine-tuning; on the contrary, it mainly improves sampling efficiency by guiding the model to generate responses with high rewards. These findings highlight a key issue: to let reinforcement learning effectively improve reasoning capabilities, supervised fine-tuning must first preserve a sufficiently diverse solution space.
[0004] Supervised fine-tuning is the most widely used method to adapt large pre-trained language models to specific target distributions. Its core idea is to continue training the base model on carefully selected input-output paired data, so that it better aligns to the expected behavior. Depending on the nature of the data, supervised fine-tuning can play different roles: it can not only give the model professional ability in a specific field and improve its performance on certain tasks, but also enhance the model's ability to follow natural language instructions when trained using "instruction-response" data.
[0005] Over time, supervised fine-tuning has gone through several important development stages. Early work such as T0 and FLAN showed that rewriting diverse tasks into an instruction-following style can significantly improve zero-shot generalization, making "instruction fine-tuning" a key paradigm. Later, Self-Instruct and Alpaca reduced the dependence on expensive human annotation by using synthetic instruction-response data, while LIMA highlighted the powerful effect of "small but precise" curated datasets. In parallel, parameter-efficient methods such as LoRA and adapters allowed supervised fine-tuning to scale to larger models; more recent work shifted from "quantity" to "quality", emphasizing data selection, difficulty-level sampling, and entropy-based regularization to mitigate overfitting and maintain output diversity. These advances collectively established supervised fine-tuning as the de facto standard for the first stage of model alignment.
[0006] Despite its simplicity and effectiveness, supervised fine-tuning has limitations. Models trained through supervised fine-tuning often inherit biases and inconsistencies from the supervised data and can overfit to narrow distributions, limiting generalization to unseen instructions or heavily reasoning-dependent tasks. These limitations have driven the development of reinforcement learning-based methods.
[0007] While supervised fine-tuning laid a solid foundation, its inherited biases, overfitting to narrow distributions, and insufficient generalization to unseen instructions have pushed the community towards reinforcement learning-based methods. Among these methods, human feedback reinforcement learning (RLHF) has emerged as the most influential paradigm, typically involving three stages: (1) behavior pre-training through SFT; (2) training a reward model based on human preferences; (3) fine-tuning using RL algorithms such as PPO. This process has been widely adopted in models like InstructGPT and ChatGPT, significantly improving human-like alignment and reasoning capabilities.
[0008] However, RLHF and its variants face challenges: reward models can be fragile or mismatched due to limited preference data, RL algorithms can be unstable or inefficient, and the entire pipeline introduces additional computational costs and engineering complexity. To address these issues, new developments such as direct preference optimization (DPO) and group relative policy optimization (GRPO) reduce or simplify the reward modeling stage, providing more efficient and robust tuning solutions.
[0009] Despite this, relying solely on RL-based alignment does not eliminate all issues left by SFT: overfitting, diversity collapse, and domain bias may still exist, while RL itself can introduce instability and "reward speculation" risks. This further indicates the need for a hybrid approach that synergistically evolves supervised fine-tuning and reinforcement learning, leveraging the strengths of each in a tightly coupled process.
[0010] And the existing supervised fine-tuning and reinforcement learning collaborative objective function is:
[0011]
[0012] Often prompt the model distribution p θ (·|x) Collapse into a narrow mode. This collapse, while ensuring alignment with the supervised signal, weakens output diversity, limiting the space for RL to further optimize the strategy in subsequent exploration. SUMMARY
[0013] The purpose of the present application is to improve the SFT objective mentioned in the background art, so as to retain the model capability acquisition while maintaining sufficient diversity to support effective RL optimization, and to propose a diversity preserving supervised fine-tuning method for reinforcement learning alignment.
[0014] To achieve the above purpose, the present application provides the following technical solutions:
[0015] The diversity preserving supervised fine-tuning method for reinforcement learning alignment comprises the following steps:
[0016] Step 1: Construction of supervised fine-tuning model
[0017] S1: Obtain a demonstration data set for the target task, the data set containing input samples and corresponding generated expected outputs;
[0018] S2: Initialize the base model, and use the demonstration data set to supervise the training of the base model to obtain an initial strategy model;
[0019] S3: Evaluate the performance of the initial strategy model on the validation set, if the preset threshold is not reached, adjust the training parameters and repeat step S2 until a fine-tuned model that meets the basic performance requirements is obtained;
[0020] S4: Optimize the fine-tuned model in S3;
[0021] Step 2: Construction of reward model
[0022] S1: Use the fine-tuned model obtained in step 1 to generate multiple candidate outputs for a preset input set;
[0023] S2: Collect the preference ordering data of the annotators on the multiple candidate outputs, the preference ordering data representing the relative quality between different candidate outputs;
[0024] S3: Use the preset input and the corresponding candidate output as input, and use the preference ordering data as the supervised signal to train the reward model;
[0025] S4: Verify the prediction accuracy of the reward model. When the prediction accuracy reaches the preset standard, determine the final reward model.
[0026] Step 3: Optimize by combining reinforcement learning
[0027] S1: Combine the fine-tuning model with reinforcement learning using a weighted summation method. Use the fine-tuning model obtained in step one as the initial policy and set the reinforcement learning training parameters.
[0028] S2: During training, the policy model generates the current output for the input, and calculates the reward value corresponding to the output using the reward model obtained in step two;
[0029] S3: Update the strategy model parameters based on the reward value.
[0030] In a preferred embodiment, the optimization process of the fine-tuning model in step one is carried out by entropy regularization, smoothing cross-entropy, and contrast or boundary regularization.
[0031] In a preferred embodiment, the entropy regularization directly combats distribution collapse by encouraging the predicted distribution to remain dispersed among multiple alternative outputs, and its objective function is:
[0032]
[0033] in To predict the entropy of the distribution, a tradeoff coefficient λ controls the strength of the model to avoid overconfidence.
[0034] In a preferred embodiment, the objective function for smoothing cross-entropy in step one is:
[0035]
[0036] Where q ∈ (y|x) is a softened target distribution that assigns the probability mass of (1-∈) to the reference label y* and distributes ∈ to other candidates.
[0037] In a preferred embodiment, the positive sample label y is given in the comparison or boundary regularization process in step one. + and the sampled negative samples {y - The boundary loss can be defined as follows:
[0038]
[0039] Where m is the boundary hyperparameter.
[0040] In a preferred embodiment, the weighted summation function in step three is:
[0041]
[0042] α(t) and β(t) can change with the training progress.
[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0044] 1. This invention redesigns or extends the CE objective to both maintain alignment and explicitly encourage the retention of multiple reasonable solutions. Entropy regularization prevents the model from collapsing into deterministic outputs, providing a wider solution space for subsequent RL. Variations include adaptive entropy weights or constraints that enforce a minimum entropy level during training;
[0045] 2. This invention's smoothed cross-entropy prevents the model from completely collapsing into a distribution, thus preserving minor hypotheses. Common implementations include label smoothing, confidence penalty CE, or temperature scaling of logits. Such smoothing mechanisms are particularly important when the supervision signal itself is noisy or incomplete, making the model more robust and less fragile.
[0046] 3. This invention employs contrastive or boundary regularization to ensure that while maintaining the advantage of the reference output, it does not excessively suppress competitive modes, thereby preserving diversity at the local decision boundary. Possible extensions include multiple negative sample sampling, dynamic boundary adjustment, or combining contrastive representation learning to further enhance robustness;
[0047] 4. This invention employs an enhanced SFT loss that balances capability acquisition and diversity preservation for training; this checkpoint provides a wider and more robust solution space for subsequent RL algorithms (such as PPO, DPO, or GRPO). Attached Figure Description
[0048] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0049] Figure 1 This is a diversity comparison chart for Qwen2.5-1.5B of the present invention.
[0050] Figure 2 This is a comparison chart of the diversity of Qwen2.5-Math-1.5B of the present invention. Detailed Implementation
[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0052] The present invention provides the following technical solution:
[0053] A supervised fine-tuning method for maintaining diversity in reinforcement learning alignment includes the following steps:
[0054] Includes the following steps:
[0055] Step 1: Construction of the supervised fine-tuning model
[0056] S1: Obtain a sample dataset for the target task, the dataset containing input samples and corresponding expected outputs;
[0057] S2: Initialize the base model and conduct supervised training on the base model using the demonstration dataset to obtain the initial policy model;
[0058] S3: Evaluate the performance of the initial strategy model on the validation set. If the preset threshold is not met, adjust the training parameters and repeat step S2 until a fine-tuned model that meets the basic performance requirements is obtained.
[0059] S4: Optimize the fine-tuning model in S3;
[0060] Step Two: Building the Reward Model;
[0061] S1: Using the fine-tuned model obtained in step one, generate multiple candidate outputs for the preset input set;
[0062] S2: Collect the preference ranking data of the annotators for the multiple candidate outputs, wherein the preference ranking data represents the relative quality between different candidate outputs;
[0063] S3: Train the reward model using a preset input and the corresponding candidate output as input, and the preference ranking data as a supervision signal;
[0064] S4: Verify the prediction accuracy of the reward model. When the prediction accuracy reaches the preset standard, determine the final reward model.
[0065] Step 3: Optimize by combining reinforcement learning
[0066] S1: Combine the fine-tuning model with reinforcement learning using a weighted summation method. Use the fine-tuning model obtained in step one as the initial policy and set the reinforcement learning training parameters.
[0067] S2: During training, the policy model generates the current output for the input, and calculates the reward value corresponding to the output using the reward model obtained in step two;
[0068] S3: Update the strategy model parameters based on the reward value.
[0069] In a preferred embodiment, the optimization process of the fine-tuning model in step one is carried out by entropy regularization, smoothing cross-entropy, and contrast or boundary regularization.
[0070] In a preferred embodiment, the entropy regularization directly combats distribution collapse by encouraging the predicted distribution to remain dispersed among multiple alternative outputs, and its objective function is:
[0071]
[0072] in To predict the entropy of the distribution, a tradeoff coefficient λ controls the strength of the model to avoid overconfidence.
[0073] In a preferred embodiment, the objective function for smoothing cross-entropy in step one is:
[0074]
[0075] Where q ∈ (y|x) is a softened target distribution that assigns the probability mass of (1-∈) to the reference label y* and distributes ∈ to other candidates.
[0076] In a preferred embodiment, the positive sample label y is given in the comparison or boundary regularization process in step one. + and the sampled negative samples {y - The boundary loss can be defined as follows:
[0077]
[0078] Where m is the boundary hyperparameter.
[0079] In a preferred embodiment, the weighted summation function in step three is:
[0080]
[0081] α(t) and β(t) can change with the training progress.
[0082] To validate our research motivation, we conducted preliminary experiments to explore the relationship between output diversity in the SFT stage and the effectiveness of subsequent RL optimization, aiming to provide preliminary evidence that maintaining diversity in the SFT stage may bring actual benefits to downstream preference optimization.
[0083] We randomly sampled 12,000 samples from the OpenR1-Math inference dataset and compared two model variants: Please refer to [link / reference]. Figure 1 and Figure 2Qwen2.5-1.5B (base) and Qwen2.5-Math (domain-adapted). For each input question, we use nucleus sampling to generate 128 outputs and quantify diversity at the level of the final extracted answer: any different final answer is considered a different solution, regardless of the wording or intermediate reasoning steps.
[0084] In the picture:
[0085] • Number of different solutions (U): The total number of different solutions in the sample.
[0086] • Dominant solution ratio (D): The frequency of occurrence of the most common solution.
[0087] • Entropy (H): Shannon entropy of the empirical answer distribution.
[0088] • Distinct-n: The proportion of unique n-grams in the output, used to measure lexical diversity.
[0089] Experiments revealed significant differences in the diversity performance of the two models. The base model, Qwen2.5-1.5B, exhibited a higher number of distinct solutions and entropy, along with a lower dominance ratio, indicating a broader exploration of the reasoning path. In contrast, the domain-adapted model had a narrower distribution, a higher dominance ratio, and lower entropy, exhibiting pattern collapse characteristics. This supports the hypothesis that "stronger domain adaptation may shrink the exploration space, thus limiting the refinement effect of RL." However, given that the analysis is based on a relatively small-scale experiment, it still requires careful interpretation.
[0090] We further used GRPO with a 2k step size to perform RL optimization. The inference baseline performance (%) of Qwen2.5-1.5B and Qwen2.5-Math-1.5B before and after GRPO fine-tuning is shown in Table 1:
[0091]
[0092] Table 1
[0093] GRPO offers some improvement on GSM8k, but provides little improvement on the more challenging AIME benchmark, especially for mathematically adapted models. This aligns with the assumption that insufficient diversity in the SFT stage may limit the benefits of RL refinement, but it's not sufficient to establish a causal relationship. Other possible factors also exist: for example, a small model size, limited model family selection, or shallow RL training (only 2000 steps) could all lead to limited improvements.
[0094] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A supervised fine-tuning method for diversity preservation in reinforcement learning alignment, characterized by: Includes the following steps: Step 1: Construction of the supervised fine-tuning model S1: Obtain a sample dataset for the target task, the dataset containing input samples and corresponding expected outputs; S2: Initialize the base model and conduct supervised training on the base model using the demonstration dataset to obtain the initial policy model; S3: Evaluate the performance of the initial strategy model on the validation set. If the preset threshold is not met, adjust the training parameters and repeat step S2 until a fine-tuned model that meets the basic performance requirements is obtained. S4: Optimize the fine-tuning model in S3; Step Two: Building the Reward Model; S1: Using the fine-tuned model obtained in step one, generate multiple candidate outputs for the preset input set; S2: Collect the preference ranking data of the annotators for the multiple candidate outputs, wherein the preference ranking data represents the relative quality between different candidate outputs; S3: Train the reward model using a preset input and the corresponding candidate output as input, and the preference ranking data as a supervision signal; S4: Verify the prediction accuracy of the reward model. When the prediction accuracy reaches the preset standard, determine the final reward model. Step 3: Optimize by combining reinforcement learning S1: Combine the fine-tuning model with reinforcement learning using a weighted summation method. Use the fine-tuning model obtained in step one as the initial policy and set the reinforcement learning training parameters. S2: During training, the policy model generates the current output for the input, and calculates the reward value corresponding to the output using the reward model obtained in step two; S3: Update the strategy model parameters based on the reward value.
2. The diversity-preserving supervised fine-tuning method for reinforcement learning alignment according to claim 1, characterized in that: In step one, the optimization of the fine-tuning model is performed using entropy regularization, smoothing cross-entropy, and contrastive or boundary regularization.
3. The diversity-preserving supervised fine-tuning method for reinforcement learning alignment according to claim 2, characterized in that: The entropy regularization directly combats distribution collapse by encouraging the predicted distribution to remain dispersed among multiple alternative outputs. Its objective function is: in To predict the entropy of the distribution, a tradeoff coefficient λ controls the strength of the model to avoid overconfidence.
4. The diversity-preserving supervised fine-tuning method for reinforcement learning alignment according to claim 2, characterized in that: The objective function for smoothing cross-entropy in step one is: Where q ∈ (y|x) is a softened target distribution that assigns the probability mass of (1-∈) to the reference label y* and distributes ∈ to other candidates.
5. The diversity-preserving supervised fine-tuning method for reinforcement learning alignment according to claim 2, characterized in that: In step one, the comparison or boundary regularization process provides a positive sample label y. + and the sampled negative samples {y - The boundary loss can be defined as follows: Where m is the boundary hyperparameter.
6. The diversity-preserving supervised fine-tuning method for reinforcement learning alignment according to claim 1, characterized in that: The function for weighted summation in step three is: α(t) and β(t) can change with the training progress.