Method and system for preference optimization of large model based on reward margin constraint

Abstract

This invention provides a method and system for large-scale model preference optimization based on reward margin constraints. It constructs a third-order margin partitioning criterion and dynamically estimates the quantile threshold of the reward margin using exponential moving averages, adaptively dividing preference pairs into uncertainty, buffer, and confidence regions. Subsequently, a differentiated mapping strategy is applied to different regions. Finally, the differentiated mapping is integrated into a truncated sigmoid function, and a TruncPO loss function is constructed to optimize model parameters for human preference alignment tasks in large language models. This invention can improve the original win rate of different models by 4% in benchmark tests such as AlpacaEval2 and Arena-Hard, while reducing KL divergence fluctuation by more than one-third, effectively balancing the order consistency and probabilistic calibration of preference optimization. It is suitable for deployment in large language model application systems requiring accurate preference alignment, such as intelligent dialogue, automatic problem solving, and intelligent education.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and large language model optimization technology, specifically a method and system for optimizing large language model preferences based on reward margin constraints. It relates to a method and system for accurately optimizing large language model preferences in human feedback alignment scenarios through dynamic margin partitioning and regional mapping strategies. It can be applied to engineering systems that rely on large language models and need to conform to human preferences, such as intelligent dialogue systems, automatic problem-solving platforms, intelligent education tools, intelligent customer service, and content generation systems. Background Technology

[0002] Large Language Models (LLMs) have demonstrated outstanding performance in various downstream tasks such as open-domain question answering, code generation, and machine translation. However, their native output often fails to fully align with human values ​​and practical usage needs. To address this issue, Learning Based on Human Feedback (RLHF) has become a core paradigm for aligning large language models with human preferences. By collecting feedback from humans on the model's output, it guides the model to optimize its generation strategy, ensuring that the output content is useful, honest, and harmless.

[0003] Traditional RLHF methods involve three stages: supervised fine-tuning, reward model training, and policy optimization. These methods suffer from high computational costs, complex training processes, and insufficient stability, making them unsuitable for the rapid iteration and engineering deployment of large-scale models. To address this, Direct Alignment Assays (DAAs) have been proposed as an offline optimization paradigm. They eliminate the need to construct an explicit reward model, directly optimizing the model using preference data by reparameterizing the reward function between the policy model and the reference model. This significantly simplifies the training process and reduces computational overhead.

[0004] Existing direct alignment algorithms are mainly divided into two technical routes: (1) Methods based on the Bradley-Terry (BT) model. This type of method relies on the sigmoid function for probability calibration, modeling preference probabilities as a logical mapping of utility differences, which can better match the statistical distribution of human preferences. However, when dealing with preference pairs with large marginal differences, this type of method is prone to gradient saturation, which weakens the training signal. At the same time, the excessively amplified probability ratio will cause the policy model to deviate too much from the reference model, weakening the KL regularization effect and causing over-optimization problems. (2) BT-free methods. This type of method abandons the sigmoid mapping and uses the identity mapping to directly optimize utility differences, emphasizing the maintenance of the order consistency of preference ranking. However, this type of method ignores the probability calibration requirements of fuzzy preference pairs, making it difficult to accurately capture the fine-grained differences of human preferences, resulting in poor alignment performance of the model in ambiguous scenarios.

[0005] On the other hand, in preference optimization tasks, balancing order consistency and probabilistic calibration is key to improving alignment quality: order consistency ensures the model follows the ranking of human preferences, while probabilistic calibration guarantees that the model's predicted preference probabilities are consistent with the actual human distribution statistics. Existing techniques often place extreme emphasis on one side, failing to achieve an effective balance between the two. Furthermore, existing methods lack a dynamic adjustment mechanism for reward margins, making them unable to adapt to preference characteristics under different data distributions, further limiting the improvement of alignment performance and generalization ability. Existing techniques have few optimization schemes that combine dynamic margin partitioning with regional differential mapping, and a systematic approach that balances alignment quality, training stability, and generalization ability has not yet been formed, making it difficult to meet the engineering needs of accurately aligning human preferences using large language models in complex scenarios.

[0006] The technical differences between this application and the prior art are as follows:

[0007] The main differences between this application and patent CN202511399136.9, "A Direct Preference Optimization Method and Apparatus Based on Importance Sampling," lie in the optimization granularity, core issues, and technical framework. Patent CN202511399136.9 focuses on importance sampling at the character tag level, achieving key character optimization through weight allocation; its core is "local element weighting." In contrast, this invention focuses on dynamic partitioning and differentiated mapping of reward margins, balancing order consistency and probability calibration through a region adaptation strategy; its core is "global margin constraints." Regarding the core issues addressed, patent CN202511399136.9 primarily addresses "low optimization efficiency caused by overall sequence processing," focusing on improving training resource utilization efficiency. This invention, however, focuses on the triple problems of "gradient saturation, over-optimization, and insufficient probability calibration," balancing optimization effect, stability, and calibration accuracy. In terms of technical framework, patent CN202511399136.9 relies on positive or negative preference models to estimate character weights, which requires the construction of additional auxiliary models. This invention achieves adaptive optimization through quantile thresholds and segmented mapping functions, eliminating the need for additional auxiliary models. The framework is simpler and the deployment cost is lower.

[0008] The differences between this application and patent CN202411944605.6, which describes a direct preference optimization method and apparatus with sparse feature constraints, lie in the constraint dimension, training method, and applicable scenarios. Patent CN202411944605.6 uses "sparse activation feature difference + dynamic marginal value" as a dual constraint, with the core being feature-level bias control. This invention, however, focuses on a "regional mapping strategy," employing a segmented design that preserves sigmoid, linearizes, and truncates, while simultaneously satisfying the three principles of order consistency, smooth differentiability, and marginal antisymmetry. Its core is the adaptive adjustment of the marginal mapping. Regarding model training, patent CN202411944605.6 requires copying reference model parameters to construct a dynamic policy model, relying on feature comparison between the reference model and the policy model. This invention fixes the reference model and achieves optimization only through marginal calculations between the policy model and the reference model, eliminating the need to copy model parameters and resulting in lower training memory overhead. In terms of applicable scenarios, patent CN202411944605.6 adapts to high-dimensional feature scenarios through sparse feature constraints, making it more suitable for specific tasks with sparse features. This invention adapts to preference data with different marginal distributions through dynamic partitioning, resulting in stronger generalization and covering various preference alignment scenarios such as general dialogue and mathematical reasoning.

[0009] The main differences between this application and patent CN202511171100.5, "A Training Method and System for Large Model Alignment Based on Direct Preference Optimization," lie in the use of the reference model, the optimization objective, and the technical implementation path. Patent CN202511171100.5 achieves personalized settings by constructing an implicit reference model, with the core being "dynamic adjustment of the reference model." This invention fixes the reference model as the optimization benchmark and controls the difference between the strategy model and the reference model through marginal partitioning and truncation strategies, with the core being "marginal constraints on the strategy model." Regarding the optimization objective, patent CN202511171100.5 focuses on solving the problems of suboptimal reference model and fixed reward margins, with the core objective being to improve alignment performance and win rate. This invention focuses on solving the balance problem of gradient saturation, overoptimization, and insufficient probability calibration, with the core objective being to balance performance, stability, and calibration accuracy. In terms of technical implementation, patent CN202511171100.5 controls KL divergence by adjusting the balance relationship between the strategy model and the reference model. This invention directly suppresses excessive deviation by truncating the confidence region margin, and at the same time ensures gradient stability through buffer linearization mapping. KL divergence control is more direct and robust, and does not require complex balance parameter tuning.

[0010] In summary, existing technologies suffer from the following problems: 1. Some methods focus on optimizing local elements without considering the overall characteristics of the reward margin, making it difficult to balance order consistency and probabilistic calibration, and failing to accurately capture the statistical distribution patterns of human preferences; 2. Some methods rely on feature comparison or additional auxiliary models, which not only increases training complexity and memory overhead but also reduces deployment flexibility, making it difficult to adapt to resource-constrained engineering scenarios; 3. Some methods emphasize adjusting the reference model or simply pursuing a higher win rate, failing to effectively address gradient saturation and overoptimization issues, resulting in insufficient training stability, limited generalization ability, and difficulty in coping with diverse preference data distributions; 4. There is a lack of an integrated solution that does not require additional auxiliary models, can adaptively adapt to different marginal distributions, and simultaneously alleviates gradient saturation, overoptimization, and insufficient probabilistic calibration, making it difficult to meet the comprehensive requirements for preference alignment accuracy, stability, and efficiency in complex scenarios. Summary of the Invention

[0011] The purpose of this invention is to propose a large model preference optimization method based on reward margin constraints. By dynamically dividing the reward margin region of preference pairs and adopting a differentiated mapping strategy, this method can solve the problems of gradient saturation, over-optimization and insufficient probability calibration in existing methods while ensuring the consistency of preference ranking in large language models, thereby improving the stability and accuracy of model alignment.

[0012] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0013] Step S1: Calculate the reward margin for preference pairs and dynamically partition them;

[0014] Step S2: Constructing the regional boundary mapping function;

[0015] Step S3: TruncPO loss function optimization and model preference alignment inference.

[0016] As a further improvement to the present invention, step S1, the calculation of the reward margin and dynamic partitioning of the preference pair, specifically includes: defining the preference dataset as... ,in Provide prompts for user-inputted commands. For preferences confirmed by human annotation, This corresponds to the unfavorable response. The strategy model to be optimized is set as follows: This model is used to learn human preferences and generate expected responses; meanwhile, a fixed reference model is used. This is used to provide a stable optimization benchmark and prevent the policy model from deviating excessively from its original capabilities. A temperature hyperparameter is introduced to quantify the differences between preference pairs. To control the sharpness of the reward margin, calculate the scaled reward margin for each preference pair: .

[0017] This marginal value directly reflects the policy model's ability to distinguish between preferred and non-preferred responses. A larger marginal value indicates a more significant difference in the model's current preference for the two types of responses. To achieve dynamic and accurate region segmentation, a quantile threshold method is used to classify the marginal values, selecting the 70th quantile of the marginal distribution. with 85th percentile As core thresholds, these two quantiles cover the vast majority of normal marginal values ​​while effectively avoiding interference from extreme outliers. To prevent excessive fluctuations in the threshold during training, an exponential moving average (EMA) mechanism is used to smooth the threshold, as shown in the formula: ,in For smoothing coefficients, The marginal value in the current training batch - Quantiles, through this mechanism, allow the threshold to adaptively adjust as the training process progresses, better reflecting the dynamic changes in data distribution. Combined with the uncertainty coefficient... Calculate the final dynamic cutoff threshold The uncertainty coefficient Calculated from the normalized binary entropy of Bernoulli variables: First through Obtain the probability percentage of the preference response, and then... Calculation uncertainty The range of values ​​is A higher value indicates greater difficulty in distinguishing the current preference pair, requiring more emphasis on probability calibration. Finally, based on the threshold... Intersection with linear mapping , Using the standard sigmoid function, all preference pairs are divided into three functionally defined regions:

[0018] Uncertainty region: satisfies In this region, the marginal differences between preference pairs are small, and the model has low discrimination between the two types of responses. This region is the core area for probability calibration, and it is necessary to preserve the fine mapping relationship to capture subtle preference differences.

[0019] Buffer: satisfies The marginal values ​​in this region are moderate, and the model can initially distinguish between the two types of responses, but gradient saturation is prone to occur, so linearization is needed to maintain a stable optimization signal.

[0020] Confidence zone: satisfies If the marginal value in this region is too large, further optimization may lead to overfitting and excessive deviation from the reference model. Therefore, it is necessary to suppress over-optimization by truncation.

[0021] As a further improvement to the present invention, step S2, the construction of the regional boundary mapping function, includes three sub-steps:

[0022] S2.1 Uncertainty Region Sigmoid Mapping: The standard sigmoid function is retained as the mapping relationship. This function can smoothly map marginal values ​​to a fixed interval, and the gradient is large when the marginal value is close to 0. This allows the model to obtain sufficient training signals for distinguishing preference pairs that are difficult to distinguish, and achieves accurate matching between probability distribution and human preferences.

[0023] S2.2 Buffer Linearization Mapping: A linear replacement function of the sigmoid function at the truncation threshold is adopted. This linear function has the same function value and slope as the sigmoid function at the threshold, realizing a smooth transition from nonlinear to linear mapping, avoiding abrupt changes in the mapping relationship, and providing a stable gradient to ensure continuous and efficient optimization of the model.

[0024] S2.3 Confidence Region Truncation Mapping: This mapping truncates preference pairs with excessively large marginal values, limiting the excessive influence of these values ​​on model parameters, preventing uncontrolled distribution differences between the policy model and the reference model, avoiding overfitting caused by unlimited upgrades in reward values, and enhancing the model's generalization ability.

[0025] The final result is a piecewise mapping function that balances probabilistic calibration accuracy, gradient stability, and optimization constraints.

[0026] As a further improvement to the present invention, step S3 TruncPO loss function optimization and model preference alignment inference includes three sub-steps:

[0027] S3.1 Loss Function Construction: Based on the piecewise mapping function in step S2, construct the TruncPO loss function. This loss function must satisfy three principles: ensuring the order consistency that the preference response always obtains a higher mapping value, maintaining smooth differentiability throughout the process, and ensuring marginal antisymmetry that balances positive and negative marginal values.

[0028] S3.2 Parameter Optimization: The AdamW optimizer is used to update the policy model parameters. During training, only policy parameters related to the current preference pair are updated, while the reference model parameters are kept frozen to maintain the stability of the optimization baseline. The training configuration can be flexibly adjusted, with batch size set to 32-64 and learning rate set to [value missing]. It employs memory-efficient training techniques to support the training of large-scale models and datasets, with the number of iterations adapted to the size of the dataset.

[0029] S3.3 Alignment Inference: After the model is optimized, multiple candidate responses are generated for new input prompts. The marginal value between each candidate response and the reference model is calculated. The preference probability is calculated through a piecewise mapping function. The response with the highest probability is selected as the final output. It can be applied to various scenarios that require precise alignment with human preferences, such as intelligent dialogue, automatic problem solving, educational assistance, and intelligent customer service.

[0030] As a further improvement of this invention, the value of the temperature hyperparameter is dynamically adjusted according to the model size: the optimal value for small-to-medium-scale models is 0.05, which ensures the discriminative power of marginal values ​​without amplifying noise; the optimal value for larger-scale models is 0.03, avoiding excessive sensitivity to marginal values ​​that could lead to optimization instability. The EMA smoothing coefficient is fixed at 0.9 to balance the threshold update speed and stability. As a further improvement of this invention, the calculation of the linear mapping intersection point must satisfy the condition that the linear mapping function is 1. By simplifying the formula, the accurate intersection point value can be calculated quickly, ensuring that the buffer exactly covers the medium marginal area and avoiding premature entry into the truncation state, which could lead to the loss of effective training signals. As a further improvement of this invention, length control and style control mechanisms can be introduced into the model preference alignment inference process: length control corrects the marginal value by normalizing the response length, avoiding alignment deviation caused by response length preference; style control filters candidate responses that match the target style through a pre-trained style classifier, ensuring preference alignment while meeting the requirements of scene-specific style.

[0031] This invention also provides a large-scale model preference optimization system based on reward margin constraints for implementing the method, including a policy model, a reference model, and a preference dataset. The policy model includes, but is not limited to, mainstream open-source or closed-source instruction fine-tuning models, possessing good compatibility and scalability. The reference model uses an initial instruction fine-tuning model with the same architecture as the policy model to ensure consistency and accuracy in margin calculations. This system is compatible with various types of preference datasets and can adapt to different data distributions by adjusting quantile thresholds, temperature hyperparameters, and other configurations. It is suitable for various engineering scenarios requiring precise alignment with human preferences, such as intelligent dialogue systems, automated problem-solving platforms, educational aids, and intelligent customer service.

[0032] Beneficial effects:

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. This invention uses a dynamic marginal partitioning and differentiated mapping strategy to accurately balance the consistency of order and the calibration of probability, ensuring the rationality of the ranking of preference responses and achieving statistical consistency between the predicted probability and the distribution of human preferences, thus solving the problem of existing methods that focus excessively on a single objective.

[0035] 2. The present invention employs a linearized mapping design in the buffer, which effectively alleviates the gradient saturation problem of the sigmoid function and provides a stable and sufficient training signal for pairs with moderate marginal preferences. At the same time, the confidence region truncation mechanism suppresses over-optimization caused by excessively large margins and weakening of KL regularization, significantly improving training stability.

[0036] 3. This invention dynamically adjusts the partition threshold through the exponential moving average (EMA) mechanism and focuses on high-value fuzzy preference pairs by combining uncertainty coefficients. It requires no additional auxiliary models or feature engineering, has a simple and efficient framework, lower training and deployment costs, and is adaptable to large language models of different sizes and diverse preference datasets.

[0037] 4. In authoritative benchmark tests such as AlpacaEval2 and Arena-Hard, this invention can improve the original win rate of models such as LLaMA3-8B, Mistral2-7B, and Gemma2-9B by 1.0%-4.2%, while reducing KL divergence fluctuation by more than 30%. It demonstrates stronger generalization ability in complex technical inference and general instruction following scenarios, and is suitable for deployment in engineering systems with high requirements for preference alignment accuracy and reliability. Attached Figure Description

[0038] Figure 1 This is a framework diagram for a large-scale model preference optimization method based on reward marginal constraints.

[0039] Figure 2 This is a schematic diagram of the regional boundary mapping function. Detailed Implementation

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0041] Example:

[0042] Assume the preference dataset is ,in Provide prompts for user-inputted commands. For preferences confirmed by human annotation, This corresponds to the unfavorable response. The strategy model to be optimized is set as follows: This model is used to learn human preferences and generate expected responses; meanwhile, a fixed reference model is used. This is used to provide a stable optimization benchmark and prevent the strategy model from deviating excessively from the original capability.

[0043] like Figure 1As shown, the overall framework of this invention includes three stages: The first stage is the calculation of the reward margin of preference pairs and dynamic partitioning: the reward margin of preference pairs is calculated, and preference pairs are divided into uncertainty region, buffer region, and confidence region by using the EMA smoothing threshold; The second stage is the construction of regional marginal mapping functions: differentiated mapping strategies (sigmoid mapping, linearized mapping, truncated mapping) are adopted for different regions to form piecewise continuous mapping functions; The third stage is the optimization of TruncPO loss function and model preference alignment inference: the loss function is constructed based on the piecewise mapping function, the strategy model parameters are optimized, and preference alignment inference is achieved by using marginal calculation and probability screening.

[0044] Step S1: Calculate the reward margin for preference pairs and dynamically partition them;

[0045] In the preference alignment task of a certain intelligent dialogue system, the daily newly added user feedback preference data... This embodiment is for The specific steps to perform dynamic partitioning are as follows:

[0046] (1) Load the initialized strategy model Compared with the reference model Input preference dataset ;

[0047] (2) Set the temperature hyperparameter β=0.05 (to fit the LLaMA3-8B model) and calculate the reward margin for each preference pair: ;

[0048] (3) Select the 70th quantile of the marginal distribution with 85th percentile As the core threshold, the threshold is smoothed using an exponential moving average (EMA) mechanism, as shown in the formula: ,in For smoothing coefficients, The marginal value in the current training batch -Quantities;

[0049] (4) Calculate the uncertainty coefficient First pass Obtain the probability percentage of preference responses, and then... Calculate uncertainty;

[0050] (5) Combination Calculate the cutoff threshold Calculate the intersection of the linear mappings, and finally divide the preference pairs into uncertain regions. ), buffer ( ), confidence zone ( ).

[0051] Through the above partitioning, the system can accurately locate high-value fuzzy preference pairs (uncertainty region) and preference pairs that need to suppress over-optimization (confidence region), and then allocate training resources accordingly to improve alignment efficiency.

[0052] Step S2: Constructing the regional boundary mapping function;

[0053] For the three regions divided in Example 1, this example constructs a differentiated mapping function, and the specific steps are as follows:

[0054] (1) Uncertainty region sigmoid mapping: for The preference pair is calculated using the standard sigmoid function. This allows for a larger gradient when the marginal value is close to 0, accurately capturing fine-grained differences in users' fuzzy preferences;

[0055] (2) Buffer linearization mapping: for The preference pair is expressed using the first-order Taylor expansion of the sigmoid at c. To achieve the transformation from nonlinear to linear. - Continuous transition alleviates gradient saturation;

[0056] (3) Confidence region truncation mapping: for Preferences, settings This limits the impact of excessively large margins on parameters and prevents the KL divergence between the strategy model and the reference model from getting out of control.

[0057] As shown in Figure 2, the above piecewise mapping function is The continuity of the function value and slope ensures both the accuracy of probability calibration and the stability of the gradient.

[0058] Step S3: TruncPO loss function optimization and model preference alignment inference;

[0059] After constructing the mapping function, this embodiment performs model optimization and inference on the GPU server. The specific steps are as follows:

[0060] (1) Constructing the TruncPO loss function: using a piecewise mapping function Construct the loss function around the core: The loss function strictly adheres to three design principles: order consistency and piecewise mapping function. It exhibits a strictly increasing trend, ensuring that the preference response always obtains a higher mapping value, consistent with human preference ranking; smooth differentiability: achieved through linear mapping of the buffer and sigmoid mapping of the uncertainty region. - Continuous, maintaining differentiability throughout, providing a stable and continuous gradient signal for model optimization; marginal antisymmetry: satisfies... This ensures that positive and negative marginal values ​​are balanced, avoiding optimization bias caused by marginal sign preference in the model.

[0061] (2) Policy model parameter optimization: The AdamW optimizer was used, with a batch size of 64 and a learning rate of 3×10⁻⁵. DeepSpeed ​​ZeRO-2 technology was enabled to reduce memory overhead. During training, only policy parameters related to the current preference pair were updated, referencing the model. The parameters are kept frozen to maintain the stability of the optimization baseline; the AlpacaEval2 dataset is iterated for 4 rounds to ensure that the model fully converges;

[0062] (3) Preference Alignment Inference: After the model completes overall optimization and parameter convergence, it responds to new user input instructions. (For example, "explaining the application scenarios of the Pythagorean theorem"), generating five candidate responses with differences in content, structure, and expression style. ; Calculate the marginal value of each candidate response ( (as neutral reference response); through Calculate the preference probability and select the response with the highest probability as the output.

[0063] In a real-world dialogue system, this reasoning process incurs no additional real-time computational overhead and can be directly integrated into existing service processes to output high-quality responses that meet user preferences.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A large-scale model preference optimization method based on reward marginal constraints, characterized in that, Includes the following steps: Step S1: Calculate the reward margin for preference pairs and dynamically partition them; Step S1 specifically includes: defining the preference dataset as... ,in Provide prompts for user-inputted commands. For preferences confirmed by human annotation, For the corresponding unfavorable response; set the strategy model to be optimized as follows: This model is used to learn human preferences and generate expected responses; meanwhile, a fixed reference model is used. This provides a stable optimization benchmark, preventing the policy model from deviating excessively from its original capabilities; a temperature hyperparameter is introduced to quantify the differences between preference pairs. To control the sharpness of the reward margin, calculate the scaled reward margin for each preference pair: ; To prevent excessive fluctuations in the threshold during training, the EMA mechanism smooths the threshold, as shown in the formula: ,in Let represent the p-quantile threshold of the reward margin after exponential moving average (EMA) smoothing in the training batch, where . For smoothing coefficients, The marginal value in the current training batch -Quantile, where t is the index of the current training batch. This mechanism allows the threshold to be adaptively adjusted as the training process progresses, better reflecting the dynamic changes in data distribution. Combining uncertainty coefficient Calculate the final dynamic cutoff threshold The uncertainty coefficient Calculated from the normalized binary entropy of Bernoulli variables: First pass Obtain the probability percentage of the preference response, and then... Calculation uncertainty The range of values ​​is Finally, based on the dynamic truncation threshold Intersection with linear mapping All preference pairs are divided into three functionally defined regions: Uncertainty region: satisfies In this region, the marginal differences between preference pairs are small, and the model has low discrimination between the two types of responses. This region is the core area for probability calibration, and it is necessary to preserve the fine mapping relationship to capture subtle preference differences. Buffer: satisfies The marginal values ​​in this region are moderate, and the model can initially distinguish between the two types of responses, but gradient saturation is prone to occur, so linearization is needed to maintain a stable optimization signal. Confidence zone: satisfies The marginal value in this region is too large. Continuing to optimize it may lead to overfitting and excessive deviation from the reference model. It is necessary to suppress over-optimization by truncation. Step S2: Constructing the regional boundary mapping function; Step S3: TruncPO loss function optimization and model preference alignment inference.

2. The large model preference optimization method based on reward marginal constraints according to claim 1, characterized in that... ， 3. The large model preference optimization method based on reward marginal constraints according to claim 2, characterized in that, Step S2 includes three sub-steps: sigmoid mapping for the uncertainty region, linearization mapping for the buffer, and truncation mapping for the confidence region. S2.1 Uncertainty Region Sigmoid Mapping: To address the characteristics of the uncertainty region—small marginal differences in preferences and the need for precise probability calibration—the standard sigmoid function is retained. As a mapping relationship, it enables precise matching between probability distribution and human preferences; S2.2 Buffer Linearization Mapping: To address the gradient saturation problem of the sigmoid function in the region of moderate marginal values, a dynamic truncation threshold is applied to the sigmoid function. The first-order Taylor expansion at the point is used as a linear substitution function, i.e. ,Right now For the sigmoid function in The derivative at point A realizes the transformation from a nonlinear mapping to a linear mapping. - Continuous transition; S2.3 Confidence Region Truncation Mapping: To address the risk of overoptimization caused by excessively large confidence region margin values, the mapping result is saturated and truncated, i.e., when... season .

4. The large model preference optimization method based on reward marginal constraints according to claim 3, characterized in that, Step S3 includes three sub-steps: loss function construction, parameter optimization, and alignment inference. S3.1 Loss Function Construction: Piecewise mapping function constructed in step S2 With the core, For the sigmoid function in Using the derivative at a given point, construct the TruncPO loss function. ,in Indicates in the preference dataset D Above, for all preference samples that follow the data distribution Take the expected value; S3.2 Parameter Optimization: The AdamW optimizer is used to optimize the policy model. The parameters are updated during training, and only the policy parameters related to the current preference pair are updated, referencing the model. The parameters are kept frozen to maintain the stability of the optimization baseline; S3.3 Alignment Inference: After model optimization, prompts are provided for new input instructions. First, multiple candidate responses are generated to form a set. To accurately assess the preference level of each candidate response, the marginal value between each candidate response and the reference model is calculated. ,in The preset baseline response; through the piecewise mapping function in step S2 Calculate the marginal value of each candidate response relative to the reference model. The preference probability is calculated, and the response with the highest probability is selected as the final output.

5. The large model preference optimization method based on reward marginal constraints according to claim 2, characterized in that, The temperature hyperparameter The value of needs to be dynamically adjusted according to the model size; EMA smoothing coefficient The value is fixed at 0.

9.

6. The large model preference optimization method based on reward marginal constraints according to claim 3, characterized in that, The intersection of the linear mapping Satisfying linear mapping function ,Right now , For the standard sigmoid function; and After substitution, it is simplified to .

7. The large model preference optimization method based on reward marginal constraints according to claim 4, characterized in that, Step S3.3 introduces length control and style control mechanisms: length control corrects the marginal value by normalizing the response length. ,in This represents the scaling reward margin for each preference pair. express The length of the response is adjusted to avoid alignment bias caused by model preference for response length; style control uses a pre-trained style classifier to select candidate responses that match the target style, ensuring preference alignment while meeting scenario-specific style requirements, and further improving user experience.

8. A large model preference optimization system based on reward marginal constraints, used to implement the large model preference optimization method based on reward marginal constraints as described in any one of claims 1-7, characterized in that: It includes a strategy model, a reference model, and a preference dataset; the strategy model includes LLaMA3-8B-Instruct, Mistral2-7B-Instruct, and Gemma2-9B-Instruct; the reference model uses the same architecture as the strategy model for initial instruction fine-tuning checkpoint; the preference dataset includes the UltraFeedback derivative dataset, the HumanEval preference dataset, and the industry-customized preference dataset.

Images