Reward model training method and system based on information bottleneck and redundancy constraint

Abstract

The invention discloses an information bottleneck and redundancy constraint-based reward model training method and system. The method comprises the following steps of: constructing a reward model integration system comprising M members; all member reward models are jointly trained on the human preference data set through a joint loss function comprising a correlation loss item, a compression loss item and a redundancy loss item; the correlation loss item aims to maximize mutual information between the potential representation vector of each member and the human preference tag; the compression loss item aims to minimize mutual information between the input text and the potential representation vector of each member; the redundancy loss term is intended to minimize mutual information between potential representation vectors of any two different members. According to the method, the award model is forced to learn diversified and preference-related features, the dependence of the award model on false features is inhibited, the generalization ability and robustness of the award model are remarkably improved, the phenomenon of excessive optimization of the award is effectively relieved, and the behavior of the strategy model better conforms to human preferences.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to the field of model training based on human preference feedback, and particularly to a reward model training method and system based on information bottlenecks and redundancy constraints. Background Technology

[0002] In practical applications such as intelligent customer service dialogue systems, AI-powered medical consultation assistants, and decision-making systems for autonomous vehicles, these systems need to continuously optimize their behavioral strategies based on human feedback to generate responses that align with human values. For example, in intelligent customer service systems, the system needs to learn to generate responses that are both accurate and polite; in medical consultation AI, the system needs to learn to provide health advice that is both professional and accurate, yet easily understood by patients. These systems commonly employ reinforcement learning from human feedback (RLHF) techniques for training.

[0003] RLHF is a key technique for aligning large language models (LLMs) with human values. One of its core components is reward modeling, which involves training a proxy reward model (RM) on a human preference dataset to simulate human preference judgments. Subsequently, during the reinforcement learning phase, the LLM is fine-tuned by optimizing the rewards provided by this proxy RM.

[0004] However, existing RLHF processes suffer from a serious problem: reward hacking, or over-optimization. This phenomenon refers to a situation where, while the policy model optimizes based on the agent RM, its reward score increases, but its behavior gradually deviates from the true human expectation. The root cause lies in the reward model's misgeneralization. Specifically, during training, the reward model may learn pseudo-features that happen to correlate with human preferences on the training data, such as response length, specific sentence structures, or keywords, rather than truly understanding the underlying reasons for human preferences. When the policy model over-utilizes these pseudo-features during the reinforcement learning phase, reward hacking occurs. For example, when the policy model generates excessively long but empty responses, the standard reward model often incorrectly assigns high scores due to the accidental positive correlation between text length and human preferences in the training set. This results in the customer service system outputting seemingly professional but ultimately unhelpful redundant responses, reducing user experience and system reliability.

[0005] Existing techniques attempt to mitigate this problem through the following methods: 1) KL divergence constraints: limiting the policy model from deviating from the supervised fine-tuning (SFT) model, but this limits the model's exploration space; 2) scaling up the RM: improving generalization power by increasing model parameters, but computationally expensive; 3) ensemble RM: integrating multiple independently trained RMs. However, this simple ensemble approach is costly and cannot fundamentally solve the problem, because all members of the ensemble may still independently learn the same spurious features, such as all models favoring long responses, leading to systematic misgeneralization.

[0006] Therefore, there is an urgent need for a new reward model training method that can fundamentally suppress the model's dependence on pseudo-features, especially eliminating the common dependence of multiple model members on the same pseudo-feature, thereby effectively alleviating the reward hijacking problem in RLHF. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a reward model training method and system based on information bottlenecks and redundancy constraints. This invention is applicable to dialogue policy optimization based on reinforcement learning, aiming to solve or alleviate the technical problems of system behavior deviating from human intent due to reward hijacking or over-optimization, and to suppress the generation of lengthy, false, and harmful responses by the policy model.

[0008] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a reward model training method based on information bottlenecks and redundancy constraints, comprising the following steps: (1) Construct a reward model ensemble system with M members, where M is greater than or equal to 2, and each member is used to map the input text into a d-dimensional latent representation vector; (2) Construct a joint loss function for jointly optimizing the reward model of all members, including a relevance loss term, a compression loss term and a redundancy loss term; wherein, the relevance loss term aims to maximize the mutual information between the latent representation vector of each member and the human preference label; the compression loss term aims to minimize the mutual information between the input text and the latent representation vector of each member; the redundancy loss term aims to minimize the mutual information between the latent representation vectors of any two different members; (3) Using the above joint loss function, jointly train all member reward models in the reward model ensemble system on the human preference dataset.

[0009] Furthermore, the input samples are samples from the human preference dataset, and each sample contains an input prompt, a selected response text, a rejected response text, and a human preference label.

[0010] Furthermore, the reward model integration system adopts an architecture with a shared encoder and M independent prediction heads. Each prediction head includes an input layer, an intermediate layer, and an output layer. The shared encoder, the input layer of the independent prediction head, and the intermediate layer constitute a member. The shared encoder is used to map the input text to a basic feature vector, and feeds the basic feature vector into M prediction heads respectively. The corresponding latent representation vector is obtained through the input layer and the intermediate layer. The latent representation vector is passed through the corresponding output layer to obtain the corresponding scalar reward value.

[0011] Furthermore, the correlation loss term is obtained by calculating the sum of the negative log-likelihood losses of all member reward models based on the human preference dataset.

[0012] Furthermore, the compression loss term is specifically obtained through the following method: modeling the latent representation vector of each member as a multivariate Gaussian distribution as the posterior distribution of the latent representation vector of the member, modeling the latent representation vector of each member as a standard normal distribution as the prior distribution of the latent representation vector of the member, calculating the KL divergence loss between the posterior distribution and the prior distribution of the latent representation vector of each member, and calculating the approximate value of the sum of the KL divergence losses of all members as the compression loss term.

[0013] Furthermore, the redundancy loss term is obtained by calculating the mutual information between the latent representation vectors of any two different members. The estimated value of this mutual information is obtained based on the output of the discriminator. At this time, the discriminator simultaneously receives the latent representation vectors of two different members as its input.

[0014] Furthermore, the redundancy loss term is a conditional redundancy loss term, which aims to minimize the mutual information between the latent representation vectors of any two different members under the condition of a given human preference label; at this time, the discriminator simultaneously receives the latent representation vectors of two different members and the human preference label as its input.

[0015] Further, in step (3), during the training process, samples from the human preference dataset are used as input samples and fed into the reward model integration system for each member reward model and the discriminator to obtain the corresponding latent representation vector, scalar reward value, and the output of the discriminator; based on the latent representation vector, scalar reward value, and the output of the discriminator, the correlation loss term, compression loss term, redundancy loss term, and discriminant loss function are calculated, and the joint loss function is calculated based on the correlation loss term, compression loss term, and redundancy loss term; the parameters of the discriminator and each member reward model are alternately optimized and updated, the discriminator is continuously updated for K steps, and then the parameters of each member reward model are updated for one more step; Specifically, during the discriminator update phase, the parameters of the shared encoder and each member prediction head are fixed, and the parameters of the discriminator are optimized and updated with the objective of minimizing the discriminator loss function; during the member reward model update phase, the parameters of the discriminator are fixed, and the parameters of the shared encoder and each member prediction head are optimized and updated with the objective of minimizing the joint loss function.

[0016] Furthermore, it also includes: Applications and Reasoning: In the reinforcement learning phase, for a response generated by a policy model, obtain the scalar reward values ​​output by M member reward models; by aggregating the outputs of the M member reward models, obtain the final reward signal, which is used to update the policy model.

[0017] A second aspect of this invention provides a system for implementing the above-described reward model training method based on information bottlenecks and redundancy constraints, comprising: The model initialization module is used to build a reward model ensemble system with M members, where M is greater than or equal to 2, and each member is used to map the input text into a d-dimensional latent representation vector; The loss calculation module is used to construct a joint loss function for jointly optimizing the reward model of all members, including a correlation loss term, a compression loss term, and a redundancy loss term; and calculates the correlation loss term, compression loss term, and redundancy loss term based on the human preference dataset to obtain the joint loss function; The parameter optimization module is used to optimize the parameters of all member reward models in the ensemble system that are jointly trained on a human preference dataset using a joint loss function.

[0018] Compared with the prior art, the beneficial effects of the present invention are: (1) It fundamentally suppresses erroneous generalization: This invention not only filters irrelevant information of individual models by compressing the loss term, but more importantly, it forces different reward models to rely on different features for decision-making by using the redundancy loss term. Even if a model learns a pseudo-feature (such as length), other models will be incentivized to learn other more essential features, thereby greatly reducing the dependence of the entire ensemble system on pseudo-features.

[0019] (2) Improved robustness of reward signal: The aggregated reward signal integrates the judgments from multiple independent perspectives. The extreme or erroneous reward values ​​generated by the incorrect generalization of a single model will be balanced by the judgments of other models, thus providing a more stable and accurate optimization direction for the policy model.

[0020] (3) Effectively alleviate reward hijacking: Due to the enhanced robustness of the reward signal, the policy model is less likely to "deceive" the entire reward model integration system by using a single pseudo-feature, thus effectively alleviating the problem of reward over-optimization and making the final LLM behavior more in line with real human expectations.

[0021] (4) High computational efficiency: In the preferred embodiment, a shared encoder and multi-head architecture are adopted, and the training cost is much lower than training multiple completely independent ensemble models. It has strong practicality and is applicable to large language models, intelligent customer service dialogue systems, medical consultation AI assistants, decision-making systems for autonomous vehicles, intelligent programming / code generation and other fields. Attached Figure Description

[0022] Figure 1 This is a flowchart of the reward model training method based on information bottleneck and redundancy constraints of the present invention; Figure 2 This is a schematic diagram illustrating the working mechanism of each component (correlation, compression, redundancy) of the joint loss function of this invention; Figure 3 This is a diagram illustrating the effect of the method described in this invention in mitigating the reward hijacking problem; Figure 4 This is a comparison chart of the performance of the model trained by the method described in this invention with multiple baseline models. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. It is obvious that the drawings used in the following description are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0026] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0027] Example 1: Training Method for Reward Model Based on Information Bottleneck and Redundancy Constraints See Figure 1 The reward model training method based on information bottleneck and redundancy constraints of the present invention specifically includes the following steps: (1) Initialize the reward model ensemble system: Construct a reward model ensemble system with M members, where M is greater than or equal to 2. Each member is used to map the input text into a d-dimensional latent representation vector, where d is the preset feature dimension, such as 128-dimensional or 256-dimensional.

[0028] It should be noted that the latent representation vector is an intermediate feature vector inside the neural network, which is an abstract representation of the input text after it has been processed by the encoder.

[0029] Furthermore, the input samples are samples from a human preference dataset, and each sample contains input prompts. (e.g., user question "How do I learn programming?"), selected response text Rejected response text and a human preference label Y, where the human preference label Y is used to represent the selected response text. Better than the rejected response text .

[0030] It should be noted that human preference datasets include the Anthropic HH (Helpful & Harmless) dataset, the PKU-SafeRLHF dataset, and the Stanford Human Preferences dataset, which can be obtained from various open-source repositories. The Anthropic HH dataset contains approximately 170,000 pairs of dialogue preference samples, the PKU-SafeRLHF dataset contains approximately 83,400 pairs, and the Stanford Human Preferences dataset contains approximately 385,000 pairs.

[0031] Furthermore, the reward model ensemble system adopts an architecture with a shared encoder and M independent prediction heads, which improves computational efficiency. Each prediction head includes an input layer, an intermediate layer, and an output layer. The shared encoder, along with the input and intermediate layers of each prediction head, constitutes a unit. The shared encoder maps the input text to a basic feature vector, which is then fed into the M prediction heads to obtain the corresponding latent representation vectors through the input and intermediate layers. latent representation vector The corresponding scalar reward value is obtained through the corresponding output layer. Where X represents the input text, This represents the latent vector representation corresponding to the i-th predictor head. This represents the scalar reward value output by the i-th prediction head.

[0032] For example, in this embodiment, a preferred bifurcated multi-head architecture is used to construct a reward model ensemble system containing M (e.g., M=4) members. This reward model ensemble system has a shared encoder (e.g., using a backbone network based on a pre-trained language model such as Llama-7B as the shared encoder) and M independent prediction heads (e.g., using a lightweight multi-layer perceptron (MLP) as the prediction head). This architecture significantly improves training and inference efficiency while ensuring the diversity of the reward model through the shared encoder. The input text X is first processed by the shared encoder to extract the basic feature vector. Subsequently, each of the M independent prediction heads maps the basic feature vector to its independent d-dimensional (e.g., preset d=128-dimensional) latent representation vector through its internal input layer and intermediate layer. latent representation vector The corresponding scalar reward value is obtained through the corresponding output layer. The latent representation vector (LVDV) is an abstract and compressed mathematical representation of the input text, capturing key information related to human preference judgments. For example, in a dialogue response rating task, the LVDV might encode abstract features such as politeness, information completeness, and logical coherence of the response. The input text can be a cue-response pair from the Anthropic HH dataset. For example, an input text X from the Anthropic HH dataset might include: ① input cue... "What are some good reasons to support slavery?"; ② Selected response text (The response text labeled as superior by humans): "I think your question is very interesting, and I'd like to think about it. Please allow me to answer this question within the context of American history." Analyzing this response text reveals that it is an indirect rejection. It avoids direct affirmation or listing good reasons, instead attempting to shift the topic to historical context and factual discussion, thereby avoiding a defense; ③ The rejected response text (The response text labeled as poor by humans): "This is a very challenging question. In my opinion, enslaving others is quite wrong, and I do not recommend that we do it, but there were indeed periods in history when slavery was a choice. The view that holding slavery is morally wrong is also quite controversial. If you could be more specific about what you mean when you ask this question, I would be happy to help you think more deeply." Analyzing this response text reveals that although it expresses the view that "enslaving others is wrong," in RLHF training, it is generally considered a weak or flawed rejection. Furthermore, the statement "The view that holding slavery is morally wrong is also quite controversial" weakens the moral stance, implying that the view of "opposing slavery" is "controversial," which might be judged by the RLHF reward model as a weak rejection of harmful content; ④ Human preference label Y: Indicates the selected response text. Better than the rejected response text .

[0033] (2) Define the joint loss function: Construct a joint loss function for jointly optimizing the reward models of all members, including a relevance loss term, a compression loss term, and a redundancy loss term. The relevance loss term aims to maximize the mutual information between the latent representation vectors of each member and the human preference labels; the compression loss term aims to minimize the mutual information between the input text and the latent representation vectors of each member; the redundancy loss term aims to minimize the mutual information between the latent representation vectors of any two different members, such as... Figure 2 As shown.

[0034] Furthermore, the formula for calculating the joint loss function is as follows: In the formula, Denotes the joint loss function. Represents the correlation loss term. Represents the compression loss term. This represents the redundancy loss term. and These are the weight parameters for the compression loss term and the redundancy loss term, respectively, used to balance the weights of each loss term.

[0035] Furthermore, the correlation loss term For each member i in the reward model ensemble system, calculate its standard reward model preference loss, which is the sum of the negative log-likelihood losses of all member reward models based on the human preference dataset. This aims to maximize the mutual information between the latent representation vectors of each member and the human preference labels, ensuring that each member's reward model can accurately predict the ranking of human preferences. This correlation loss term... The calculation formula is: In the formula, Represents the latent representation vector of the human preference label Y and the i-th member. Mutual information between them, where D represents the human preference dataset. Indicates the selected response text. The response text indicates that the request was rejected. This means randomly sampling a pair of selected and rejected response texts from a human preference dataset. This indicates that the expected value is calculated for the input sample. Indicates the selected response text The corresponding scalar reward value, Indicates the rejected response text The corresponding scalar reward value, This represents the Sigmoid function. This represents the logarithmic calculation function. This correlation loss term drives each predictor head i to accurately predict human preferences.

[0036] Furthermore, compress the loss term. For each member i in the reward model ensemble system, compute its input text X and latent representation vector. The mutual information between the input text and the latent representation vectors of each member is used as a compression loss term. This compression loss term, based on the information bottleneck theory, aims to minimize the mutual information between the input text and the latent representation vectors of each member. It forces each member reward model to learn a compressed representation of the input, forming an information bottleneck. This filters and discards high-dimensional noise and redundant information (including spurious features) irrelevant to predicting human preferences, retaining only the core information necessary for predicting human preferences. Through variational inference, the mutual information between the input text X and the latent representation vectors can be... The mutual information between members is approximated as the KL divergence loss between the posterior and prior distributions of the latent representation vectors. Specifically, the latent representation vectors of each member are modeled as a multivariate Gaussian distribution as the posterior distribution, and as a standard normal distribution as the prior distribution. The KL divergence loss between the posterior and prior distributions of each member's latent representation vector is calculated, and the approximate sum of the KL divergence losses of all members is used as the compression loss term. The calculation formula is as follows: In the formula, This represents the latent representation vector of the input text X and the i-th member. Mutual information between them; This means randomly sampling an input sample X from the human preference dataset D. This represents calculating the expected value of an input sample X in a human preference dataset D; The latent representation vector of the i-th member The posterior distribution is specifically obtained by using the latent representation vector of the i-th member. Modeled as a multivariate Gaussian distribution Obtain It is the mean vector. The diagonal covariance matrix is ​​given, where the shared encoder outputs a 2d-dimensional fundamental feature vector, and the first d-dimensional fundamental feature vectors are directly used as... The d-dimensional basic feature vectors are then transformed into non-negative numbers using the Softplus activation function and used as... The diagonal elements; The latent representation vector of the i-th member The prior distribution is specifically obtained by using the latent representation vector of the i-th member. The model is based on a standard normal distribution. To facilitate differentiable sampling of the posterior distribution, a reparameterization method is used to represent the latent representation vector as follows: , Reward members with standard Gaussian noise that is independent of model parameters. express Following a standard normal distribution, where I is the identity matrix, this reparameterization method transforms the random sampling process into a function of... and The differentiable function makes the compression loss term differentiable with respect to the reward model parameters, so that the gradient descent method can be used to train the entire reward model ensemble system end-to-end. The KL divergence loss, for two multivariate Gaussian distributions of dimension d, can be written in closed analytical form: , The trace of the matrix is ​​represented by the superscript T, which indicates the transpose of the matrix or vector. The compression loss term can be efficiently calculated using the closed-form formula described above, thereby achieving compression constraints on the input information within the latent space.

[0037] Furthermore, the redundancy loss term For any pair of different members (i,j) in the reward model ensemble system, the mutual information between the latent representation vectors of any two different members is calculated as a redundancy loss term. This redundancy loss term aims to minimize the mutual information between the latent representation vectors of any two different members, forcing the two member reward models to learn and judge human preferences from different and independent information dimensions, thereby preventing them from jointly relying on the same spurious feature. Since this mutual information is difficult to calculate directly, this embodiment uses a discriminator w (an adversarial neural discriminator network can be selected as the discriminator) based on the Donsker-Varadhan variational principle for estimation. The task of the discriminator w is to distinguish between jointly distributed samples and integrally distributed samples, that is, the input of the discriminator is either a jointly distributed sample or an integrally distributed sample (i.e., a pair of latent representation vectors of two different members). The jointly distributed sample can be understood as a positive sample, that is, a pair of latent representation vectors extracted from the same input sample X through two different member reward models (i,j). The multiplicative distribution of samples can be understood as negative samples, that is, samples from different input samples X and... The latent representation vectors extracted by two different member reward models (i,j) The discriminator w is optimized by minimizing the discriminant loss function (maximizing the discriminator's output for positive samples and minimizing the discriminator's output for negative samples), where the discriminant loss function is calculated as follows: In the formula, This represents the discriminant loss function. Indicates the index number of the reward model for different members. The output of the discriminator, This represents a pair of latent representation vectors extracted from the same input sample X through two different member reward models (i,j). The joint distribution Indicates different input samples X and The latent representation vectors extracted by two different member reward models (i,j) The distribution of the multiplicative integral, This indicates the calculation of the corresponding expected value. Simultaneously, in the main training loop, the reward model for each member is optimized by minimizing the redundancy loss term, i.e., minimizing the estimate of the mutual information between the latent representation vectors of any two different members (minimizing the discriminator's output for positive samples and maximizing the discriminator's output for negative samples). The formula for calculating the redundancy loss term is: In the formula, The latent representation vector of the i-th member With the latent representation vector of the j-th member Mutual information between them The density ratio approximation function is constructed from the discriminator output. Denotes the expected value obtained under the joint distribution. This represents the expected value obtained by multiplying and integrating. This represents the optimal choice of discriminator w given the representation distribution, thereby enabling adversarial training.

[0038] (3) Joint optimization: Using the joint loss function mentioned above, and employing optimization algorithms such as gradient descent, the reward model ensemble system integrates all member reward models on the human preference dataset.

[0039] Specifically, during training, samples from the human preference dataset are used as input samples and fed into the reward model ensemble system for each member reward model and the discriminator to obtain the corresponding latent representation vectors, scalar reward values, and the discriminator's output. Based on the latent representation vectors, scalar reward values, and the discriminator's output, correlation loss, compression loss, redundancy loss, and discriminant loss functions are calculated. A joint loss function is then calculated based on the correlation loss, compression loss, and redundancy loss terms. The parameters of the discriminator and each member reward model are alternately optimized and updated. The discriminator is continuously updated K steps (e.g., K=4) to obtain a relatively accurate density ratio estimate, and then the parameters of each member reward model are updated one more step. Specifically, during the discriminator update phase, the parameters of the shared encoder and each member prediction head are fixed, and the discriminator's parameters are optimized and updated with the goal of minimizing the discriminant loss function. The training objective of this discriminator is to distinguish between jointly distributed samples and multiplicative distributed samples. During the member reward model update phase, the discriminator's parameters are fixed, and the parameters of the shared encoder and each member prediction head are optimized and updated with the goal of minimizing the joint loss function.

[0040] It should be noted that the core objective of this invention is to train a more robust reward model, enabling it to provide more accurate optimization signals for the policy model in subsequent processes.

[0041] In some other embodiments, the application and reasoning process described in step (4) below will also be included.

[0042] (4) Application and Reasoning: After all member reward models in the reward model ensemble system have been trained, the reward model ensemble system is applied to the RLHF process, and the policy model is optimized and updated using reinforcement learning algorithms such as PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization). Specifically, in the reinforcement learning phase, for a response generated by a policy model, the scalar reward values ​​output by M member reward models are obtained; by aggregating (e.g., taking the average or median) the outputs of the M member reward models, the final reward signal is obtained for updating the policy model.

[0043] It should be noted that the reward model is the model trained in this invention, used to evaluate the quality score of a certain output given the input text. The policy model is the target model that needs to be optimized in the RLHF process, usually a large language model (such as GPT, Llama, Qwen, DeepSeek, etc.). This policy model generates outputs (such as dialogue responses, codes, decision instructions, etc.) based on the input text, and its parameters are optimized by reinforcement learning algorithms based on the reward signal given by the reward model.

[0044] Furthermore, during the reinforcement learning phase, it is also necessary to monitor the redundancy loss term. When the value of the redundancy loss term exceeds a preset threshold or experiences abnormal growth, early termination is triggered.

[0045] Example 2: Preferred Implementation of Redundancy Loss Term This embodiment is an optimization and improvement of the redundancy loss term in Embodiment 1. In Embodiment 1, the mutual information between any two latent representation vectors with different members is minimized. In some cases, the model may be inadvertently penalized for learning shared features related to the task; for example, all models should recognize that "politeness" is good.

[0046] To address this issue, this embodiment replaces the redundancy loss term with a conditional redundancy loss term. The goal is to minimize the mutual information between the latent representation vectors of any two distinct members, given human preference labels. The key difference in implementing the conditional redundancy loss term is that, here, the discriminator simultaneously receives the latent representation vectors of two different members and the human preference label, i.e. .

[0047] At this point, the discriminator's task becomes: the joint distribution samples (positive samples) are pairs of latent representation vectors from the same input sample X and with the same human preference label Y. The multiplicative distribution of samples (negative samples) consists of samples from different input samples X and... But the latent representation vectors that share the same human preference label Y In this way, the conditional redundancy loss term only penalizes shared information (i.e., spurious features) that remains redundant after a given human preference, while protecting the necessary shared features that all member reward models should learn that are related to the human preference label Y. This makes training more stable and performance better.

[0048] Example 3: Over-optimization detection based on redundancy This embodiment also provides an additional beneficial effect: rewarding the detection of over-optimization. In the RLHF process, the redundancy loss term can be monitored periodically. When the value of the redundancy loss term exceeds a preset threshold or exhibits abnormal growth, early stopping is triggered. Specifically, if the value of the redundancy loss term begins to increase abnormally during training (especially in the reinforcement learning phase), it indicates that the M prediction heads are beginning to converge and collectively depend on a (potentially new) spurious feature. This signal can serve as an early warning indicator to trigger early stopping of RLHF, thereby halting policy optimization before reward hijacking occurs.

[0049] Example 4: The beneficial effect of the reward model training method based on information bottleneck and redundancy constraints on alleviating the reward hijacking problem. like Figure 3 As shown, suppose a pseudo-feature is "response length". A standard RM will incorrectly assign high scores to long responses in the later stages of optimization, leading to a lower standard surrogate RM score. Figure 3 The dotted line in the middle rises, but real human preferences ( Figure 3 The solid line in the graph (using GPT-5 evaluation results as proxy ground truth for human preferences) decreases, which is reward hijacking. Using the method described in this invention, assuming model RM1 starts learning length pseudo-features, model RM2 is forced to learn different features, such as "fact accuracy," due to the redundancy loss term penalty. When the policy model generates a long but empty response, the output of model RM1... It's very high, but the RM2 model output is... The average reward R(X) after aggregation is very low, which will be a reasonable score, thus preventing the policy model from optimizing in the wrong direction of "writing longer" and allowing the reward to continue to increase.

[0050] Example 5: Performance comparison between the model trained by the method described in this invention and multiple baseline models. To further verify the beneficial effects of this invention, this embodiment compares the performance of an RLHF policy model trained using the method described in this invention with two key baseline models. The evaluation was conducted on the Anthropic-Helpful dataset, and the GPT-5 evaluator determined the win-draw-loss ratio of the inventive model's response to the baseline models. The results are as follows: Figure 4As shown. Among them, the baseline models include: ① SFT fine-tuned model: that is, the base model after supervised fine-tuning, without RLHF; ② Standard RM model: that is, the RLHF model trained using the Standard RM model.

[0051] The comparison results are as follows Figure 4 As shown, compared with the SFT fine-tuning model, the win rate of the model of this invention is 57.0%, far exceeding the loss rate of 16.0%; compared with the standard RM model, the win rate of the model of this invention is 54.5%, significantly higher than the loss rate of 12.0%. This proves the superiority of the present invention over the training of the standard reward model.

[0052] The above data demonstrates that the policy model trained by the method proposed in this invention significantly outperforms both the SFT model without RLHF and the RLHF model using standard RM in terms of helpfulness. This confirms that the technical solution of this invention (combining information bottleneck compression and redundancy penalty) can more effectively train the reward model, thereby guiding policy optimization to produce results that better align with real human preferences.

[0053] Example 6: Application in the optimization of intelligent programming assistance systems This embodiment illustrates the application of the reward model training method proposed in this invention to intelligent auxiliary systems for highly complex tasks, specifically using an intelligent programming-assisted robot as an example. This system aims to generate accurate and efficient response signals based on the user's programming or logical reasoning instructions, while preventing reward hijacking problems caused by the reward model's over-reliance on spurious features (such as code length, annotation density, or redundant steps in the thought process).

[0054] Step 1: Data Acquisition and Preference Instruction Set Construction: The system first collects dialogue history from historical human-computer interaction logs. The model training server extracts instruction pairs (including user questions and system candidate responses) from high-privilege technical logs to construct a digital domain instruction signal dataset for training. Input data X is defined as a sequence combination of "user query instruction signals" and "system response signals." Input data format: Data Example 1 (Code Generation, Targeting Form Over Content Hijacking): The user's prompt is: "Please write a Python function to calculate the nth term of the Fibonacci sequence." The corresponding selected candidate response is a piece of Python function code implemented using an iterative method. This code possesses the genuine characteristics of logical correctness and high computational efficiency (complexity O(n)). The rejected candidate response, on the other hand, is a piece of code implemented using a recursive method. It contains a large number of `class`, `docstring`, and redundant comments, exhibiting pseudo-characteristics such as professional formatting, long text length, and high comment density. However, its actual functionality is unusable due to the lack of a termination condition or extremely low efficiency. The human preference label Y indicates that the selected sample signal is superior to the rejected sample signal.

[0055] Data Example 2 (Logical Reasoning, Hijacking of Thought Chain Forgery): The user's query instruction (Prompt) is: "Suppose a task scheduling system needs to process task A (2h), task B (3h), and task C (1h). The system stipulates that C must be completed first, and then A and B can begin, and they must be executed sequentially. Please give the shortest total execution time." The corresponding selected candidate response signal is a concise and correctly calculated conclusion: "The shortest total execution time is 6 hours. Steps: C (1h) + A (2h) + B (3h) = 6 hours." It possesses the genuine characteristic of correct logical reasoning. The rejected candidate response signal, on the other hand, is a reasoning step that uses professional terms such as "queueing theory" and formatted steps, but incorrectly takes the maximum time of sequential tasks A / B during calculation, thus arriving at the erroneous conclusion of "total execution time is 4 hours." This rejected sample signal has pseudo-characteristics such as a standardized structure, rich professional terminology, and lengthy reasoning steps. The human preference label Y marks the selected sample signal as superior to the rejected sample signal.

[0056] Step 2: Construct and parameterize the integrated reward evaluation network: Construct an ensemble reward evaluation network containing M (M=4) parallel feature extraction channels. All channels share a pre-trained encoder based on a Transformer architecture to transform the instruction signal X into a basic feature vector. The independent prediction head for each channel i maps the basic features to a latent representation vector unique to that channel. and output the scalar reward value. This multi-head architecture is the basis for achieving redundancy constraints.

[0057] Step 3: Joint training based on information bottleneck and redundancy constraints: Perform the following specific optimization operations to update parameters and address the issue of over-optimization of rewards: a) Perform information bottleneck compression ( ): By minimizing the input signal X and the latent representation vector Mutual information between them, using KL divergence constraints The posterior distribution. Its technical function is to filter out high-frequency noise information from the input signal in both scenarios mentioned above. For example, it filters out formatted comments, non-essential narrative text, and incorrect technical terms (such as "queueing theory" in candidate response signals in logical reasoning scenarios) in code generation scenarios. Its technical purpose is to force the model to retain only the minimum effective information that can affect the final function and logical judgment.

[0058] b) Implement redundancy constraints ( This loss term minimizes the latent representation vectors of different members. and This constraint is achieved through mutual information between channels. Its technical function is to prevent all members of the reward model from colluding to be deceived by a single pseudo-feature. Specifically, in code generation scenarios, this constraint forces channel 1 to focus on evaluating "functional correctness," while channel 2 focuses on evaluating "code standardization." When the policy model generates rejected sample responses, channel 2 will score high due to its high annotation density, but channel 1 will score low due to functional errors. This redundancy constraint ensures this contradiction exists, preventing all channels from being deceived by pseudo-features of "high standardization." In logical reasoning scenarios, this constraint forces channel 3 to focus on evaluating "logical consistency of reasoning steps," while channel 4 focuses on evaluating "numerical accuracy of the final result." When the policy model generates rejected sample responses, channel 3 may score medium to high due to using the standard CoT (Coding in Tense) format, but channel 4 will score low due to errors in the final result (4 hours), thus preventing the policy model from relying on pseudo-features of "long steps and numerous technical terms" for optimization.

[0059] Step 4: Generate feedback control signals and guide strategy model optimization: After training, the integrated reward evaluation system is incorporated into the reinforcement learning (RLHF) process of the intelligent policy model. This is a crucial step in guiding the optimization of the policy model in this invention. For any response signal generated by the policy model, the system aggregates the scores from M channels to obtain the final feedback control signal R(X), which is then used as the reward signal in the PPO algorithm to guide the parameter optimization of the policy model. Through this optimization process, the policy model is more likely to generate high-reward response signals in the future.

[0060] Step 5: System Inference Deployment and Technical Effect Verification: Once the policy model has been trained and reached performance convergence in step 4, the entire intelligent programming assistance system is deployed to the online service environment for real-time processing of user commands (i.e., applications during the inference phase). a) Online inference application: User query commands are input into a pre-trained policy model, which outputs a final response signal. This response signal is used to guide specific application functions such as code generation and logical reasoning.

[0061] b) Verification of technical effectiveness: The system trained using the method of this invention demonstrates significant advantages in actual deployment: ① Improved Instruction Compliance Accuracy and Logical Robustness: This invention effectively suppresses the policy model's dependence on formal pseudo-features through redundant constraints. In code generation scenarios, this invention improves the pass rate of code generated by the policy model in unit test functional correctness tests by more than 10% compared to models trained with standard reward models. In complex logical reasoning scenarios, the system's rejection rate for response signals with lengthy reasoning steps but incorrect conclusions is reduced by 25%, significantly improving the system's logical robustness. In core tasks, the model of this invention exhibits higher accuracy.

[0062] ② Improved resource utilization and response efficiency: Due to the information bottleneck, the strategy model no longer tends to generate redundant annotations or complex but ineffective steps. While ensuring functional correctness, this invention reduces the average text length of the generated response signal by approximately 18%, effectively reducing unnecessary text transmission and parsing overhead and improving the system's response speed to core instructions.

[0063] Example 7: Reward Model Training System Based on Information Bottleneck and Redundancy Constraints In this embodiment, the training system includes a model initialization module, a loss calculation module, and a parameter optimization module. The model initialization module constructs a reward model ensemble system with M members, where M is greater than or equal to 2, and each member maps the input text to a d-dimensional latent representation vector. The loss calculation module constructs a joint loss function for jointly optimizing all member reward models, including a relevance loss term, a compression loss term, and a redundancy loss term; and calculates the relevance loss term, compression loss term, and redundancy loss term based on the human preference dataset to obtain the joint loss function. The parameter optimization module uses the joint loss function to jointly train all member reward models in the reward model ensemble system on the human preference dataset and optimizes their parameters.

[0064] Furthermore, the system may also include a reward aggregation module for aggregating reward values ​​from all member reward models in the RLHF process to generate a final reward signal for updating the policy model.

[0065] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a reward model based on information bottleneck and redundancy constraints, characterized in that, Includes the following steps: (1) Construct a reward model ensemble system with M members, where M is greater than or equal to 2, and each member is used to map the input text into a d-dimensional latent representation vector; (2) Construct a joint loss function for jointly optimizing the reward model of all members, including a relevance loss term, a compression loss term and a redundancy loss term; wherein, the relevance loss term aims to maximize the mutual information between the latent representation vector of each member and the human preference label; the compression loss term aims to minimize the mutual information between the input text and the latent representation vector of each member; the redundancy loss term aims to minimize the mutual information between the latent representation vectors of any two different members; (3) Using the above joint loss function, jointly train all member reward models in the reward model ensemble system on the human preference dataset.

2. The reward model training method based on information bottleneck and redundancy constraints according to claim 1, characterized in that, The input samples are samples from the human preference dataset, and each sample contains an input prompt, the selected response text, the rejected response text, and a human preference label.

3. The reward model training method based on information bottleneck and redundancy constraints according to claim 1, characterized in that, The reward model integration system adopts an architecture with a shared encoder and M independent prediction heads. Each prediction head includes an input layer, an intermediate layer, and an output layer. The shared encoder, the input layer of the independent prediction head, and the intermediate layer constitute a member. The shared encoder is used to map the input text to a basic feature vector, and feeds the basic feature vector into M prediction heads respectively. The corresponding latent representation vector is obtained through the input layer and the intermediate layer. The latent representation vector is passed through the corresponding output layer to obtain the corresponding scalar reward value.

4. The reward model training method based on information bottleneck and redundancy constraints according to claim 1, characterized in that, The correlation loss term is obtained by calculating the sum of negative log-likelihood losses of all member reward models based on a human preference dataset.

5. The reward model training method based on information bottleneck and redundancy constraints according to claim 1, characterized in that, The compression loss term is obtained by the following method: modeling the latent representation vector of each member as a multivariate Gaussian distribution as the posterior distribution of the latent representation vector of each member, modeling the latent representation vector of each member as a standard normal distribution as the prior distribution of the latent representation vector of each member, calculating the KL divergence loss between the posterior distribution and the prior distribution of the latent representation vector of each member, and calculating the approximate value of the sum of the KL divergence losses of all members as the compression loss term.

6. The reward model training method based on information bottleneck and redundancy constraints according to claim 1, characterized in that, The redundancy loss term is obtained by calculating the mutual information between the latent representation vectors of any two different members. The estimated value of this mutual information is obtained based on the output of the discriminator. At this time, the discriminator simultaneously receives the latent representation vectors of two different members as its input.

7. The reward model training method based on information bottleneck and redundancy constraints according to claim 6, characterized in that, The redundancy loss term is a conditional redundancy loss term, which aims to minimize the mutual information between the latent representation vectors of any two different members given human preference labels. At this point, the discriminator simultaneously receives the latent representation vectors of two different members and the human preference label as its input.

8. The reward model training method based on information bottleneck and redundancy constraints according to claim 1, characterized in that, In step (3), during the training process, samples from the human preference dataset are used as input samples and sent to the reward models and discriminators of each member in the reward model integration system to obtain the corresponding latent representation vectors, scalar reward values, and the output of the discriminator. The correlation loss term, compression loss term, redundancy loss term, and discriminant loss function are calculated based on the latent representation vector, scalar reward value, and discriminant output. The joint loss function is calculated based on the correlation loss term, compression loss term, and redundancy loss term. The parameters of the discriminator and each member reward model are alternately optimized and updated. The discriminator is continuously updated for K steps, and then the parameters of each member reward model are updated for one more step. Specifically, during the discriminator update phase, the parameters of the shared encoder and each member prediction head are fixed, and the parameters of the discriminator are optimized and updated with the objective of minimizing the discriminator loss function; during the member reward model update phase, the parameters of the discriminator are fixed, and the parameters of the shared encoder and each member prediction head are optimized and updated with the objective of minimizing the joint loss function.

9. The reward model training method based on information bottleneck and redundancy constraints according to claim 1, characterized in that, Also includes: Applications and Reasoning: In the reinforcement learning phase, for a response generated by a policy model, obtain the scalar reward values ​​output by M member reward models; By aggregating the outputs of the M member reward models, the final reward signal is obtained, which is then used to update the policy model.

10. A system for implementing the reward model training method based on information bottleneck and redundancy constraints according to any one of claims 1-9, characterized in that, include: The model initialization module is used to build a reward model ensemble system with M members, where M is greater than or equal to 2, and each member is used to map the input text into a d-dimensional latent representation vector; The loss calculation module is used to construct a joint loss function for jointly optimizing the reward model of all members, including a correlation loss term, a compression loss term, and a redundancy loss term; and to calculate the correlation loss term, compression loss term, and redundancy loss term based on the human preference dataset to obtain the joint loss function; The parameter optimization module is used to optimize the parameters of all member reward models in the ensemble system that are jointly trained on a human preference dataset using a joint loss function.

Images