Reinforced learning algorithm LAPO based on combined reward model

Abstract

The invention discloses a reinforcement learning algorithm LAPO based on a combined reward model. In order to solve the problem that an existing reward model carries out evaluation from a single dimension, and safety and generation quality are difficult to consider at the same time, a combined reward model is provided, and two evaluation results of toxicity judgment and text quality judgment are fused. In addition, by introducing a toxicity discrimination strategy oriented to a multi-round dialogue scene, the reward model further enhances the overall toxicity control ability of the model in a multi-round interaction process. On the basis, aiming at the problems of unstable training and gradient signal attenuation existing in a long sequence generation task, the invention provides an LAPO reinforcement learning algorithm, and the algorithm retains KL divergence and adopts a token-by-token strategy gradient loss calculation mode to realize stable optimization of a strategy model and improvement of long-distance dependence modeling capability. The invention provides a new design thought for solving the problem that toxicity control and text quality are difficult to consider in the text generation process in a multi-round dialogue scene.

Description

Technical Field

[0001] This invention belongs to the field of controlled text generation, and particularly relates to reinforcement learning, toxicity detection and control, and other related fields. Background Technology

[0002] In recent years, toxicity control in multi-turn dialogues has been a key bottleneck for the practical application of intelligent interactive systems. Research shows that toxic content exhibits a cumulative amplification effect in multi-turn interactions—a single-turn low-toxic response, after 3-5 rounds of contextual reinforcement, can increase the probability of serious violations to 4.2 times the initial value, while traditional single-turn filtering methods (such as keyword blocking) have a defense success rate of less than 20% against this type of progressively toxic induction. An even more serious challenge lies in the "toxicity-fluency paradox": strict toxicity suppression often leads to stiff responses (such as repeating "cannot answer"), reducing dialogue fluency by more than 35%, while an excessive pursuit of naturalness may allow latent toxicity to infiltrate. This contradiction is particularly prominent in the current era of large-scale models.

[0003] To address these challenges, reinforcement learning offers a novel approach. By designing structured reward functions, reinforcement learning can transform the abstract "safety-fluency" balance into a quantifiable optimization objective. For example, the output of a toxicity classifier can be used as a penalty, context similarity scores as fluency rewards, and policy matching metrics to control the dialogue flow. Compared to the "passive fitting" of supervised learning, RL can proactively explore "how to adjust the strength of toxicity control in real time during dynamic dialogues," making it particularly suitable for dynamic constraint updates in multi-turn scenarios.

[0004] Current reward models rely on a single evaluation dimension, making it difficult to balance toxicity control and text quality during the optimization process. A more reasonable reward model needs to be designed to balance toxicity control and text quality requirements within the reward feedback dimension.

[0005] In multi-turn dialogue detoxification scenarios, GRPO suffers from the problem of long sequences contributing too little to the gradient. In the GRPO training framework, policy updates primarily rely on an advantage signal constructed based on relative rewards within groups. This advantage is typically normalized at the sequence level and used for calculating the policy gradient of the entire sequence. While this design effectively reduces gradient variance caused by differences in reward scale, it also reveals limitations when handling long sequence generation tasks. Specifically, when the generated sequence is long, the advantage signal needs to be distributed across a large number of tokens over time, significantly diluting the actual gradient weights received by each token, thus reducing the contribution of individual tokens to the overall policy update. This is especially true in scenarios with long texts or long inference chains, where there is a significant time interval between the tokens in the first half of the sequence and the final reward, making their gradient signals more susceptible to weakening by discounting and normalization operations. This uneven gradient distribution slows down the model's convergence speed when optimizing long sequence structures, making it difficult to effectively strengthen key inference steps or intermediate decision nodes. Therefore, despite its advantages in overall stability and variance control, GRPO may still face problems of weak gradient signals and limited optimization efficiency in long sequence modeling tasks.

[0006] DAPO's high diversity makes it well-suited for training long-chain reasoning models. By directly constructing advantage signals for policy optimization, DAPO weakens or even completely removes the dependence on explicit value functions, allowing the model greater freedom in policy updates during training. This design objectively enhances the diversity of model outputs, making it easier for the model to explore different generation paths and reasoning methods under the same input conditions. Compared to optimization strategies like GRPO that emphasize intra-group relative convergence, DAPO tends to encourage the parallel existence of multiple potentially reasonable solutions, thus avoiding premature convergence to a single generation pattern. This strong diversity is particularly important for long-chain reasoning tasks, as complex reasoning processes often lack a single intermediate reasoning path; different reasoning sequences and unfolding methods can all lead to the correct conclusion. DAPO allows the model to fully explore multi-step reasoning structures during training, helping the model gradually learn how to organize and extend long-distance logical dependencies. Therefore, compared to optimization methods that emphasize stable convergence and relative comparison, DAPO is more adaptable and has potential advantages in training long-chain reasoning models.

[0007] To address the aforementioned issues, research on a new reinforcement learning algorithm, LAPO, based on a combined reward model, is essential. Summary of the Invention

[0008] This invention proposes a reinforcement learning algorithm, LAPO, based on a combined reward model. Compared to previous reward models that reward from a single dimension, this model simultaneously utilizes the results of toxicity and text quality assessments, adjusting the weights of these two assessment metrics based on the reward value. This ensures that the policy model controls toxicity without compromising text quality. Furthermore, this combined reward model enhances toxicity control in multi-turn dialogues through a toxicity assessment strategy. This invention also proposes the LAPO reinforcement learning algorithm for multi-turn dialogue applications. This algorithm combines the advantages of GRPO and DAPO algorithms, preserving KL divergence to ensure the stability of the policy model; and utilizes token-by-token policy gradient loss to address the issue of long sequences contributing too little to the gradient.

[0009] This invention proposes a reinforcement learning algorithm, LAPO, based on a combined reward model, comprising the following steps:

[0010] 1) Sample Construction. Based on the output of the policy model, a trajectory containing state and action is obtained. This trajectory represents a complete sampling, that is, the large model generates a complete sentence.

[0011] For a question in a single round of dialogue, four trajectories are generated.

[0012] 2) The four trajectories are processed using a reward model to calculate the corresponding reward R. For toxicity control, not only is the output content of the current round considered, but historical context information is also incorporated to form a complete context-output sequence, allowing for a more comprehensive evaluation of the model's toxicity control performance in multi-round interaction scenarios. A combined reward is generated for both toxicity and text quality through a toxicity-biased combination strategy.

[0013] 3) Calculate intra-group advantage based on rewards

[0014] 4) Optimize the model based on the loss function of the LAPO algorithm. After optimization, replace the parameters of the model network.

[0015] Furthermore, in step 1), the policy model LLaMA-7B was used for output, and a subset of instruction tasks based on Alpaca-GPT4 was selected for the experiment.

[0016] Furthermore, in step 2), to simultaneously constrain the toxicity risk and text quality of the generated content during model training, a joint reward modeling mechanism is constructed. First, regarding toxicity assessment, a pre-trained toxicity classifier is used to discriminate the model output, resulting in a toxicity score T. t ∈[0,1], where a higher score indicates a greater potential toxicity risk in the text, and is determined by weight w. tThe strength of the toxicity constraint is adjusted. The toxicity classifier generates the toxicity of the current dialogue and the toxicity of multiple turns of dialogue, and the final toxicity score is the maximum value between the two.

[0017] Secondly, regarding text quality assessment, a quality evaluation benchmark was constructed based on a subset (2k samples) of Alpaca-GPT4 instruction tasks. GPT-4 was used as the evaluation model to comprehensively judge the generated text from dimensions such as task completion, semantic relevance, and reasoning correctness. A continuous scoring method was adopted for text quality, and the scoring direction was uniformly adjusted to ensure a consistent quality score Q. t Consistent with the toxicity score, where a higher value indicates poorer text quality, the corresponding control weight is w. q

[0018] Furthermore, in step 4), to avoid the training process becoming unstable due to excessive differences in the distribution of the old and new policies, we explicitly retained the KL divergence constraint in the optimization objective to limit the policy update step size, thereby ensuring that the model can maintain a smooth and controllable evolution trajectory during the iteration process.

[0019] Meanwhile, to address the issue of low propagation efficiency of rewards over time in long sequences, this paper adopts a token-by-token strategy for gradient loss calculation. This allows each generated token to directly participate in the gradient update process, significantly enhancing the gradient contribution of key tokens in long sequences. This fine-grained optimization mechanism effectively alleviates the gradient decay problem in long text generation tasks, enabling the model to more fully learn long-distance dependencies and complex generation structures.

[0020] Through the above improvements, the proposed method significantly improves the optimization efficiency and overall performance of the model in long sequence generation scenarios while ensuring training stability.

[0021] The method of this invention can effectively control the toxicity of multi-turn dialogue models, and has the following advantages compared with the prior art:

[0022] 1. This invention designs a combined reward modeling mechanism that introduces toxicity risk constraints in both single-turn and multi-turn dialogue scenarios when evaluating model output, and achieves a synergistic balance between toxicity suppression and text generation quality, thereby effectively improving the safety and usability of the model in complex interaction scenarios.

[0023] 2. To address the common problems of gradient signal decay and unstable policy updates in long sequence generation tasks, this invention introduces KL divergence constraints during the optimization process and combines them with a token-by-token policy gradient loss calculation method to finely control the policy update process, enabling the model to more fully capture long-distance dependencies and stably learn complex generation structures. Attached Figure Description

[0024] Figure 1 This is a diagram illustrating the overall framework of the reinforcement learning algorithm LAPO, which is part of the combined reward model of this invention. The policy model, centered on a large language model, generates corresponding output sequences based on the input context information and obtains the logits of the action distribution through the output layer. The output results are fed into the reward model for evaluation. The reward model simultaneously combines historical context outputs with the current generation results to comprehensively judge the generated content.

[0025] The reward model includes a toxicity discrimination module and a text quality discrimination module, which output toxicity scores and quality scores respectively. The two are then weighted and fused according to preset weight coefficients to obtain the final combined reward signal.

[0026] The combined reward serves as an optimization objective, and is fed back to the strategy model through the LAPO algorithm to update the model parameters. This ensures the stability of the strategy while achieving synergistic optimization of the toxicity risk of the generated content and the quality of the text, ultimately outputting text results that meet both safety and quality requirements.

[0027] Figure 2 This is a schematic diagram of the LAPO algorithm. Based on the output of the policy model, four trajectories containing states and actions are obtained. Based on the output of the reward model, the inter-group advantage of each trajectory is obtained. Finally, the loss value is obtained by combining the KL divergence and the policy gradient loss per token to optimize the policy model. Detailed Implementation

[0028] To make the above-mentioned features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings. The specific process is as follows: Figure 1 As shown, its main steps include:

[0029] Step 1: Based on the output of the policy model, obtain the trajectory containing the state and action, and construct the sample.

[0030] Step 2: Obtain the corresponding reward based on the reward model.

[0031] Step 201, Toxicity Score: Using a pre-trained toxicity classifier, the output probability value T t ∈[0,1], a higher score indicates a greater toxicity risk. The toxicity classifier generates the toxicity T for the current dialogue. cur Toxicity T in multiple rounds of dialogue sum Final toxicity score T t =max(T) cur ,T sum The weight of the toxicity score is w. t .

[0032] Step 202, Text Quality Score: To align with the toxicity scoring direction, this paper unifies the direction of the quality score, making a higher score indicate worse text quality. During training, a quality control weight w is introduced. q .

[0033] Step 203: The combined reward model employs the following strategy to balance detoxification and text quality:

[0034] w t +w q =1, if T t >=0.6, w t =0.8, w q =0.2; if T t <0.6, w t =0.5, w q =0.5. The final reward R = w t ×T t +w q ×Q t

[0035] Step 3: Calculate the inter-group advantage, substitute it into the loss function of the LAPO algorithm, and optimize the model.

[0036] Step 301: When constructing the loss function, we preserved the KL divergence and combined it with the token-by-token gradient loss strategy to construct the loss function of the LAPO algorithm.

[0037]

[0038] Step 4: After each sample participates in 4 parameter updates, the parameters of the entire network are updated.

[0039] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A reinforcement learning algorithm based on a combined reward model, LAPO (Dialogue Advantage Policy Optimization), characterized in that, include: A. A combined reward model ensures simultaneous improvement in text quality while conducting multiple rounds of toxicity control; B. The reinforcement learning algorithm LAPO combines the advantages of GRPO and DAPO algorithms. It ensures the stability of the policy model by preserving KL divergence and uses token-by-token policy gradient loss to address the problem of long sequences contributing too little to the gradient. For a large language model with poison control requirements in a multi-turn dialogue scenario, this algorithm has better training performance.

2. The LAPO reinforcement learning algorithm based on a combined reward model according to claim 1, characterized in that, A further includes the following: A1. Toxicity Score: Utilizes a pre-trained toxicity classifier, outputting the probability value T. t ∈[0,1], the higher the score, the greater the risk of toxicity. The toxicity classifier generates the toxicity T for the current dialogue. cur Toxicity T in multiple rounds of dialogue sum Final toxicity score T t =max(T) cur ,T sum The weight of the toxicity score is w. t . A2. Text Quality Score: A text quality assessment dataset was constructed based on a subset (2k samples) of Alpaca-GPT4 instruction tasks. To avoid the computational overhead and evaluation bias caused by using a very large language model for review, we trained a lightweight text quality discrimination model to perform automated quality evaluation on the model output. This discriminative model takes instructions and generated text as input, comprehensively judges the output quality from three dimensions: task completion, semantic relevance, and reasoning consistency, and outputs a text quality score Q using a continuous scoring mechanism. t . To align with the toxicity scoring direction, this paper unifies the direction of the quality score, making a higher score indicate poorer text quality. During training, a quality control weight w is introduced. q . A3. The combined reward model employs the following strategy to balance detoxification and text quality: In t +in q =1; If T t >=0.6, w t =0.8, w q =0.2; If T t <0.6, w t =0.5, w q =0.5; The final reward is: R = w t ×T t +w q ×Q t .

3. The LAPO reinforcement learning algorithm based on a combined reward model according to claim 1, characterized in that, The loss function combines KL divergence with loss calculated using the token-wise policy gradient. B further includes the following: B1. Sample Construction. Based on the output of the policy model, a trajectory containing the state and action is obtained. This trajectory represents a complete sample, meaning the large model generates a complete sentence. τ=(s0,a0,s1,a1,...,s T-1 ,a T-1 ) The trajectory is sampled from the policy and consists of a series of states and actions. Therefore, the probability of sampling a complete trajectory from the policy model is: For a question in a round of dialogue, generate 4 answers τ1...τ4. B2. The strategy for calculating advantage is as follows: 1) According to the reward model described in claim 2, the rewards R(τ) for the four complete trajectories are obtained. The optimization objective is to maximize the expected reward of the trajectory: After calculating the gradient, input it into B1 as P(τ|π) θ The expression is obtained as follows: 2) Calculate the group's strength A based on rewards. i : As can be seen, R(τ) represents the reward for the entire trajectory. Directly using the reward will introduce state-independent bias and lead to high gradient variance. Therefore, by constructing relative advantage within the group, the absolute reward is transformed into a relative improvement signal, thereby achieving stable and directional policy optimization. in, mean(R(τ i )) refers to the average reward, std(R(τ) i )) refers to the variance of the reward. B3. When constructing the loss function, we preserved the KL divergence and combined it with a token-by-token gradient loss strategy to construct the loss function of the LAPO algorithm. The specific process is as follows: As can be seen from the above formula, the trajectory is obtained by sampling from the current policy, that is, the gradient is calculated and the model is optimized using the data generated by the current policy. This is a same-policy model, where the data generated each time is only used once, which is very wasteful. In order to use the data generated by the old strategy model to optimize the current model, we need to introduce importance sampling, so that the optimization objective becomes as follows: At this point, the trajectory is sampled from the old strategy model and can be used to optimize the current model by adjusting a scaling factor. When DAPO uses reinforcement learning to train long-chain reasoning models, the model needs to explore more freely, and its distribution may deviate significantly from the initial model's distribution. In this case, limiting it with KL divergence is unnecessary. However, in scenarios where the output should conform to human preferences, it shouldn't deviate too far from the original model, and KL divergence is needed to constrain it. In GRPO, the loss for each sample is the average of the token losses within the sample. This ignores the effect of length, meaning that short and long samples contribute to the final loss with the same weight. However, long samples have more tokens, which can lead to a low contribution of tokens in long samples to the overall loss. This may result in the inability to reward high-quality long samples and penalize low-quality short samples. We take the average of all tokens in the group. Longer samples, because they have more tokens, will contribute more to the final loss, which is beneficial for detoxification optimization in multi-turn dialogues. B4. After updating the parameters four times for each sample, update the parameters of the entire network and replace θ. old .

Images