A large model toxicity mitigation method based on toxicity-aware reward and dynamic temperature adjustment collaborative optimization

Abstract

The application discloses a large model toxicity mitigation method based on toxicity-aware reward and dynamic temperature regulation collaborative optimization, which evaluates the toxicity level of generated text by constructing a sequence-aware reward model, designs an entropy regularization mechanism to prevent candidate path collapse, and introduces a dynamic temperature regulation strategy to realize adaptive sampling control. It includes: reward model architecture design and sequence-aware training, multi-layer Transformer decoder and linear projection layer to realize text to toxicity reward score mapping, combined with Shannon entropy monitoring, logarithmic probability penalty stable distribution, based on real-time toxicity linear adjustment temperature and smooth sampling. The method realizes Top-K candidate reward through multi-GPU parallel computing, fuses original logit and reward value to generate enhanced probability distribution, and balances safety and diversity relying on a double-layer collaborative optimization framework. The application effectively mitigates the path collapse and semantic degradation problem, and provides an efficient solution for the large model content safety field.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and natural language processing, and particularly relates to large language model reinforcement learning, reward modeling, text toxicity mitigation generation and other related fields. Background Technology

[0002] In recent years, with the rapid development of large language model technology, generative artificial intelligence, represented by ChatGPT, has demonstrated powerful capabilities in various fields. However, large language models learn from massive amounts of data on the internet during the pre-training stage, inevitably absorbing toxic content, biased information, and inappropriate expressions. These problems can lead to the generation of harmful, offensive, or incompatible text in practical applications, seriously affecting user experience and social impact.

[0003] To address the content security issues of large language models, text detoxification has gradually become an important research direction in the field of natural language processing. Controlling toxic text typically employs methods such as post-processing filtering, model retraining, or introducing security constraints. However, these methods mostly have significant drawbacks: post-processing filtering, while simple and direct, cannot intervene at the generation source and is prone to missing hidden toxic expressions; retraining models requires a large amount of cleaned, safe corpus, which is costly and may impair the model's generality; while introducing hard security constraints can reduce toxicity, it often leads to overly conservative generated content, resulting in semantic poverty, repetition, and monotony.

[0004] In early work on text toxicity mitigation generation, rule-based filtering methods were commonly used, decomposing the detoxication task into two independent subtasks: toxicity detection and content replacement. However, this method has a significant drawback: it is difficult to maintain generation quality while ensuring security. To achieve a balance between security and fluency, researchers have proposed reward-guided decoding strategies in recent years. For example, reward-enhanced decoding methods reduce the probability of generating toxic content by adjusting the logit score of tokens, showing good performance and improving detoxication to some extent, but they cannot effectively handle the "path collapse" problem. However, in practical applications, over-detoxication often leads to monotonous generation and semantic degradation. Over-optimization of rewards causes the model to repeatedly select a very small number of low-toxicity tokens, resulting in a lack of diversity and semantic coherence in the output content. This imbalance greatly limits the practical value of detoxication methods. Furthermore, there are relatively few adaptive decoding strategies currently used in the field of text toxicity mitigation. According to research, methods that can simultaneously balance security, diversity, and fluency are still lacking, making it difficult to meet the growing application demands in the field of large-scale model content security.

[0005] Given that real-time toxicity feedback is crucial for adjusting decoding strategies, detoxification models need to dynamically adjust sampling intensity based on the risk level during generation. Traditional methods, employing fixed temperature parameters or reward weights, cannot adaptively respond to toxicity fluctuations in different scenarios, resulting in insufficient detoxification in high-risk scenarios and overly conservative approaches in low-risk scenarios. Therefore, maintaining generation diversity and semantic coherence while ensuring content security, and avoiding path collapse and semantic degradation, is a critical issue that urgently needs to be addressed.

[0006] To address the aforementioned issues, research into novel collaborative optimization methods for generating text toxicity mitigation algorithms for large-scale models is essential. Summary of the Invention

[0007] This invention proposes a large-scale toxicity mitigation method based on toxicity perception reward and dynamic temperature regulation co-optimization. It evaluates the toxicity level of generated text by constructing a reward model, designs a distribution adjustment mechanism based on entropy regularization to prevent candidate path collapse, introduces a dynamic temperature regulation strategy to achieve adaptive sampling control, and balances security and diversity through a two-layer co-optimization framework. At the same time, it adopts an exponential moving average smoothing mechanism to avoid sampling instability, effectively improving the quality and security of large-scale text toxicity mitigation generation.

[0008] This invention proposes a large-scale toxicity mitigation method based on the synergistic optimization of toxicity perception reward and dynamic temperature regulation, comprising the following steps:

[0009] 1) Reward model architecture design and sequence awareness training: The input data in the form of sequences are used as training samples. The input data contains text sequences with different toxicity levels and the sequence length is T. The reward model sequentially passes through a token embedding layer, a position encoding layer, a multi-layer Transformer decoder block, a Dropout layer and a linear projection layer to complete the mapping from text sequences to one-dimensional toxicity reward scores. The training process aims to minimize the error between the real toxicity label and the model's predicted reward score.

[0010] 2) Entropy Regularization and Dynamic Temperature Adjustment Mechanism: After reward-enhanced decoding, Shannon entropy monitoring is performed on the candidate token distribution. The flatness of the distribution is quantified by calculating normalized entropy. A log-probability-based penalty strategy is used to apply negative adjustments to high-probability candidates. In the next stage, the regularized logits are incorporated into the sampling process to prevent path collapse caused by excessively sharp distributions. Simultaneously, a dynamic temperature adjustment strategy is introduced to achieve adaptive sampling control. The target temperature parameter is linearly adjusted based on the toxicity level of the currently generated sequence to balance exploration and exploitation. An exponential moving average smoothing mechanism is used in temperature updates to avoid sampling abrupt changes, improving the model's adaptability to different risk scenarios.

[0011] 3) Parallel Reward-Enhanced Decoding Based on Top-K Candidates: In each generation step, the language model first generates the Top-K most likely subsequent token candidates. The system distributes these candidate sequences in parallel to multiple GPU devices, where the reward model simultaneously calculates the expected reward for each candidate. Subsequently, based on a dynamically determined adjustment strength, the original log-probability of each candidate is weighted and fused with its corresponding reward value to obtain the enhanced probability distribution, and the next token is generated accordingly. This process iterates until generation is complete.

[0012] 4) Two-layer collaborative optimization framework and secure decoding strategy: During the generation stage, entropy regularization and dynamic temperature adjustment are unified in the same decoding framework. The regularized logits are sampled by temperature scaling and input into the Softmax layer. In actual deployment, the toxicity threshold and regularization parameters are set according to the application scenario to ensure that the detoxicated generation maintains semantic fluency and generation diversity while ensuring content security.

[0013] Furthermore, the construction process of the reward model in step 1) specifically includes the following sub-steps:

[0014] a. Input sequence encoding: The input text sequence is processed by a token embedding layer to map each token into a vector of fixed dimensions, and then the sequence position information is injected through a position encoding layer to generate an embedding representation of dimension T×d, where d is the dimension of the hidden layer.

[0015] b. Deep semantic feature extraction: The embedding represents the input of a multi-layer Transformer decoder block. Each decoder block contains a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism captures the dependencies between tokens in the sequence, and the feedforward neural network performs a non-linear transformation on the attention output features. The deep semantic features of the sequence are extracted layer by layer through multi-layer stacking.

[0016] c. Reward score prediction: Deep semantic features are regularized through a Dropout layer to prevent model overfitting; then, a linear projection layer maps the d-dimensional features to a one-dimensional reward prediction value; the model extracts the output of the last valid token position of each sample as the final reward score r of the sequence, realizing the capture of sequence-level toxic reward signals.

[0017] Further, the entropy regularization mechanism in step 2) consists of three stages: First, after reward enhancement, the candidate distribution is monitored for entropy and regularized; then, adaptive temperature parameters are calculated based on the real-time toxicity level; finally, the regularized logits are input into the sampler after temperature scaling. Specifically, for each decoding step t, the adjusted logits are first normalized using Softmax to obtain the probability distribution p_t, and then the Shannon entropy H(p_t) of this distribution is calculated, where the normalized entropy H_norm quantifies the flatness of the distribution. When the normalized entropy is lower than a preset threshold, the distribution is considered too sharp and regularization needs to be applied. This invention adopts a penalty strategy based on log probability: negative adjustment is applied to the top K / 5 candidates with the highest probability to weaken their dominance in sampling. The adjustment intensity is determined by the entropy deviation, where α is the basic regularization weight. Negative penalty is applied to high-probability candidates to make their scores regress to the average level. The mathematical essence of this mechanism is to guide the distribution towards the direction of maximum entropy through gradient, thereby leveling the differences in candidate probabilities and avoiding a single path dominating the sampling process.

[0018] Furthermore, in step 2), a dynamic temperature adjustment mechanism is introduced based on entropy regularization to adjust the sampling strategy according to real-time toxicity feedback. The core idea is to dynamically adjust the influence of the reward signal on the logits distribution based on the uncertainty of the language model's current output distribution. When the language model uncertainty is high, the weight of the reward signal is appropriately reduced to encourage exploration; when the uncertainty is low, the reward guidance is enhanced, thus establishing a dynamic balance between detoxication optimization and language diversity. When the toxicity exceeds a threshold, the target temperature decreases linearly with increasing toxicity, making the sampling distribution sharper and prioritizing low-toxicity candidates. To avoid sampling instability caused by temperature abrupt changes, an exponential moving average (EMA) is used for smoothing the transition, where β is the smoothing coefficient. This mechanism effectively avoids the excessive greed or excessive randomization problems that may result from a fixed-weight scheme.

[0019] Furthermore, the parallel reward-enhanced decoding strategy based on Top-K candidates in step 3) is a key engineering optimization in this work. In each generation step, the standard language model first generates the Top-K most likely subsequent token candidates; then, the system distributes these candidate sequences in parallel to multiple GPU devices, where the reward model simultaneously calculates the expected reward and uncertainty estimate for each candidate. This parallel computing architecture significantly improves the efficiency of large-scale text generation. Next, the system employs a linear reward enhancement mechanism, weightedly fusing the log-odds probability of the original language model for each candidate token with the expected reward from the reward model. This mechanism injects the long-term reward signal into the generation decision in real time, enabling the model to prioritize text paths that maintain contextual coherence and lead to high final rewards. Finally, the system samples and generates the next token based on the enhanced probability distribution, iteratively executing the above process until generation is complete.

[0020] Furthermore, the innovative design of the two-layer collaborative optimization framework in step 4) lies in unifying distribution pattern adjustment and sampling intensity control within the same framework. It adaptively adjusts the decoding strategy based on the real-time generation status, preventing path collapse caused by over-optimization of rewards and achieving a dynamic exploration-exploitation balance based on risk levels. This ensures that depoisoning generation maintains semantic fluency and generation diversity while guaranteeing content security. The framework prevents candidate distribution collapse to a single path through entropy regularization, maintaining generation diversity; and achieves adaptive sampling control through dynamic temperature adjustment, adjusting the exploration-exploitation balance according to the real-time toxicity level. These two mechanisms work synergistically to ensure content security while avoiding semantic degradation caused by excessive conservatism.

[0021] The method of this invention can effectively construct reward models for large-scale text toxicity mitigation generation, and has the following advantages compared with existing technologies:

[0022] 1. This invention designs a two-layer adaptive control mechanism of entropy regularization and dynamic temperature regulation, unifying distribution pattern adjustment and sampling intensity control within the same framework; through entropy monitoring and regularization, it effectively solves the path collapse problem caused by reward over-optimization, maintaining the diversity of generated content; based on the dynamic temperature regulation of real-time toxicity level, it realizes the adaptive adjustment of exploration-exploitation balance, ensuring low toxicity while avoiding semantic degradation, and achieving synergistic optimization of security and generation quality.

[0023] 2. This invention designs a parallel reward-enhanced decoding architecture based on Top-K candidates. By using multiple GPUs to compute the expected reward of each candidate in parallel, it significantly improves the efficiency of large-scale text generation. Simultaneously, it employs a sequence-aware reward model training strategy, assigning differentiated weights to predictions at different positions in the sequence, enabling the model to more accurately learn the toxicity assessment task and improving the stability and reliability of detoxification generation in practical applications.

[0024] 3. The proposed two-layer collaborative optimization framework unifies entropy regularization and dynamic temperature regulation within the same decoding process. By sampling the regularized logits after temperature scaling, a balance between security and diversity is achieved. This framework prevents distribution collapse at the source and achieves adaptive risk management through dynamic regulation, providing a solution that balances performance and efficiency for the field of large model content security. Attached Figure Description

[0025] Figure 1 This is a flowchart of the large-scale toxicity mitigation method based on the synergistic optimization of toxicity perception reward and dynamic temperature regulation according to the present invention.

[0026] Figure 2 This is a diagram illustrating the overall framework of the large-scale toxicity mitigation method based on toxicity-sensing reward and dynamic temperature regulation co-optimization of the present invention. It mainly consists of three key technical components: reward model architecture and training strategy, adaptive temperature regulation mechanism based on information entropy, and multi-GPU parallel reward computing architecture. Detailed Implementation

[0027] To make the above-mentioned features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings. The specific process is as follows: Figure 1 As shown, its main steps include:

[0028] Step 101: Based on the pre-trained language model, replace its language modeling head with a single-output linear layer to construct a sequence-level scalar reward predictor and reuse the pre-trained language representation capabilities.

[0029] Step 102: Construct a fine-tuned dataset adapted to a specific quality assessment task, fine-tune the reward predictor end-to-end, and improve data efficiency and generalization ability with the help of transfer learning.

[0030] Step 103: Configure the cumulative mean squared error loss function, assigning linearly increasing weights to the token predictions at different positions in the sequence, with the highest weight given to the predictions at the end of the sequence.

[0031] Step 201: Design an adaptive temperature regulation strategy based on information entropy. According to the uncertainty of the language model output distribution, dynamically adjust the influence of the reward signal on the logits distribution to balance detoxification and diversity.

[0032] Step 202: During the generation process, calculate the information entropy of the language model output distribution in real time to quantify uncertainty.

[0033] Step 203: When information entropy is high, reduce the reward weight to encourage exploration; when entropy is low, increase the reward weight to strengthen guidance.

[0034] Step 301: In each generation step, control the autoregressive language model to generate Top-K subsequent token candidates.

[0035] Step 302: Distribute the candidate sequences to multiple GPUs in parallel and calculate the expected reward and uncertainty of each candidate synchronously through the reward model to improve efficiency.

[0036] Step 303: Adopt a linear reward enhancement mechanism to integrate the original logarithmic probability of candidate tokens with the expected reward.

[0037] Step 304: Generate the next token based on the enhanced probability distribution sampling.

[0038] Step 305: Iterate through the above steps until a complete text sequence that meets the conditions is generated.

[0039] This invention uses publicly available datasets related to text toxicity mitigation and a constructed dataset of diverse toxic text generation for training and validation, and conducts experiments under different toxicity levels and text generation scenarios. Experimental results show that this invention improves the quality assessment adaptability by constructing a sequence-level scalar reward predictor based on a pre-trained language model, combining it with a weighted incremental design of the cumulative mean square error loss function, and introducing an entropy regularization dynamic temperature adjustment strategy. This strategy quantifies the uncertainty of the language model output based on information entropy to dynamically adjust the reward signal weights, and coordinates a reward enhancement real-time guided generation mechanism, achieving synergistic optimization between the reward model and dynamic temperature adjustment. This method effectively alleviates the path collapse problem caused by over-reliance on rewards in large-scale text toxicity mitigation generation, establishes a dynamic balance between detoxication optimization and linguistic diversity, and significantly improves the efficiency of large-scale text generation. It can stably output text content that meets detoxication requirements and possesses both informational value and diversity in various complex text generation scenarios.

[0040] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A large-scale toxicity mitigation method based on the synergistic optimization of toxicity perception reward and dynamic temperature regulation, characterized in that, include: A. Collect text sequences labeled with different toxicity levels as training samples, construct a reward model architecture, extract semantic features through token embedding, position encoding and multi-layer Transformer decoder, obtain reward prediction values ​​through Dropout regularization and linear projection, extract the effective token at the end of the sequence as the toxicity reward score, and train the model using the cumulative mean squared error loss function to minimize the prediction error. B. After decoding the reward-enhanced tokens, monitor the Shannon entropy of the Top-K candidate token distribution and quantify the flatness of the distribution by normalized entropy. When the entropy is lower than the threshold, adopt an adaptive penalty strategy to adjust the scores of high-probability candidates and guide the distribution to move towards the direction of maximum entropy to defend against the path collapse problem. C. Introduce a dynamic temperature adjustment mechanism based on real-time toxicity feedback. Calculate the target temperature according to the toxicity level and smooth the transition through exponential moving average (EMA). Input the regularized logits into Softmax sampling after temperature scaling to balance detoxication optimization and language diversity, thereby improving the model's risk adaptability. D. Distribute the Top-K candidate tokens generated by the language model to multiple GPUs for parallel computation of expected reward and uncertainty; weightedly fuse the original logits and the adjusted logits with the reward, and use co-entropy regularization and dynamic temperature adjustment for sampling generation; uniformly optimize the distribution pattern and sampling intensity to ensure the safety, semantic fluency and diversity of the detoxified generation.

2. The large-scale toxicity mitigation method based on the synergistic optimization of toxicity perception reward and dynamic temperature regulation according to claim 1, characterized in that, Step A further includes the following steps: A1. Training Data Collection and Toxicity Labeling System Construction: Collect publicly available text datasets containing different toxicity levels, and use regular expressions and keyword filtering to identify and label harmful content such as abusive language, discriminatory expressions, and violent incitement; label each text sequence with a continuous toxicity score in the 0-1 interval to construct a toxic text dataset, while retaining some boundary samples to enhance the model's ability to discriminate in ambiguous contexts; A2. Reward Model Architecture Design and Feature Extraction: A reward model is built based on a pre-trained language model. The input sequence is first processed through a token embedding layer to generate an initial embedding vector. ∈R^(T×d_model), where R is the set of real numbers, T is the length of the input text sequence, and d_model is the dimension of the hidden layer; subsequently, positional encoding is added. ∈R^(T×d_model) to obtain the position-aware representation: The position encoding uses a sine-cosine function, The input is a multi-layer Transformer decoder block, each layer of which contains a multi-head self-attention mechanism and a feedforward neural network; A3. Sequence-level reward signal extraction and cumulative loss function design: After obtaining the final hidden state H^L∈R^(T×d_model), where H is the hidden state and L is the total number of decoder layers, a Dropout layer is applied for regularization. A linear projection layer maps the d_model-dimensional features to one-dimensional reward predictions, and the output of the last valid token position for each sample is extracted as the final reward score for the sequence. A cumulative mean squared error loss function is used to assign linearly increasing weights to token predictions at different positions in the sequence, enabling the model to achieve more accurate quality assessment at the end of the sequence. in For sequence length, For sequence position index, For positional weights, Let t be the predicted reward score for position t. This is the true toxicity label. The reward model is trained end-to-end by minimizing this cumulative mean squared error loss function, ultimately constructing a reward model capable of evaluating text toxicity. .

3. The large-scale toxicity mitigation method based on the synergistic optimization of toxicity perception reward and dynamic temperature regulation as described in claim 1, characterized in that, Step B further includes the following steps: B1. Reward-enhanced logits calculation and Top-K candidate selection: At each decoding step t, the language model generates the original logits vector. The reward model calculates the expected reward for each candidate token. The enhanced logits score is obtained by linear weighted fusion: in As a reward scaling factor, As a candidate token, from Select the top-K highest-scoring candidate tokens to form a candidate set; B2. Shannon entropy monitoring and distribution sharpness assessment: Obtain the probability distribution by normalizing the Top-K candidate logits using Softmax. : Calculate the Shannon entropy of this distribution. and normalized entropy : Normalized entropy ∈[0,1] quantifies the flatness of the distribution, when When = 1, the distribution is uniform (maximum entropy). The distribution is highly concentrated when it approaches 0; B3. Adaptive penalty intensity calculation and log probability regularization: setting a normalized entropy threshold ∈(0,1), when < If the distribution is deemed too sharp, regularization needs to be applied; calculate the entropy deviation and determine the adjustment strength: Where α is the basic regularization weight, and λ t It increases with increasing entropy bias; for the top N with the highest probability top =⌊K / 5⌋ candidates are subject to a negative adjustment based on log probability.

4. The large-scale toxicity mitigation method based on the synergistic optimization of toxicity perception reward and dynamic temperature regulation as described in claim 1, characterized in that, Step C further includes the following steps: C1. Real-time toxicity level assessment and target temperature calculation: At each decoding step t, the cumulative toxicity level of the currently generated sequence is assessed using a reward model; the target temperature parameter is calculated linearly based on the relationship between the toxicity level and a preset threshold; firstly, the reward model trained in step A is used... For the current sequence Perform real-time toxicity scoring: in For the currently generated prefix sequence, The toxicity score of the prefix sequence is calculated; then the target temperature parameter is calculated based on a piecewise linear function; when the toxicity level is below the safety threshold, the maximum temperature is maintained to encourage the generation of diversity; when the toxicity level is in the warning range, the temperature decreases linearly with toxicity; when the toxicity level exceeds the danger threshold, the temperature is reduced to the minimum to force risk avoidance. C2. Exponential Moving Average Smoothing Mechanism: To avoid sampling instability and generation quality fluctuations caused by sudden temperature changes, an exponential moving average is used to smooth the target temperature. in ∈(0,1) is the smoothing coefficient, which is usually taken as... =0.3; C3. Temperature Scaling Sampling and Final Probability Distribution Generation: Regularizing the logits scores by entropy Sampling distribution adjustment via temperature scaling: The temperature-scaled logits are input into the Softmax layer to generate the final sampling probability distribution: according to Perform sampling to generate the next token.

5. The large-scale toxicity mitigation method based on the synergistic optimization of toxicity perception reward and dynamic temperature regulation according to claim 1, characterized in that, Step D further includes the following steps: D1. Top-K Candidate Generation and Parallel Sequence Construction: At each decoding step t, the language model LM generates the original logits distribution based on the current context; from this distribution, the top-K candidate tokens with the highest scores are selected to construct K candidate sequences: in The i-th candidate token generated for the language model; D2. Multi-GPU Parallel Reward Computation Architecture: Distribute K candidate sequences evenly across N GPU devices according to batch size, with each GPU processing a subset; execute the forward propagation of the reward model in parallel on each GPU to calculate the expected reward for each candidate; after the parallel computation is completed, the main process aggregates the results of all GPUs to obtain the complete reward vector; D3. Linear reward enhancement and logits fusion strategy: Employing a linear weighting mechanism to integrate the original logits vectors of the language model. With reward model signal To merge: Where β is the reward scaling factor; this mechanism injects the long-term reward signal into the generation decision in real time, so that the model prioritizes the text path that maintains contextual coherence and leads to a high final reward. D4. Two-layer collaborative optimization decoding process: Under the unified decoding framework, entropy regularization and dynamic temperature adjustment constitute a two-layer adaptive system, which combines distribution adjustment and sampling control to balance detoxification security and generation diversity. The decoding step first weights and fuses the language model logits and detoxification reward; then entropy regularization is applied; then the target temperature is calculated and smoothed according to the real-time toxicity, and the sampling is controlled by scaling logits; finally, Softmax sampling is used to generate tokens, and the process is iterated until the end.

Images