A Controllable Text Generation Method Based on Fine-Grained Feedback Reinforcement Learning

Abstract

This invention discloses a controllable text generation method based on fine-grained feedback reinforcement learning, comprising the following steps: 1) For the controllable text generation task, initialize a large policy model m for text generation; initialize a copy of the policy model as a reference model m_ref, an attribute scorer s; also initialize a data pool D, and prepare a prefix corpus for exploration; 2) Exploration; given a prefix x for text generation, use the current policy model to generate subsequent text; 3) Segmentation and noise addition; segment the rewards within D, and inject noise into each reward; 4) Model learning; 5) Repeat steps 2) to 4) using the updated model for exploration until the training reaches its maximum number of iterations; 6) Based on the prefix, use the learned model to generate subsequent text, obtaining the final generated text. This invention designs fine-grained rewards, which can provide more intensive feedback to the model.

Description

Technical Field

[0001] This invention relates to machine learning technology, and more particularly to a controllable text generation method based on fine-grained feedback reinforcement learning. Background Technology

[0002] Large autoregressive language models trained on extensive corpora can generate high-quality text. However, to meet real-world applications, making the generation more controllable is a priority. The aim is to enhance specific properties of the generated text to meet practical needs and reduce the inherent flaws of pre-trained language models (e.g., toxicity, repetition).

[0003] Because the parameter scale becomes enormous, retraining the model (e.g., cocon, ctrl) requires huge computational overhead. Post-processing methods (e.g., gedi, fudge, dexpert) utilize small-scale discriminators to bias the probability distribution of the next output, but this often results in low text quality. Some methods employ small-parameter fine-tuning, such as prefix adjustment, but they are susceptible to unwanted properties in supervised corpora. Recent research (e.g., diffusionlm, Dist-lens, normal flow) introduces non-traditional autoregressive language model frameworks, such as diffusion models and normalized flow, but these typically incur higher computational costs during training and longer inference times, making them difficult to deploy in practical applications.

[0004] Some studies have introduced reinforcement learning (RL) into controlled text generation tasks. The RL paradigm can alleviate the aforementioned problems by training on self-generated sentences to mitigate overfitting and by combining parameter-efficient policies with a canonical LLM backbone. However, RL-based methods typically use sentence-level (or paragraph-level) rewards to update the language model, resulting in suboptimal performance and slow convergence. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a controllable text generation method based on fine-grained feedback reinforcement learning, which addresses the shortcomings of existing technologies.

[0006] The technical solution adopted by this invention to solve its technical problem is: a controllable text generation method based on fine-grained feedback reinforcement learning, comprising the following steps:

[0007] 1) For the controllable text generation task, initialize a large model m for text generation strategy;

[0008] The text generation strategy model m includes partially generated sentences y. <i The next marker information y i Each y <i -y iThe next generated partial sentence y of the guided sequence <i+1 ; the partially generated sentence y <i As a state, the next marker y i As an action, the vocabulary is an action space;

[0009] Initialize a reference model m ref A copy of the strategy model, an attribute scorer s; also initialized a data pool D, and prepared a prefix corpus for exploration;

[0010] Reference model m ref Frozen throughout the process;

[0011] 2) Exploration;

[0012] Given a prefix x for text generation, based on the controllable text generation task, use the current policy model to generate subsequent text.

[0013] For each generated token, calculate the score offset as its reward r(y). i |y <i ), and (y i ,y <i ,r) is added to the data pool D, and a lifetime is set for each piece of data to indicate the events it can still experience;

[0014] 3) Segmentation and noise addition;

[0015] The reward range is divided into intervals by segmenting the reward within D and obtaining the q-quantile; then, noise is injected into each reward while ensuring that each reward remains within the original interval. Through this process, the reward order is disrupted, interfering with the classifier's fixed scoring pattern, while maintaining the relative order between intervals, thus guiding the LLM toward the target attribute.

[0016] 4) Model optimization learning;

[0017] The loss function used during model training is as follows:

[0018] logm(y i |y <i )+aEntropy[p(y i |y <i )]+bKL(m|m ref )

[0019] The minimum objective of the optimization problem is to maximize the total reward.

[0020] By adding the standard maximum entropy gradient Entropy[p(y)] i |y <iTo relax convergence, it can help capture different behavioral patterns, and also inserts KL divergence penalty KL(m|m) ref ), to prevent model m from deviating from the original m ref Too far;

[0021] 5) Repeat steps 2) through 4) to explore using the updated model, and repeat the exploration quantization and noise learning cycles until the training reaches the maximum number of times;

[0022] 6) Based on the prefix, use the learned model to generate subsequent text, and then concatenate them into the final generated text.

[0023] According to the above scheme, in step 2), the reward r(y) for text generation is determined based on the state and operation. i |y <i );

[0024] The fine-grained reward design is as follows:

[0025] r(y i ,y <i ) = logp(c|y <i+1 )-logp(c|y <i )

[0026] Where c represents the specific attribute that you want to enhance in the generated text; y <i+1 For each y <i -y i The next part of the sentence generated by the guide.

[0027] According to the above scheme, in step 2), a lifetime is set for each data point. Specifically, after the data is added to the data pool D, the lifetime is initialized to L and is reduced by 1 each time it is included in the training set. When the lifetime of the data drops to 0, the data will be deleted from the data pool D.

[0028] According to the above scheme, in step 2), if different parts of the input prefix involve different attributes, the contribution of the token to multiple attributes is weighed separately.

[0029] Train a weighing device W to balance the rewards of multiple scorers. Given the last hidden state H of model m, the weighing device output W(H) serves as the weight of the multiple rewards.

[0030] That is, the final reward is

[0031] r1W(H) 1 +r2W(H) 2 +…+r n W(H) n

[0032] Where, r iLet W(H) be the reward corresponding to attribute i. i The corresponding weights;

[0033] The weighing device W does not require a complex model structure. A simple structure is sufficient to help our algorithm achieve excellent performance. In our implementation, the weighing device consists of two linear layers with ReLU functionality and an output layer with softmax functionality.

[0034] The beneficial effects of this invention are:

[0035] 1. Compared with previous reinforcement learning methods, this invention designs fine-grained rewards, which can provide more intensive feedback to the model. At the same time, it can be extended to multiple attribute constraints with minimal computational overhead. Attached Figure Description

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0037] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] like Figure 1 As shown, a controllable text generation method based on fine-grained feedback reinforcement learning includes the following steps:

[0040] 1) For the controllable text generation task, initialize a large model m for text generation strategy;

[0041] The text generation strategy model m includes partially generated sentences y. <i The next marker information y i Each y <i -y i The next generated partial sentence y of the guided sequence <i+1 ;

[0042] A standard Markov Decision Process (MDP) can be represented as (S, A, T, r). At each step, an action a is performed based on the current state s. The state is then transformed to s′ with T(s′|s, a). The function r defines the reward returned based on the state and the action. The policy is determined by a policy model, which is a predicted distribution of actions based on the state. Applying the Markov Decision Process to a text generation scenario, the state can be defined as the partially generated sentence y. <iAnd the action is the next marker y. i The vocabulary is the action space. The transition dynamics T are deterministic because each state-action pair leads to a unique next state.

[0043] The sentence y to be partially generated <i As a state, the next marker y i As an action, the vocabulary is an action space;

[0044] Initialize a reference model m ref A copy of the strategy model, an attribute scorer s; also initialized a data pool D, and prepared a prefix corpus for exploration;

[0045] Reference model m ref Frozen throughout the process;

[0046] 2) Exploration;

[0047] Given a prefix x for text generation, based on the controllable text generation task, use the current policy model to generate subsequent text.

[0048] For each generated token, calculate the score offset as its reward r(y). i |y <i ), and (y i ,y <i ,r) is added to the data pool D, and a lifetime is set for each data point to indicate the events it can still experience; the lifetime is set for each data point as follows: after data is added to the data pool D, the lifetime is initialized to L and is decremented by 1 each time it is included in the training set; when the lifetime of data drops to 0, the data is removed from the data pool D.

[0049] In step 2), the reward r(y) for text generation is determined based on the state and operation. i |y <i );

[0050] The fine-grained reward design is as follows:

[0051] r(y i ,y <i ) = logp(c|y <i+1 )-logp(c|y <i )

[0052] Where c represents the specific attribute that you want to enhance in the generated text; y <i+1 For each y <i -y i The next part of the sentence generated by the guide.

[0053] Existing RL-based methods typically use sentence-level (or paragraph-level) rewards to update the language model, resulting in suboptimal performance and slow convergence. This invention refines the coarse-grained feedback by using partial sentence rewards to update the language model, providing more precise guidance.

[0054] If different parts of the input prefix involve different attributes, then the contribution of the token to multiple attributes is weighed separately.

[0055] Train a weighing device W to balance the rewards of multiple scorers. Given the last hidden state H of model m, the weighing device output W(H) serves as the weight of the multiple rewards.

[0056] That is, the final reward is

[0057] r1W(H) 1 +r2W(H) 2 +…+r n W(H) n

[0058] Where, r i Let W(H) be the reward corresponding to attribute i. i The corresponding weights;

[0059] The weighing device W does not require a complex model structure. A simple structure is sufficient to help our algorithm achieve excellent performance. In our implementation, the weighing device consists of two linear layers with ReLU functionality and an output layer with softmax functionality.

[0060] 3) Segmentation and noise addition;

[0061] The reward range is divided into intervals by segmenting the reward within D and obtaining the q-quantile; then, noise is injected into each reward while ensuring that each reward remains within the original interval. Through this process, the reward order is disrupted, interfering with the classifier's fixed scoring pattern, while maintaining the relative order between intervals, thus guiding the LLM toward the target attribute.

[0062] 4) Model optimization learning;

[0063] The loss function used during model training is as follows:

[0064] logm(y i |y <i )+aEntropy[p(y i |y <i )]+bKL(m|m ref )

[0065] The minimum objective of the optimization problem is to maximize the total reward.

[0066] By adding the standard maximum entropy gradient Entropy[p(y)] i |y <i To relax convergence, it can help capture different behavioral patterns, and also inserts KL divergence penalty KL(m|m) ref ), to prevent model m from deviating from the original m ref Too far;

[0067] 5) Repeat steps 2) through 4) to explore using the updated model, and repeat the exploration quantization and noise learning cycles until the training reaches the maximum number of times;

[0068] 6) Based on the prefix, use the learned model to generate subsequent text, and then concatenate them into the final generated text.

[0069] Based on the prefix, the learned model generates subsequent text, which is then concatenated and used as a new prefix input into the model. This process is repeated until the desired text (x, y1, y2, ..., y) is generated. t ).

[0070] The specific types of reinforcement learning algorithms mentioned above are not specifically limited in this specification; they may include PPO (Proximal Policy Optimization) algorithm, DQN (Deep Q-Network) algorithm, or Actor-Critical, etc.

[0071] To evaluate the effectiveness of the proposed solution in this invention, we mainly conduct experimental verification through experiments on real-world datasets.

[0072] Single attribute control:

[0073] Experimental Setup: Following previous work, we used 10K naturally occurring prefixes from the OpenWebText corpus, categorized into 5K "neutral" prompts, 2.5K "negative" prompts, and 2.5K "positive" prompts. The sentiment polarity of the prompts was determined by the category of the sentence generated by their GPT2 dataset. We used a large GPT2 model as the base PLM and employed a prompt technique instead of tuning the entire model. The sentiment scorer, based on GPT2 and trained on SST-5, used PPL,dist-n to measure generational fluency and diversity, and correctness to measure the proportion of generations satisfying the target sentiment. An open-source sentiment classifier was used to distinguish the generational categories. The comparison results are shown in Table 1.

[0074] Table 1 Comparison of experimental results on the VQA-CPv2 dataset

[0075]

[0076] Experimental comparison with state-of-the-art models: The experimental results of single-attribute emotion control are shown in Table 1, and we obtained the following observations. Although existing methods are close to our results in terms of diversity of expression (DIST), they generally have higher PPL scores, indicating lower fluency. Furthermore, their control ability is lower than that of our proposed model. Our method achieves the best control accuracy among all methods and significantly outperforms the baseline model.

[0077] Multi-attribute control:

[0078] Experimental Setup: We conducted our experiments on a two-attribute control task. We used the widely adopted Yelp benchmark, which included restaurant reviews tagged with sentiment (positive and negative) and topic (US, Mexico, and Asia). To measure whether sentences satisfy the given attributes, we fine-tuned two RoBERTa-based classifiers to evaluate sentiment and topic under their original settings. We demonstrate the fluency and diversity of the generated text using perplexity (PPL) and average diversity dist-n.

[0079] Table 2 Comparison of experimental results on the VQA-CPv2 dataset

[0080]

[0081] Experimental comparison with state-of-the-art models: The experimental results of multi-attribute emotion control are shown in Table 2, and we obtained the following observations. While maintaining fluency and diversity, our method achieved the best control accuracy among all methods.

[0082] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A controllable text generation method based on fine-grained feedback reinforcement learning, characterized in that, Includes the following steps: 1) For the controllable text generation task, initialize a large-scale text generation strategy model. ; The text generation strategy model m includes partially generated sentences. Next tag information Each - The next generated partial sentence of the guided sequence ; the partially generated sentences As a state, the next marker As an action, the vocabulary is an action space; Initialize one as a reference model A copy of the strategy model, an attribute scorer A data pool was also initialized. And a prefix corpus has been prepared for exploration; 2) Exploration; Based on the controllable text generation task, given the prefix of the generated text... Use the current strategy model to generate subsequent text; For each generated token, calculate its reward. and the data Add to data pool In this context, a lifetime is set for each piece of data to indicate the events it can still experience; 3) Segmentation and noise addition; For data pool Within the reward tiers, and obtain Quantiles divide the reward range into intervals; then, noise is injected into each reward while ensuring that each reward remains within its original interval. This process disrupts the reward order, interferes with the classifier's fixed scoring pattern, and maintains the relative order between intervals. 4) Model optimization learning; The loss function used during model training is as follows: The minimum objective of the optimization problem is to maximize the total reward. By adding the standard maximum entropy gradient To relax convergence, help capture different behavioral patterns, and also incorporate KL divergence penalty. To prevent the model Deviating from the original Too far; 5) Repeat steps 2) through 4) to explore using the updated model until the maximum number of training iterations is reached; 6) Based on the prefix, use the learned model to generate subsequent text to obtain the final generated text.

2. The controllable text generation method based on fine-grained feedback reinforcement learning according to claim 1, characterized in that, In step 2), the reward for text generation is determined based on the state and operation. ; The fine-grained reward design is as follows: Where c represents the specific attribute that you want to enhance the generated text; For each - The next part of the sentence generated by the guide.

3. The controllable text generation method based on fine-grained feedback reinforcement learning according to claim 1, characterized in that, In step 2), a lifespan is set for each piece of data, specifically by adding the data to the data pool. Afterwards, the lifespan will be initialized to The timeout period is reduced by 1 each time a data point is included in the training set; when the data's lifetime reaches 0, the data is removed from the data pool. Delete it.

4. The controllable text generation method based on fine-grained feedback reinforcement learning according to claim 1, characterized in that, In step 2), if different parts of the input prefix involve different attributes, the contribution of the token to multiple attributes is weighed separately. Training a weighing machine To balance the rewards of multiple scorekeepers, given a model The last layer is hidden. Weighing output As a weighting factor among multiple rewards; That is, the final reward is in, The reward corresponding to attribute i. for The corresponding weights.

5. An electronic device, characterized in that, include: One or more processors; as well as Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 4.

Images