Multi-attribute controllable text generation method for context-aware multi-granularity prompt fusion

Abstract

The invention provides a multi-attribute controllable text generation method based on context-aware multi-granularity prompt fusion. The method comprises the steps of constructing a single-attribute prompt variant set, constructing an intra-layer multi-granularity prompt fusion module and performing general multi-attribute task prompt. According to the method, diversified soft prompt variants are trained on a single-attribute data set, so that the flexibility of single-attribute control is improved; an in-layer multi-granularity prompt fusion mechanism based on context awareness is designed, and an attribute prompt fusion weight is dynamically generated, so that hierarchical fusion of prompts is realized; by introducing universal multi-attribute prompt and adopting two-stage optimization training, the generation accuracy and text fluency under multi-attribute combination control are effectively improved. According to the method, flexible and efficient text generation control can be realized under different attribute combination conditions, and the method is widely applicable to multi-attribute controllable generation tasks in natural language processing.

Description

Technical Field

[0001] The present application relates to natural language processing technology, and in particular, to the field of controllable text generation in natural language processing. Background Art

[0002] In recent years, the task of multi-attribute controllable text generation has garnered widespread attention in the field of natural language processing. Its goal is to generate text that simultaneously satisfies multiple specific attribute constraints (such as sentiment, theme, style, and keywords) to meet users' diverse generation needs. Traditional multi-attribute control methods primarily include strategies such as conditional generation, inference-based regulation, and prompt learning. However, existing methods commonly suffer from the following problems when simultaneously controlling multiple attributes: First, the attribute interference effect after prompt combination is significant, resulting in a trade-off between control accuracy and text fluency in the generated text; second, the prompt fusion method is simple, making it difficult to fully explore the potential features of each prompt at different levels of semantics; and third, there is a lack of a mechanism for dynamically adapting the user input context to the attribute control prompt, resulting in reduced generation quality under complex attribute combination conditions.

[0003] Current mainstream methods, such as prefix fine-tuning, prompt fine-tuning, and adding adaptation modules, usually insert soft prompts in the input layer or specific layers for conditional control. Although they can achieve attribute control to a certain extent, in the case of multi-attribute combinations, mutual interference between prompts will lead to control failure or generation degradation. In addition, existing methods lack flexible regulation and fusion strategies for prompt granularity, making it difficult for the model to meet the dual requirements of multi-attribute control and text naturalness.

[0004] In response to the above problems, the present invention proposes a context-aware multi-granularity prompt fusion multi-attribute controllable text generation method, which aims to enhance the flexibility and accuracy of multi-attribute control, alleviate the interference effect of multi-attribute prompts, and improve the naturalness and diversity of generated text under multi-attribute control conditions by introducing a single-attribute prompt variant set, an intra-layer multi-granularity prompt fusion module and a general multi-attribute task prompt, thereby effectively meeting the text generation tasks under complex control requirements. Summary of the Invention

[0005] In response to the problems of existing multi-attribute controllable text generation methods, such as significant prompt interference, a single prompt fusion method, and a lack of context dynamic adaptation mechanism, this paper proposes a multi-attribute controllable text generation method with context-aware multi-granularity prompt fusion. The method aims to improve the flexibility and accuracy of multi-attribute control and the naturalness of generated text by introducing a single-attribute prompt variant set, designing a multi-granularity prompt fusion module within the layer, and constructing a general multi-attribute task prompt, so as to meet the high-quality text generation tasks under complex control requirements.

[0006] The present invention provides a context-aware multi-granularity prompt fusion multi-attribute controllable text generation method, which is characterized by comprising the following steps:

[0007] Step 1: Obtain natural language control instructions input by the user;

[0008] Step 2: The natural language control instructions input by the user are mapped into a context vector through the encoder word embedding layer:

[0009] X=[x1;x2;…;x m ]

[0010] E(X)=[e1,e2,…,e m ]

[0011] Among them, X represents the natural language control instruction input by the user, x i represents the natural language control fragment corresponding to the i-th attribute in the instruction, E(·) represents the encoder embedding layer, and e i Represents x i The corresponding word embedding representation vector;

[0012] Step 3: Sequentially concatenate the multi-granularity embedding representations to form the initial input to the encoder Transformers module in the order of user input control instructions, single-attribute soft prompts, and general multi-attribute task soft prompts:

[0013]

[0014] Among them, H (0) represents the state vector of the first Transformer module input after the encoder embedding layer, E(X) represents the embedding representation of the natural language control instructions input by the user, Represents attribute a i The single attribute soft hint embedding representation of participates in the input of the first Transformer module, Embedding representations for general multi-attribute task prompts;

[0015] Step 4: Send the multi-granularity embedding representation to the model Transformers module for further processing. Each layer of Transformer module performs the same processing operation.

[0016] Step 5: Pass each layer of Transformer module input through the self-attention layer to obtain the intermediate hidden state vector

[0017]

[0018] Among them, the input of each Transformer module except the first Transformer module is the hidden state vector output by the previous Transformer module, and the state vector input by the first Transformer module is H (0) , represents the hidden state output by the self-attention layer of the j+1th Transformer module, and SelfAttention(·) represents the self-attention layer;

[0019] Step 6: Transform the intermediate hidden state vector output by the attention layer of each Transformer module Send it to the multi-granularity prompt fusion module for processing, including steps 6.1 and 6.2:

[0020] Step 6.1: For each attribute a i , according to its corresponding prompt fusion weight generator Calculate the corresponding fusion weight

[0021]

[0022] in, Represents the attribute a in the middle layer of the j+1th Transformer module i The corresponding prompt fusion weight to be used for prompt fusion, Represents the natural language instruction part of the j-th Transformer module output hidden state, Represents the attribute a output by the j-th Transformer module i The single attribute hint hidden state part;

[0023] Step 6.2: Fusion weights for hints Using the softmax function for processing, the fused prompt is calculated as:

[0024]

[0025] in, Represents the fused attribute a in the j+1th Transformer module i The prompt fragment, Softmax(·) function converts the logits output by the model into a probability distribution, and LayerNorm(·) represents the layer normalization operation. Represents attribute a in the output of the self-attention layer in the j+1th Transformer module i The hidden state fragment of Represents attribute a i The kth hint variant corresponds to the part of the j+1th layer;

[0026] Step 7: After passing through the intra-layer multi-granularity prompt fusion module, the hidden state is re-joined to obtain the intermediate fused hidden state The concatenation method is the same as the concatenation method of the multi-granularity embedding representation in step 3:

[0027]

[0028] in, represents the output hidden state of the multi-granularity prompt fusion module in the middle layer of the j+1th Transformer module, Respectively represent the user input control instruction corresponding to the hidden state segment, attribute a i The prompt corresponds to the fused hidden state segment and the hidden state part of the general multi-attribute task prompt;

[0029] Step 8: Fusion of the processed intermediate hidden states The output hidden state H of the Transformer module of this layer is obtained by feeding it into the feedforward layer of the Transformer module of this layer for forward calculation:

[0030]

[0031] Among them, H (j+1) represents the feedforward network output hidden state of the j+1th Transformer module, as the output hidden state of the j+1th Transformer module, and FFN(·) represents the feedforward network;

[0032] Step 9: Pass the output hidden state H of the Transformer module to the cross attention layer in the Transformer module of the corresponding layer of the decoder to participate in the decoder text generation.

[0033] The main innovations of the present invention include:

[0034] 1. Single-attribute prompt diversity modeling: In order to improve the diversity and expressiveness of single-attribute prompts, thereby enhancing the modeling robustness and generalization of single-attribute features, this paper constructs a single-attribute prompt variant set for each attribute. Multiple prompt variants are trained through sub-dataset partitioning or different initialization strategies, which enables the prompt representation to form rich and diverse expressions in the feature space.

[0035] 2. Intra-layer multi-granularity prompt fusion: In order to alleviate the interference effect between multi-attribute prompts and improve the accuracy and coherence of text generation under multi-attribute control, the present invention designs an intra-layer multi-granularity prompt fusion module. Within each basic module of the encoder, attribute prompt fusion weights are dynamically generated based on the user input context to achieve fine-grained prompt fusion at different levels.

[0036] 3. General Multi-attribute Task Prompt Modeling

[0037] In order to improve the overall control performance and generation quality of the model under multi-attribute combination tasks, the present invention introduces a universal multi-attribute task prompt and adopts a two-stage optimization mechanism: first, the universal prompt is trained under the condition of freezing the single-attribute prompt, and then the prompt fusion weight generator and the universal prompt representation are jointly optimized, so as to effectively model the multi-attribute combination relationship and improve the multi-attribute control effect.

[0038] Through the above technical solution, the present invention can realize a multi-attribute controllable text generation system with controllable prompt granularity, dynamically adjustable fusion, excellent control effect and natural and smooth text generation in a multi-attribute control scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The structure diagram of the traditional Transformer-based encoder-decoder pre-trained language model is shown.

[0040] Figure 2 The structural diagram of the context-aware multi-granularity prompt fusion model of the present invention is shown.

[0041] Figure 3 The intra-layer multi-granularity hint fusion module of the present invention is demonstrated. DETAILED DESCRIPTION

[0042] This invention improves the flexibility of prompt control and optimizes fusion methods in multi-attribute controllable text generation tasks. It aims to enhance the control accuracy, naturalness, fluency, and diversity of generated text under complex attribute combinations. By designing a context-aware, multi-granularity prompt fusion method, this invention proposes a systematic improvement to existing methods, which suffer from significant prompt interference, a single prompt fusion method, and a lack of dynamic adaptation mechanisms.

[0043] In the field of natural language processing, the Transformer (https: / / paperswithcode.com / method / transformer) model structure has become the mainstream basic framework. In the traditional Transformer-based codec pre-training language model, the encoder-decoder architecture is usually used. Figure 1As shown in the figure, the encoder consists of an embedding layer and several layers of Transformer encoder modules stacked together. The encoder accepts the input sequence and, after multiple layers of processing, gradually extracts the deep semantic representation of the input. The decoder is also composed of an embedding layer and several layers of Transformer decoding modules stacked together. Compared with the Transformer encoder module, the Transformer decoder module additionally introduces an encoder-decoder cross-attention layer (hereinafter referred to as the cross-attention layer) so that when the decoder generates the output of each position, it can combine the partial sequence it has generated and the global context information provided by the encoder. A single Transformer encoder module mainly consists of a self-attention layer and a feedforward layer, while a single Transformer decoder module mainly consists of a self-attention layer, a cross-attention layer, and a feedforward layer.

[0044] This invention aims at the traditional Transformer-based codec pre-trained language model. While maintaining the stability of the overall framework, it specifically enhances the internal structure of the encoder and proposes a new module organization and information processing method to improve the modeling granularity and control accuracy of input features. The decoder part maintains the original standard structure to ensure the compatibility and reasoning efficiency of the generation process. For the sake of unified expression, "Transformer module" is used below to represent the Transformer encoder module, and "pre-trained model" is used to represent the traditional Transformer-based codec pre-trained language base model.

[0045] Specifically, the present invention mainly includes the following technical links: constructing a single-attribute prompt variant set, designing an intra-layer multi-granularity prompt fusion module, and introducing a general multi-attribute task prompt. The single-attribute prompt variant set improves the richness of single-attribute feature modeling through diversified training strategies; the intra-layer multi-granularity prompt fusion module dynamically generates fusion weights according to the user input context to achieve fine-grained fusion of multi-attribute prompts within the basic module; the general multi-attribute task soft prompt improves the overall generation quality of the model under multi-attribute combination control tasks through a two-stage optimization mechanism. The modules cooperate with each other and work together to ensure that in multi-attribute complex control scenarios, the model can flexibly adapt to different control requirements and generate text that meets multiple attribute constraints and is natural and fluent.

[0046] like Figure 2As shown, the context-aware multi-granularity prompt fusion model of the present invention is mainly composed of three parts: embedding preprocessing, encoder and decoder: In the embedding preprocessing part, the control instruction input by the user obtains the embedded representation of the control instruction through the embedding layer, and then is spliced ​​with the single-attribute soft prompt and the general multi-attribute task soft prompt to form a complete multi-granularity embedding representation as the input hidden state of the encoder; the encoder part is composed of a plurality of stacked Transformer modules, and each Transformer module uses its self-attention layer, the multi-granularity prompt fusion module within the layer and the feedforward layer to process the hidden state in turn, which becomes the output of the Transformer module; the decoder part receives the hidden state output by the encoder in the cross-attention layer of the corresponding Transformer decoder module to participate in the calculation of text generation. Further, as Figure 3 As shown, the intra-layer multi-granularity prompt fusion module of the present invention is mainly composed of three parts: taking the fragment of the position corresponding to the user input control instruction and the fragment of the position corresponding to the specific attribute soft prompt in the input hidden state as input, the multi-granularity prompt fusion weight of the attribute is calculated through the multi-granularity prompt fusion weight generator of the present invention; then taking the hidden state fragment of the position corresponding to the attribute soft prompt and all single attribute soft prompt variants of the attribute as input, the hidden state of the position corresponding to the fused attribute soft prompt is calculated through the multi-granularity prompt fusion weight of the attribute; finally, each hidden state fragment is spliced ​​as the output hidden state of the module.

[0047] The present invention improves upon the traditional Transformer-based codec pre-trained language model by:

[0048] 1. Construct a single-attribute soft prompt variant set

[0049] "Attributes" refer to external conditions or labels used to control the characteristics of generated text, such as sentiment polarity (positive / negative) and topic type (sports / politics). "Single-attribute" refers to controlling only a specific attribute for each generation task, rather than processing multiple attributes simultaneously. For example, controlling only the sentiment of generated text to be positive or only the topic of generated text to be technology. Attributes typically correspond to specific generation control objectives. "Soft prompts" refer to a set of trainable, continuous vector embeddings used to guide the model's input to generate text that meets specific objectives without adjusting the parameters of a pre-trained language model (PLM). Unlike traditional discrete token prompts, soft prompts are typically learned and directly incorporated into the model input representation in the form of vectors. "Variants" refer to different instances of the same object generated under different training conditions, initialization methods, or sub-datasets. While maintaining overall functionality, variants may have subtle parameter or behavioral differences to increase system diversity or robustness. In this paper, a "Single-Attribute Soft Prompt Variant" is defined as follows: A Single-Attribute Soft Prompt Variant refers to multiple soft prompt instances trained independently for a specific attribute using different initialization methods, sub-datasets, or other parameter perturbations. Each Single-Attribute Soft Prompt Variant can independently guide the pre-trained language model to achieve differentiation in style, fluency, or fine-grained features while maintaining consistency in the target attribute.

[0050] Let the attribute set be M is the number of attribute controls, for each attribute Assume that it corresponds to a single attribute data set The present invention first trains K single-attribute soft prompt variants on a single-attribute dataset:

[0051]

[0052] in, Represents the set of prompt variants corresponding to attribute a, represents the hidden state of the kth a-attribute soft hint variant between the self-attention layer and the feed-forward layer in the i-th Transformer module of the encoder ( represents the embedded representation of the soft prompt, participating in the input of the first Transformer module), l represents the length of the single attribute soft prompt, d represents the hidden state dimension in the Transformer module, and N represents the number of Transformer modules in the encoder.

[0053] In order to obtain diverse prompt variants, this paper proposes two strategies to train the prompt set corresponding to each attribute:

[0054] (1) Sub-dataset strategy: Sub-attribute dataset The dataset is divided into several subsets, and each hint variant is trained independently on different subsets to encourage different single-attribute hints to learn different feature patterns in single-attribute data.

[0055] During training, freeze the pre-trained model parameters θ to minimize the language modeling loss

[0056]

[0057] Among them, x represents the input feature of a data in the training set, and y represents the corresponding target label (target output). Represents the training data set of the kth single attribute variant. The construction of the training data set adopts the principle of random shuffling and even distribution. First, the single attribute complete data set Randomly shuffle and then divide the complete dataset equally according to the number of single attribute variants K Finally got Subdataset Indicates The mathematical expectation of training on P(y|x;θ,P a,k ) represents the kth attribute hint variant P given input x, pre-trained model parameters θ and a,k Under the condition of , the probability of the language model output y is processed by the softmax function commonly used in the field of natural language processing after processing the output of the language model logits as a simulation of the probability distribution.

[0058] (2) Different initialization strategies: Each hint variant is independent on the same single-attribute dataset In the optimization, different randomly initialized prompt vectors are used for independent training to guide the prompts to explore different parameter spaces during the optimization process, thereby forming expressions with different styles.

[0059] During training, freeze the pre-trained model parameters θ to minimize the language modeling loss

[0060]

[0061] Similar to the sub-dataset strategy, x represents the input feature of a data in the training set, and y represents the corresponding target label (target output); Indicates The mathematical expectation of training on P(y|x;θ,P a,k) represents the kth attribute hint variant P given input x, pre-trained model parameters θ and a,k Under the condition of , the probability of the language model outputting y is simulated by processing the output of the language model logits using the softmax function commonly used in the field of natural language processing. a,k The initialization methods include: Gaussian random initialization, uniform distribution initialization, standard embedding vector perturbation, orthogonal initialization, and pre-initialization by attribute subcategory. The following describes these five initialization methods:

[0062] (1) Gaussian random initialization: The embedding vector of each single-attribute soft hint variant is drawn from a standard normal distribution. Or initialize with a normal distribution with smaller variance; each variant is sampled independently to ensure that the initialization parameters are different.

[0063] (2) Uniform distribution initialization: Initialization is performed by uniform random sampling within a uniform interval [-1, 1]. Each variant is sampled independently, which can control the occurrence of extreme values ​​and keep the initial value distribution stable.

[0064] (3) Perturbation of standard embedding vector: Based on the embedding of common words related to attribute a in the pre-trained vocabulary, a small random noise perturbation is added; the soft prompt vector initialized in this way has a certain semantic similarity with the real language, while introducing random diversity.

[0065] (4) Orthogonal initialization: When generating the initial vector, ensure that the initial vector sets between each single-attribute soft prompt variant are approximately orthogonal; ensure that the initial expression directions of different variants are dispersed, which helps to learn different style features.

[0066] (5) Initialization by attribute subcategory: Based on the word embedding of attribute subcategory a (e.g., positive and negative emotions), different subcategory-related word embeddings are used as the initial centers of different variants; this can guide the diversification of prompts in different subfields.

[0067] 2. Build a multi-granularity prompt fusion module within the layer

[0068] First, embed the natural language sequence input by the user. The control instruction of the attribute requirement is composed of multiple natural language segments X = [x1; x2; ...; x m ], each fragment corresponds to a property requirement control, that is, x i Corresponding attribute a i The natural language control fragment of . After the input passes through the word embedding layer, we get:

[0069]

[0070] Among them, e i Represents a natural language control fragment x i The corresponding embedding representation, that is, attribute a i Natural language control embedding.

[0071] The vector after the model embedding layer is composed of the user input, the single-attribute soft prompt variant, and the general multi-attribute task soft prompt sequentially:

[0072]

[0073] Among them, H (0) represents the state vector of the first Transformer module input after the encoder embedding layer; E(X) represents the embedded representation of the control instruction input by the user through the encoder embedding layer; As mentioned above, the attribute a i The single-attribute soft hint embedding representation of is used as the input of the first Transformer module; Embedding representations for soft prompts for general multi-attribute tasks.

[0074] In order to better fuse multi-granularity information, the present invention proposes a multi-granularity hint fusion module within the encoder layer. For the j+1th (j=0,1,……,N)th Transformer module of the encoder, its input is represented by H (j) , which is the output of the j-th Transformer module:

[0075]

[0076] in, represents the hidden state output by the self-attention layer of the j+1th Transformer module, and SelfAttention(·) represents the self-attention layer; H represents the hidden state after the multi-granularity cue fusion module in the j+1th Transformer module. Fusion(·) represents the multi-granularity cue fusion module in the layer. The specific processing details are shown below. (j) represents the output hidden state of the feedforward network of the j-th Transformer module, i.e., the output hidden state of the j-th Transformer module, and FFN(·) represents the feedforward network; Represents the hidden state H (j) The natural language instruction part in Represents attribute a i The hint hidden state part, The hidden state part representing the soft hint for general multi-attribute tasks.

[0077] For each attribute a i , define the multi-granularity prompt fusion weight generator and its corresponding fusion weight

[0078]

[0079] in, Represents the attribute a in the middle layer of the j+1th Transformer module i The multi-granularity cue fusion weights used in the corresponding cue fusion; The natural language instruction part representing the output hidden state of the j-th Transformer module; Represents attribute a i The single attribute hint hidden state part output by the jth Transformer module; K is attribute a i The number of prompt variants. To ensure that the weight sum is 1, the prompt fusion weight is processed using the softmax function. The fused prompt is:

[0080]

[0081] in, Represents the fused attribute a in the j+1th Transformer module i The prompt fragment, Softmax(·) function (https: / / proceedings.neurips.cc / paper_files / paper / 1989 / file / 0336dcbab05b9d5ad24f4333c7658a0e-Paper.pdf) converts the logits output by the model into a probability distribution, and LayerNorm(·) represents the layer normalization operation. Represents attribute a in the output of the self-attention layer in the j+1th Transformer module i The hidden state fragment of Represents attribute a i The kth hint variant corresponds to the part of the j+1th layer.

[0082] After the intra-layer multi-granularity cue fusion module, the model reassembles the hidden states in the same way as the embedding representation:

[0083]

[0084] in, represents the output hidden state of the prompt fusion module in the j+1th Transformer module, Respectively represent the user input corresponding to the hidden state segment, attribute ai Hints correspond to hidden state segments and hidden state parts of general multi-attribute task soft hints.

[0085] During training, all single-attribute soft hint variants are first frozen, and only the multi-granularity hint fusion weight generator f is optimized. a Parameters, minimize attribute control loss:

[0086]

[0087] in, represents the control loss of attribute a, Indicates The mathematical expectation of training on a single attribute dataset, P(y| Represents a set of attribute hint variants given input x, pre-trained model parameters θ and a Under the condition of , the probability of the language model output y is processed by the softmax function commonly used in the field of natural language processing after processing the output of the language model logits as a simulation of the probability distribution.

[0088] Finally, the multi-granularity hint fusion weight generator will participate in the second stage training of general multi-attribute task soft hints (see Section 3 below) for further optimization on all attribute tasks.

[0089] 3. Build a general multi-attribute task soft prompt

[0090] In order to improve the ability to model the combination relationship between attributes, this paper introduces a general multi-attribute task soft prompt P multi , used to jointly represent multi-attribute task information. First, a set of continuous vectors is randomly initialized as P multi The present invention then designs a two-stage optimization process for the general multi-attribute task soft prompt:

[0091] Phase 1: After the individual attribute soft hint set is built, use the frozen single attribute soft hint P multi Splice to the end of the input prompt:

[0092]

[0093] Labeling data with multiple attributes Optimize P multi The representation ability of P multi To train:

[0094]

[0095] in, represents the first-stage general multi-attribute task soft prompt loss function; Indicates The mathematical expectation of training on a multi-attribute labeled training set; P(y|x; θ, P) represents the probability of the language model outputting y given the input x, pre-trained model parameters θ, and input prompt P. The softmax function commonly used in the field of natural language processing is also used to process the output of the language model logits as a simulation of this probability distribution.

[0096] The second stage: further training the multi-granularity prompt fusion weight generator f a In the process of the first training, the P multi As one of the inputs, it participates in the training of the multi-granularity prompt fusion weight generator, and continues to backpropagate and update it to complete the joint secondary optimization:

[0097]

[0098] in, represents the multi-attribute joint training loss, Indicates The mathematical expectation of training on the multi-attribute labeled training set, P(y|x;θ,f a ,P multi ) represents the probability of the language model outputting y given the input x, pre-trained model parameters θ and input prompt P. During training, θ and Freeze does not participate in the update, the model only updates the multi-granularity prompt fusion weight generator f a and general multi-attribute task soft prompt P multi .

[0099] Finally, in the inference phase, all module parameters of the model are frozen, and according to the process introduced in Section 2, dynamic fusion prompts are generated and spliced ​​to achieve multi-attribute control generation.

[0100] To systematically validate the effectiveness of the proposed context-aware multi-granularity prompt fusion (CMPF) method in multi-attribute controllable text generation tasks, experiments were conducted under single-attribute, two-party, three-party, and four-party control scenarios. The evaluation tasks covered four control attribute categories: sentiment, topic, tense, and lexical, comprehensively examining the performance of the proposed method under complex control conditions. The T5 model (https: / / paperswithcode.com / paper / exploring-the-limits-of-transfer-learning) was used as the base model for the encoder-decoder, on which the soft prompt and prompt fusion modules were trained and tested. The dataset used was the widely recognized YELP dataset (https: / / huggingface.co / datasets / Yelp / yelp_review_full). The baseline models used are the GEDI model (https: / / paperswithcode.com / paper / gedi-generative-discriminator-guided-sequence), the DIST.LENS model (https: / / paperswithcode.com / paper / a-distributional-lens-for-multi-aspect), the PROMPT-TUNING model (https: / / paperswithcode.com / paper / the-power-of-scale-for-parameter-efficient), the PREFIX-TUNING model (https: / / paperswithcode.com / paper / prefix-tuning-optimizing-continuous-prompts), the Tailor model (https: / / aclanthology.org / 2023.acl-long.25 / ), and the Prompt Gating model (https: / / paperswithcode.com / paper / prompt-gating-a-parameter-efficient-tuning).

[0101] The present invention first evaluated the performance of the model on the dual-attribute control task, and the experimental results are shown in Table 1. The experimental results show that the control accuracy of the present invention on the dual-attribute control task is stronger than or close to the mainstream controllable text generation model. In terms of emotion control attributes, the accuracy of the CMPF method of the present invention reached 86.30%, and in terms of topic control attributes, it reached 76.80%, both of which exceeded the PROMPT GATING method with outstanding multi-attribute controllability, and the comprehensive average accuracy was improved to 81.55%, which was 1.64 percentage points higher than PROMPT GATING, verifying the effectiveness of the context-aware multi-granularity prompt fusion mechanism proposed by the present invention in the dual-attribute control task. Although the GEDI model achieved the highest scores in emotion control accuracy and text diversity (Distinctness) indicators, the perplexity (Perplexity) of its generated text was extremely high, and the tense control accuracy was significantly reduced. Under the condition of ensuring a certain text diversity, the method of the present invention reduces the perplexity of the generated text and improves the fluency and consistency of the text to a certain extent.

[0102] Table 1 Experimental results of dual-attribute control task

[0103]

[0104] The present invention then evaluated the performance of the model on three-attribute and four-attribute control tasks, and the experimental results are shown in Tables 2 and 3. The CMPF method of the present invention achieved 78.45%, 64.33%, and 63.70% in emotion control, topic control, and tense control, respectively, with an overall average control accuracy of 68.83%, an improvement of 1.53 percentage points over the PROMPT GATING method. In particular, for the more difficult attribute of tense control, the CMPF method still maintained a high level of accuracy, demonstrating the advantages of the present method in its ability to coordinate multi-attribute modeling. While achieving excellent three-attribute control performance, the present method still ensured a low perplexity of generated text, demonstrating the ability to maintain natural and fluent text generation under complex multi-attribute conditions. Compared with the DIST.LENS and PREFIX-TUNING methods, the present method achieved a significant lead in multi-attribute accuracy while maintaining the best fluency, indicating that the present method can effectively suppress text degradation while taking into account the requirements of multi-attribute control. In the most complex four-way control task, the CMPF method of the present invention maintains a clear advantage: the CMPF method achieves 77.14%, 64.22%, 61.08%, and 55.12% for the four control attributes of emotion, theme, tense, and keyword, respectively, with an overall average control accuracy of 64.39%, an improvement of nearly 1.3 percentage points over the PROMPT GATING method. This demonstrates the strong modeling and generation capabilities of the present method for more complex attribute combinations. In terms of keyword (Lexical) control accuracy, the CMPF method achieves 55.12%, a significant improvement over the PROMPT GATING method, demonstrating its ability to achieve refined content control. Furthermore, in terms of scalability, DIST.LENS, due to its joint training strategy, requires complete retraining when expanding to more attributes. In contrast, PREFIX-TUNING, PROMPT GATING, and the CMPF method of the present invention only require training of newly added attribute plug-ins and combining them for use, demonstrating excellent modular expansion capabilities. Especially in the process of expanding from three attributes to four attributes, the PREFIX-TUNING method suffers from significant performance degradation due to severe interference between plug-ins. Although the PROMPT GATING method has made some improvements in reducing attribute interference, it does not make good use of multi-granularity contextual information, and thus shows a trend of performance degradation in more complex multi-attribute control tasks. The method of the present invention effectively suppresses interference through a context-aware fusion mechanism, maintains minimal performance degradation, and has low training overhead, demonstrating excellent scalability and application potential.

[0105] Table 2 Experimental results of three-attribute control tasks

[0106]

[0107] Table 3 Experimental results of four-attribute control tasks

[0108]

[0109] Comprehensive experimental results of dual-attribute control, three-attribute control and four-attribute control tasks show that the context-aware multi-granularity prompt fusion method (CMPF) proposed in the present invention is systematically superior to the existing mainstream controllable text generation methods in terms of attribute control accuracy, text naturalness and text diversity, and its advantages in scalable multi-attribute combination control tasks are more significant, which fully verifies the effectiveness and applicability of the method of the present invention.

Claims

1. A context-aware multi-granularity prompt fusion multi-attribute controllable text generation method, characterized by: The following steps are involved: Step 1: Obtain natural language control instructions input by the user; Step 2: The natural language control instructions input by the user are mapped into a context vector through the encoder word embedding layer: X=[x1;x2;…;x m ] E(X)=[e1,e2,…,e m ] Among them, X represents the natural language control instruction input by the user, x i represents the natural language control fragment corresponding to the i-th attribute in the instruction, E(·) represents the encoder embedding layer, and e i Represents x i The corresponding word embedding representation vector; Step 3: Sequentially concatenate the multi-granularity embedding representations to form the initial input to the encoder Transformers module in the order of user input control instructions, single-attribute soft prompts, and general multi-attribute task soft prompts: Among them, H (0) represents the state vector of the first Transformer module input after the encoder embedding layer, E(X) represents the embedding representation of the natural language control instructions input by the user, Represents attribute a i The single attribute soft hint embedding representation of participates in the input of the first Transformer module, Embedding representations for general multi-attribute task prompts; Step 4: Send the multi-granularity embedding representation to the model Transformers module for further processing. Each layer of Transformer module performs the same processing operation. Step 5: Pass each layer of Transformer module input through the self-attention layer to obtain the intermediate hidden state vector Among them, the input of each layer of Transformer module outside the first Transformer module is the previous layer The hidden state vector output by the Transformer module, the state vector input to the first Transformer module is H (0) , represents the hidden state output by the self-attention layer of the j+1th Transformer module, and SelfAttention(·) represents the self-attention layer; Step 6: Transform the intermediate hidden state vector output by the attention layer of each Transformer module Send it to the multi-granularity prompt fusion module for processing, including steps 6.1 and 6.2: Step 6.1: For each attribute a i , according to its corresponding prompt fusion weight generator Calculate the corresponding fusion weight in, Represents the attribute a in the middle layer of the j+1th Transformer module i The corresponding prompt fusion weight to be used for prompt fusion, Represents the natural language instruction part of the j-th Transformer module output hidden state, Represents the attribute a output by the j-th Transformer module i The single attribute hint hidden state part; Step 6.2: Fusion weights for hints Using the softmax function for processing, the fused prompt is calculated as: in, Represents the fused attribute a in the j+1th Transformer module i The prompt fragment, Softmax(·) function converts the logits output by the model into a probability distribution, and LayerNorm(·) represents the layer normalization operation. Represents attribute a in the output of the self-attention layer in the j+1th Transformer module i The hidden state fragment of Represents attribute a i The kth hint variant corresponds to the part of the j+1th layer; Step 7: After passing through the intra-layer multi-granularity prompt fusion module, the hidden state is re-joined to obtain the intermediate fused hidden state The concatenation method is the same as the concatenation method of the multi-granularity embedding representation in step 3: in, represents the output hidden state of the multi-granularity prompt fusion module in the middle layer of the j+1th Transformer module, Respectively represent the user input control instruction corresponding to the hidden state segment, attribute a i The prompt corresponds to the fused hidden state segment and the hidden state part of the general multi-attribute task prompt; Step 8: Fusion of the processed intermediate hidden states The output hidden state H of the Transformer module of this layer is obtained by feeding it into the feedforward layer of the Transformer module of this layer for forward calculation: Among them, H (j+1) represents the feedforward network output hidden state of the j+1th Transformer module, as the output hidden state of the j+1th Transformer module, and FFN(·) represents the feedforward network; Step 9: Pass the output hidden state H of the Transformer module to the cross attention layer in the Transformer module of the corresponding layer of the decoder to participate in the decoder text generation.

2. The method according to claim 1, wherein the single-attribute soft prompt variant set is constructed using a method based on a sub-dataset construction strategy or a method based on different initialization strategies; The method for constructing a strategy based on a sub-dataset includes the following steps: Step A1: Convert single attribute dataset Randomly shuffle and evenly divide into K sub-datasets with the same number of variants, and the number of data contained in each sub-dataset is Step A2: Each prompt variant is trained independently on a different subset to encourage different single-attribute prompts to learn different feature patterns in single-attribute data; during training, freeze the pre-trained model parameters θ and minimize the language modeling loss Among them, x represents the input feature of a data in the training set, and y represents the corresponding target label (target output). represents the training dataset of the k-th single-attribute variant, Finally got Subdataset Indicates The mathematical expectation of training; P(y|x; θ,p a,k ) represents the kth attribute hint variant P given input x, pre-trained model parameters θ and a,k Under the condition of , the probability of the language model outputting y is simulated by processing the output of the language model logits with the softmax function commonly used in the field of natural language processing; The methods based on different initialization strategies include the following steps: Step B1: Initialize different single-attribute soft hint variants using different random initialization methods, including Gaussian random initialization, uniform distribution initialization, standard embedding vector perturbation, orthogonal initialization, and pre-initialization by attribute subcategory; Step B2: During training, freeze the pre-trained model parameters θ and minimize the language modeling loss Where x represents the input feature of a piece of data in the training set, and y represents the corresponding target label (target output); Indicates The mathematical expectation of training on P(y|x;θ,P a,k ) represents the kth attribute hint variant P given input x, pre-trained model parameters θ and a,k Under the condition of , the probability of the language model output y is processed by the softmax function commonly used in the field of natural language processing after processing the output of the language model logits as a simulation of the probability distribution.

3. The method according to claim 1, wherein the context-aware multi-granularity cue fusion model training process comprises the following steps: Step 1: After the individual attribute soft hint set is built, use the frozen single attribute soft hint The general multi-attribute task soft prompt P multi Splice to the end of the input prompt: Step 2: Label the data with multiple attributes Individual training P multi : in, represents the first-stage general multi-attribute task soft prompt loss function; Indicates The mathematical expectation of training on a multi-attribute labeled training set; P(y|x; θ, P) represents the probability of the language model outputting y given the input x, pre-trained model parameters θ, and input prompt P. The softmax function commonly used in the field of natural language processing is also used to process the output of the language model logits as a simulation of this probability distribution. Step 3: Freeze all single-attribute soft hint variants and only train the multi-granularity hint fusion weight generator f a Parameters, minimize attribute control loss: in, represents the control loss of attribute a, Indicates The mathematical expectation of training on a single-attribute dataset, Represents a set of attribute hint variants given input x, pre-trained model parameters θ and a Under the condition of , the probability of the language model output y is processed by the softmax function commonly used in the field of natural language processing after processing the output of the language model logits as a simulation of the probability distribution. Step 4: Further train the joint training and train the multi-granularity prompt fusion weight generator f a and general multi-attribute task soft prompt P multi , the P obtained from the first training multi As one of the model inputs, it continues to backpropagate and update. During training, θ and Freeze does not participate in the update, the model only updates the multi-granularity prompt fusion weight generator f a and general multi-attribute task soft prompt P multi , complete the joint secondary training: in, represents the multi-attribute joint training loss, Indicates The mathematical expectation of training on the multi-attribute labeled training set, P(y|x;θ,f a ,P multi ) represents the probability of the language model output y given the input x, pre-trained model parameters θ and input prompt P.

4. An information processing device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

Images