A controlled text generation method based on latent variable manipulation of variational autoencoders

Abstract

The present invention relates to a controlled text generation method based on latent variable manipulation of a variational autoencoder, comprising: training a variational autoencoder for text generation; calculating the association between the latent variables of the variational autoencoder and the attributes of the generated text; and generating text with specified attributes by manipulating the latent variables of the variational autoencoder. The controlled text generation method provided by the present invention can control multiple different attributes of the generated text using only a trained variational autoencoder model, and can also use text with different attributes to jointly improve the quality of the generated text.

Description

Technical Field

[0001] The present invention relates to the field of controlled text generation, and in particular to a controlled text generation method based on latent variable manipulation of a variational autoencoder. Background Art

[0002] With the rapid development of today's economy, culture, and technology, text generation technology has become crucial as a tool for improving the efficiency of text content production. It has been widely applied in fields such as automated news writing, intelligent customer service systems, and chatbots. In practical applications, automatically generated text generally requires controllable attributes such as content, style, sentiment, theme, and length. Generating text with specified attributes is known as controlled text generation. Controlled text generation has become a widely researched and discussed issue in the field of text generation.

[0003] The traditional approach to generating controlled text involves using templates and rules. By filling a pre-written text template with specified text attributes according to certain rules, text with the specified attributes can be generated. However, this approach suffers from poor fluency and diversity in the generated text, and requires maintaining a large library of templates and rules, which is very costly. The current mainstream approach to generating controlled text is based on deep learning. Neural networks trained on large-scale corpora can generate natural, fluent text. However, due to the black-box nature of deep learning, the process by which neural networks generate text is difficult to explain, and the properties of the generated text are also difficult to control.

[0004] At present, controlled text generation methods based on deep learning are mainly divided into two directions.

[0005] The first area involves controlled text generation based on pretrained language models. This approach typically involves pretraining a generative language model on a large unsupervised corpus. The pretrained language model can then generate text given certain natural language prompts (prefix inputs). The resulting text can be viewed as a function of the natural language prompts, and the pretrained language model can control the resulting text by modifying the natural language prompts.

[0006] The second direction is controlled text generation based on variational autoencoders. A specific approach to controlled text generation based on variational autoencoders is the conditional variational autoencoder (CVA). This approach builds on the framework of a text variational autoencoder by adding an additional control condition vector representation to the VA's latent variables. By modifying the control condition vector, the properties of the generated text are controlled.

[0007] The present invention mainly focuses on a controlled text generation method based on variational autoencoders. Summary of the Invention

[0008] Existing controlled text generation methods based on variational autoencoders cannot simultaneously control multiple text attributes using a single model, nor can they leverage text with multiple different attributes to jointly improve the quality of generated text. The present invention addresses this problem through a method based on latent variable manipulation in a variational autoencoder. This method allows for simultaneous control of multiple text attributes using a single variational autoencoder model, and allows for the use of text with multiple different attributes to jointly improve the quality of generated text.

[0009] In response to the problems existing in the existing technical solutions, the present invention provides a controlled text generation method based on latent variable manipulation of a variational autoencoder, comprising the following steps:

[0010] S1: Train a variational autoencoder model for text generation;

[0011] S2: Calculate the association between the latent variables of the variational autoencoder and the attributes of the generated text;

[0012] S3: Generate text with specified attributes by manipulating the latent variables of the variational autoencoder.

[0013] To further supplement this technical solution, the specific process of calculating the association between the latent variables of the variational autoencoder and the attributes of the generated text in S2 is as follows:

[0014] S 2.1 :Based on the trained variational autoencoder model for text generation, a set of latent variables and the text corresponding to the latent variables are sampled;

[0015] S 2.2 : Calculate a set of text attributes for the sampled text;

[0016] S 2.3 :For different text attributes, different association calculation methods are used to calculate the association between generated text attributes and latent variables.

[0017] In order to further supplement this technical solution, the text attributes include text category attributes and text value attributes.

[0018] In order to further supplement the present technical solution, a region in the latent space of the variational autoencoder corresponds to a value of a category attribute of the text, and an association between the latent variable and the text category attribute is established by finding the corresponding latent space region for each category attribute value.

[0019] In order to further supplement this technical solution, for each numerical attribute of the text, the latent variable of the variational autoencoder has a projection direction, and correspondingly, the projection of the latent variable in the projection direction determines the value of this attribute of the generated text.

[0020] To further supplement this technical solution, the specific process of generating text with specified attributes by manipulating the latent variables of the variational autoencoder in S3 is as follows:

[0021] S 3.1 : Sample a latent variable that meets the conditions according to the control conditions;

[0022] S 3.2 : The sampled latent variables are input into the decoder of the variational autoencoder to generate text that meets the control conditions.

[0023] In order to further supplement the present technical solution, the process of manipulating the latent variables of the variational autoencoder refers to sampling the latent variables that meet the conditions according to the control conditions.

[0024] In order to further supplement the present technical solution, when controlling the attributes of the generated text, different control methods are adopted for the category attributes of the text and the numerical attributes of the text.

[0025] In order to further supplement this technical solution, when generating text with specified category attributes, latent variables that meet the conditions are sampled from the control condition, that is, the area corresponding to an attribute value, and the sampling method is a rejection sampling algorithm with confidence.

[0026] In order to further supplement the present technical solution, when generating text with specified numerical attributes, a latent variable is first sampled from the prior distribution, and then the sampled latent variable is moved along the direction corresponding to the text attribute to be controlled, so that the projection of the moved latent variable in the direction corresponding to the attribute to be controlled satisfies the control condition, that is, obtains the specified value.

[0027] The beneficial effects of the present invention are:

[0028] From a technical perspective, the technical method proposed in this invention: 1. A trained variational autoencoder model can be used to simultaneously control multiple attributes of the generated text, significantly saving the resources required for model training, model online services, and model storage; 2. Since the attribute labels of the training text are not required when training the variational autoencoder, text training with multiple different attributes can be used to jointly improve the quality of the generated text.

[0029] From the application level, the technical solution of the present invention: 1. It is applicable to various languages, including but not limited to English, Chinese, French, German, etc.; 2. The encoder and decoder can adopt a variety of different structures, such as LSTM and GRU, etc., with good flexibility; 3. It can naturally use unsupervised corpus training models to further improve the quality of generated text; 4. The process of latent variables and controlled text generation has a certain degree of interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart according to one embodiment of the present invention;

[0031] Figure 2 It is a flowchart for calculating the association between the latent variables of the variational autoencoder and the attributes of the generated text;

[0032] Figure 3 It is a flowchart for controlling the properties of generated text by manipulating the latent variables of the variational autoencoder;

[0033] Figure 4 It is a flowchart of the rejection sampling algorithm with confidence level;

[0034] Figure 5 is a graph of a variational autoencoder model for text generation. DETAILED DESCRIPTION

[0035] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0036] like Figure 1 As shown in FIG, a controlled text generation method based on latent variable manipulation of a variational autoencoder comprises the following steps:

[0037] S1: Train a variational autoencoder model for text generation; the model diagram is as follows Figure 5 As shown in Figure 2, three training techniques are used during training: decoder end word dropout, KL divergence term coefficient annealing, and KL divergence term weight reduction. This allows the latent variables in the trained variational autoencoder model to encode a large amount of information about the generated text.

[0038] S2: Calculate the association between the latent variables of the variational autoencoder and the attributes of the generated text; including the following sub-steps: Figure 2 As shown,

[0039] S 2.1 :Based on the trained variational autoencoder model for text generation, a set of latent variables and the text corresponding to the latent variables are sampled;

[0040] S2.2 : Calculate a set of text attributes for the sampled text. Text attributes include text categorical attributes and text numerical attributes. Text categorical attributes refer to attributes whose value range is a finite discrete set, such as sentiment (value set {positive, negative}). Different values ​​of the same categorical attribute have no size relationship. Text numerical attributes refer to attributes whose value range is an integer or real number, such as sentence length. Different values ​​of the same numerical attribute have size relationship.

[0041] S 2.3 :Different association calculation methods are used for text category attributes and text numerical attributes to calculate the association between text attributes and latent variables.

[0042] S3 generates text with specified attributes by manipulating the latent variables of the variational autoencoder. This process requires the association between the latent variables calculated in S2 and the attributes of the generated text. (The output of S2 is one of the inputs of S3.)

[0043] Each region in the latent space of a variational autoencoder corresponds to a value of a categorical attribute in the text. The association between the latent variable and the categorical attribute is established by finding the corresponding region in the latent space for each categorical attribute value. Specifically, this establishes a correlation between the latent variable z and the categorical feature a that generates the text x. The specific method is to train a linear classifier to predict a from z. The trained classifier divides the latent space into several regions, each of which corresponds to a value of a.

[0044] For each text numerical attribute, the latent variable of the variational autoencoder has a projection direction, so that the text numerical attribute is mainly determined by the projection of the latent variable in this direction.

[0045] The specific explanation is: for a numerical attribute a, there is a projection direction u a , so that the attribute a of the text x generated by the latent variable z is determined by the latent variable z in u a Projection z on T u a Decision. a It is to control the projection direction of the numerical attribute a. a Move, the attribute a of the text x generated by z will change smoothly. a The drop shadow on the a property controls the generated text.

[0046] S3: Generate text with specified attributes by manipulating the latent variables of the variational autoencoder, including the following sub-steps: Figure 3 As shown,

[0047] S 3.1 : Sample a latent variable that meets the conditions based on the control conditions; the control conditions consist of two parts: the text attribute to be controlled and the value of the text attribute, such as "emotion = positive" and "length = 8". The meanings of these two control conditions are "generate a text with positive emotions" and "generate a text with 8 words" respectively.

[0048] S 3.2 : The sampled latent variables are input into the decoder of the variational autoencoder to generate text that meets the control conditions.

[0049] The process of manipulating the latent variables of the variational autoencoder refers to sampling latent variables that meet the conditions according to the control conditions.

[0050] When controlling the attributes of generated text, different control methods are used for the categorical attributes and numerical attributes of the text. In S2, different methods are used to establish the association between the categorical attributes and numerical attributes of the text and the latent variables. When using this association to control the attributes of the generated text in S3, different methods are also used.

[0051] When generating text with a specified categorical attribute, latent variables that meet the control condition are sampled from the region corresponding to an attribute value. This sampling method uses a confidence-based rejection sampling algorithm. This rejection sampling algorithm accurately samples latent variables that can generate text with the specified categorical attribute, which helps improve the accuracy of controlled text generation.

[0052] When generating text with a specified numerical attribute, we first sample a latent variable from the prior distribution, and then move the sampled latent variable in the direction corresponding to the text attribute to be controlled, so that the projection of the moved latent variable in the direction corresponding to the attribute to be controlled satisfies the control condition and obtains the specified value. The specific explanation is: when controlling the attribute a of the generated text, we move the randomly sampled latent variable z along the direction u of the control attribute a. a Move so that the hidden variable z' after the move is in u a The projection on is a specified value m a (t). Where t represents the value of the attribute a that you want to generate the text x. a (t) indicates that all texts with attribute a taking value t have corresponding latent variables in u a The average projection on . The shift formula is shown in formula (9) in the manual.

[0053] The following combination Figure 1-5 The present invention is further illustrated by the following examples.

[0054] Example:

[0055] S1: Train a variational autoencoder model for text generation;

[0056] like Figure 5 As shown in Figure 1, the variational autoencoder consists of an encoder E and a decoder G. In the variational autoencoder for text generation, the encoder is a GRU sequence encoder, which accepts text x as input and outputs a vector representation r = E(x) of text x. The variational autoencoder calculates an approximate posterior distribution q of the latent variable z that generates text x based on r. φ (z|x)=N(μ,σ 2 ). Where μ and σ are calculated by the neural network according to formula (1) and (2) based on r.

[0057] μ=W1r+b1# (1)

[0058] σ=exp(W2r+b2)# (2)

[0059] Among them, W1, b1, W2, and b2 are parameters in the neural network, which are learned during the training process.

[0060] As shown in formula (3), the variational autoencoder approximates the posterior distribution q φ A hidden variable z is sampled in (z|x), and the decoder G generates a new text based on z.

[0061] z~q φ (z|x)# (3)

[0062] The decoder G of the text generation variational autoencoder is an autoregressive GRU sequence generator, which takes the latent variable z as input and generates the entire text word by word, as shown in formula (4).

[0063]

[0064] At the same time, the decoder can be used to calculate the probability p of generating text x when using the hidden variable z as input θ (x|z).

[0065] When training a variational autoencoder, it is necessary to optimize the objective function (5), where p(z) is the prior distribution of the latent variable, usually using the standard normal distribution:

[0066]

[0067] This objective function consists of two parts. The first part is to maximize the reconstruction probability That is, maximize the probability of reconstructing the input x using the latent variable z sampled from the approximate posterior distribution. The purpose of maximizing the reconstruction probability is to encode as much information about the text x as possible in the latent variable z. The second part is to minimize the KL divergence KL(q φ (z|x)||p(z)). The goal of minimizing the KL divergence is to make the approximate posterior distribution and the prior distribution as close as possible, so that sampling latent variables from the prior distribution can also generate high-quality text. At the same time, the second part of the objective function is a regularization term that prevents the variance of the approximate posterior distribution from degenerating to zero.

[0068] Training text generation variational autoencoders often encounters the problem of posterior collapse, which is a situation where the decoder is too strong or the regularization effect is too strong, so that the decoder ignores the latent variables. To deal with this problem, the present invention adopts three strategies. (a) Decoder word dropout [3], that is, during training, the target word of the decoder input is randomly replaced with [UNK]; (b) KL divergence term coefficient annealing [3], in which a coefficient α of the KL divergence term of the objective function is gradually increased from 0 to 1 in a linear annealing manner; (c) another coefficient β of the KL divergence term is set to a value between 0 and 1, reducing the weight of the KL divergence term [4].

[0069] The variational autoencoder is optimized using the Adam algorithm [7]. The trained variational autoencoder can generate text randomly by ancestor sampling, as shown in formula (6).

[0070] x=G(z),z~p(z)# (6)

[0071] S2: Calculate the association between the latent variables of the variational autoencoder and the attributes of the generated text;

[0072] like Figure 2 As shown, the present invention uses a trained variational autoencoder to randomly sample n latent variables and their corresponding texts, then uses a pre-trained model to calculate a set of text attributes for each generated text, and finally calculates the association between the latent variables and the attributes of the generated text.

[0073] Let the i-th latent variable obtained by sampling be z i , according to z i The generated text is x i , x i The value of attribute a is is a sample consisting of a latent variable and its corresponding attribute a value, and n such samples constitute a data set Concatenate n latent variables to get the matrix Z=[z1,…,z n ]∈Rn×d , concatenate the a attributes of n texts to get a vector

[0074] Considering two common attributes: categorical attributes and numerical attributes, the present invention adopts different methods to calculate the association between latent variables and attributes for the two attributes.

[0075] For category attributes, such as the sentiment of the text (including two discrete values ​​of positive sentiment and negative sentiment) and the theme of the text (such as games, jewelry, clothing, and other discrete values), the present invention assumes that a region in the latent space of the variational autoencoder corresponds to a value of a category attribute of the text (such as positive sentiment). Inputting the latent variables sampled from a certain region into the decoder of the variational autoencoder will generate a sentence with a specific attribute value with a higher probability. The latent variables corresponding to most sentences with specific attribute values ​​are also distributed in a specific region. The present invention establishes an association between latent variables and text attributes by finding the corresponding latent space region for each attribute value. For each text category attribute a, the present invention uses the latent variable in the dataset D a Train a logistic regression model LR a . Trained model LR a According to the latent variable z, the probability distribution p of the attribute a of the text x it generates is predicted a (·|z)=

[0076] LR a (z). The probability that attribute a of text x has value y is p a (y|z). Let the latent space region corresponding to the value t of attribute a be R a＝t , then

[0077]

[0078] Where ε is a small constant value, such as 0.000001.

[0079] For numerical attributes a, such as the length of the text and the depth of the text component tree, the present invention assumes that the latent variable z of the variational autoencoder has a projection direction u a , so that the numerical attribute of the text is mainly determined by the projection z of the latent variable in this projection direction T u a The present invention calls the projection direction u a is the main direction of attribute a.

[0080] The present invention formalizes the main direction of attribute a as satisfying the projection z T u a With the numerical attribute f a The direction with the largest Pearson correlation coefficient is shown in formula (8).

[0081]

[0082] Main direction u a It can be calculated using the canonical correlation analysis (CCA) method [8].

[0083] S3: Generate text with specified attributes by manipulating the latent variables of the variational autoencoder;

[0084] like Figure 3 As shown, the present invention utilizes the association between latent variables and generated text attributes to generate text with specified attributes by manipulating latent variables. Specifically, given a text attribute control condition a = t, a latent variable z that satisfies the condition is sampled, and the latent variable z is input into the decoder G of the variational autoencoder to generate text x = G(z) that satisfies the control condition. Due to the different associations between latent variables and categorical attributes and numerical attributes, the methods of manipulating latent variables to control categorical attributes and numerical attributes are also different.

[0085] For categorical attributes, since a value t of attribute a corresponds to a region R in the latent space a＝t , so only from the region R a＝t Sampling a hidden variable z~p(z)∧z∈R a＝t , input z into the decoder G of the variational autoencoder to generate a sentence x=G(z) that satisfies the category attribute a and has a value t. a＝t Sampling latent variables z can be done by rejection sampling. a It is possible to predict the probability / confidence p that the attribute a of the sentence generated by the latent variable z has the value t a (t|z), using LR during sampling a The prior provided can significantly improve the accuracy of controlled generation of category attributes. This rejection sampling algorithm with confidence is as follows Figure 4 shown.

[0086] For numerical attributes, since the numerical attribute a of the generated text is determined by the hidden variable z in the main direction u of attribute a a Projection z on T u a The present invention moves a hidden variable z obtained by random sampling so that its projection z T u a Equal to a specified value m a (t). The hidden variable z after the shift * Input into the decoder G of the variational autoencoder to generate a sentence x that meets the conditions. The movement method is shown in formula (9).

[0087] z *=z+(m a (t)-z T u a )u a # (9)

[0088] m a The value of (t) is equal to the hidden variable of attribute a in all sampled texts equal to t in the main direction u a The average value on .

[0089] By shifting the latent variables obtained by sampling, the numerical properties of the generated text can be efficiently and interpretably controlled.

[0090] The above embodiments are only for illustrating the technical concept and features of the present invention. Its purpose is to enable people familiar with this technology to understand the content of the present invention and implement it. It cannot be used to limit the scope of protection of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A controlled text generation method based on latent variable manipulation of a variational autoencoder, characterized by: The following steps are involved: S1: Train a variational autoencoder model for text generation; S2: Calculate the association between the latent variables of the variational autoencoder and the attributes of the generated text; S3: Generate text with specified attributes by manipulating the latent variables of the variational autoencoder; The specific process of calculating the association between the latent variables of the variational autoencoder and the attributes of the generated text in S2 is as follows: S 2.1 :Based on the trained variational autoencoder model for text generation, a set of latent variables and the text corresponding to the latent variables are sampled; S 2.2 : Calculate a set of text attributes for the sampled text; S 2.3 :For different text attributes, different association calculation methods are used to calculate the association between generated text attributes and latent variables; The text attributes include text category attributes and text value attributes.

2. The controlled text generation method based on latent variable manipulation of a variational autoencoder according to claim 1, characterized in that: A region in the latent space of the variational autoencoder corresponds to a value of a category attribute of the text, and an association between the latent variable and the text category attribute is established by finding the corresponding latent space region for each category attribute value.

3. The controlled text generation method based on latent variable manipulation of a variational autoencoder according to claim 1, characterized in that: For each numerical attribute of the text, there is a projection direction corresponding to the latent variable of the variational autoencoder, and the projection of the latent variable in the projection direction determines the value of this attribute of the generated text.

4. The controlled text generation method based on latent variable manipulation of a variational autoencoder according to claim 1, characterized in that: The specific process of generating text with specified attributes by manipulating the latent variables of the variational autoencoder in S3 is as follows: S 3.1 : Sample a latent variable that meets the conditions according to the control conditions; S 3.2 : The sampled latent variables are input into the decoder of the variational autoencoder to generate text that meets the control conditions.

5. The controlled text generation method based on latent variable manipulation of a variational autoencoder according to claim 4, characterized in that: The process of manipulating the latent variables of the variational autoencoder refers to sampling latent variables that meet the conditions according to the control conditions.

6. The controlled text generation method based on latent variable manipulation of a variational autoencoder according to claim 4, characterized in that: When controlling the properties of generated text, different control methods are used for the category properties of the text and the numerical properties of the text.

7. The controlled text generation method based on latent variable manipulation of a variational autoencoder according to claim 6, characterized in that: When generating text with specified category attributes, latent variables that meet the conditions are sampled from the control condition, that is, the area corresponding to an attribute value, and the sampling method is a rejection sampling algorithm with confidence.

8. The controlled text generation method based on latent variable manipulation of a variational autoencoder according to claim 6, characterized in that: When generating text with specified numerical attributes, first sample a latent variable from the prior distribution, and then move the sampled latent variable in the direction corresponding to the text attribute to be controlled, so that the projection of the moved latent variable in the direction corresponding to the attribute to be controlled satisfies the control condition and obtains the specified value.

Images