Method for preventing posterior collapse of variational automatic encoder in text modeling

Abstract

The invention discloses a method for preventing posterior collapse of a variational automatic encoder in text modeling. According to the method, two paths of Path A and Path B are set in a decoder of a model, in the Path A, a text instance is directly input into the decoder, and in the Path B, part of words in the text instance are randomly replaced by unregistered word unified symbols and then input into the decoder. A Path stop strategy is set for the model, in the initial stage of model training, Path A and Path B exist at the same time, and when the model approaches convergence, the Path B is cancelled. And meanwhile, a KL weight zero setting method is used, and a part of dimensions of the KL weight are randomly set to be 0 in the KL annealing process. According to the method, the model is forced to encode more information about input data in the potential variable through the Path B, the model can fully utilize the representation capability of a decoder through the Path A and the Path stop strategy, meanwhile, the model is prevented from entering the local optimal state when the Path B exists, the number of active units in the potential variable is increased through the KL weight zero setting method, and the method is more efficient and efficient. And finally, the problem of posterior collapse of the variational automatic encoder in text modeling is better solved.

Description

Technical Field

[0001] The present invention relates to the technical field of variational autoencoders, and in particular to a method for preventing a posterior collapse problem from occurring in a variational autoencoder in text modeling. Background Art

[0002] Variational autoencoders are a highly-regarded generative model with wide applications in many challenging areas such as natural language processing. A key component of a variational autoencoder is a latent variable with distributional properties, which enables the model to model high-level grammatical and semantic features of text, and to generate new text by sampling from the prior probability distribution of the latent variable and then decoding it. However, the information about the input text stored in the latent variable is closely related to the representation power of the variational autoencoder decoder. When the representation power of the decoder is sufficient to model the data, the model ignores the latent variable and obtains a posterior probability distribution that is almost identical to the prior probability distribution. This problem is often referred to as posterior collapse.

[0003] The posterior collapse problem was first elaborated in the paper "Generating sentences from a continuous space" published by Bowman et al. This paper uses long short-term memory networks as the decoder of variational autoencoders to model text, and proposes two methods to prevent the model from posterior collapse. The first method is KL (Kullback-Leibler divergence) annealing (KL cost annealing), that is, adding a coefficient that gradually increases from 0 to 1 before the KL term of the objective function. The second method is word dropout and historyless decoding, that is, removing some of the input words used as conditions in the decoder. Although some subsequent works have found that KL annealing is not suitable for more complex text datasets and larger decoders, and the optimal annealing mode is different under different datasets and model structures, this method has been widely used and used as the basis for many other methods. However, the word dropout and historyless decoding method has not received enough attention. The main reason is that this method fails to fully utilize the representation ability of the decoder, which will have an adverse effect on the performance of the model in data density estimation and generating new data. Therefore, it can still be further studied and improved. Summary of the invention

[0004] The purpose of the present invention is to provide a method for preventing posterior collapse of variational autoencoders in text modeling, in view of the problem that the existing word-dropping and history-free decoding methods fail to fully utilize the decoder representation capability.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] 1. A method for preventing posterior collapse of a variational autoencoder in text modeling, characterized in that a variational autoencoder model is trained using an objective function, wherein the objective function is:

[0007]

[0008] Where L represents the objective function, path is the path identifier of the decoder in the model, L(1) is the objective function when Path A and Path B exist at the same time, L(2) is the objective function when only Path A exists, p(X) represents the probability of the real text data X, x = [x1, x2, x3, …, x t ,…,x T ] is a text instance from X, x t represents the words in the text instance x, and denote the likelihood distribution of x learned by Path A and Path B in the decoder, α is the hyperparameter for controlling the word drop ratio in Path B, z denotes the latent variable, represents the posterior probability distribution of the latent variable z corresponding to x learned by the encoder in the model, p(z) represents the prior probability distribution of the latent variable z, using the standard Gaussian distribution N(0,I), represents the parameters of the encoder, θ represents the parameters of the decoder, E.[·] represents the expectation, D KL [·||·] represents KL divergence.

[0009] 2. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 1, characterized in that it comprises the following steps:

[0010] S1, obtain text data set and preprocess the data;

[0011] S2, constructing a variational autoencoder model, wherein the variational autoencoder model includes an encoder and a decoder, and the encoder and the decoder adopt an autoregressive neural network;

[0012] S3, divide the text dataset described in S1 into a training set, a validation set, and a test set, input the variational autoencoder model described in S2, and use the objective function L to optimize the model training.

[0013] 3. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 2, characterized in that S3 comprises the following steps:

[0014] S31, initializing the parameters, hyperparameters, and Path identifiers of the variational autoencoder, wherein the parameters include encoder parameters and decoder parameter θ, the hyperparameter is α which controls the word discard ratio in Path B of the decoder, and the Path identifier path is set to True;

[0015] S32, dividing the training set into several small batches of data, using the objective function L to train the optimization model for each small batch of data, and updating the Path identifier path according to the Path stop strategy until convergence.

[0016] 4. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 3, wherein S32 comprises the following steps:

[0017] ① Input the small batch data into the encoder, for each text instance x = [x1, x2, x3, ..., x t ,…,x T ] to obtain the posterior probability distribution The mean μ x and variance

[0018] ②Using the heavy parameter technology, from the posterior probability distribution Sampling in the middle, we get the latent variable z;

[0019] ③If path is True, reconstruct x through the decoder's Path A and Path B, and use the gradient descent algorithm to update the parameters according to the objective function L(1) and θ; if path is False, reconstruct x through the decoder's Path A and use the gradient descent algorithm to update the parameters according to the objective function L(2) and θ.

[0020] 5. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 3 is characterized in that the Path stopping strategy in S32 is to set the Path identifier path to True when the model is initialized, and determine whether the model is close to convergence based on the number of learning rate decays. If so, set the Path identifier path to False.

[0021] 6. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 4, characterized in that Path A of the decoder is to convert the text instance x = [x1, x2, x3, ..., x t ,…,x T ] is input into the decoder to reconstruct and decode x.

[0022] 7. The method for preventing posterior collapse of variational autoencoders in text modeling according to claim 4, characterized in that Path B of the decoder is to randomly convert text instances x=[x1, x2, x3, ..., x t ,…,x T ] replace the α-ratio words with the unified symbol of unregistered words <unk>Get the replaced word sequence Will Input into the decoder and reconstruct and decode x.

[0023] 8. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 4, characterized in that in the step of updating the parameters according to the objective function In the process of θ and θ, a modified KL cost annealing method is adopted. The KL weight is in vector form, and the dimension of the λ ratio in the KL weight is randomly set to 0 during the annealing process.

[0024] The beneficial effects of the present invention are: it can prevent the posterior collapse problem of variational autoencoders in text modeling; compared with word dropping and history-free decoding methods, it can make more full use of the representation ability of the decoder; it has better probability density estimation and representation learning effects; the KL distance between the aggregated posterior probability distribution and the prior probability distribution of latent variables is smaller, and high-quality and diverse data can be generated from a continuous and smooth latent variable space. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of the method for preventing posterior collapse of the variational autoencoder in text modeling according to the present invention.

[0026] Figure 2 This is a model structure diagram for preventing the posterior collapse of the variational autoencoder in text modeling according to the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0028] like Figure 1 As shown, the technical solution for preventing the posterior collapse problem of the variational autoencoder in text modeling according to the embodiment of the present invention comprises the following steps:

[0029] S1, obtain the text dataset and preprocess the data.

[0030] S2, build a variational autoencoder model, such as Figure 1 As shown, the variational autoencoder model includes an encoder and a decoder, and the encoder and the decoder adopt an autoregressive neural network.

[0031] S3, divide the text dataset in S1 into a training set, a validation set, and a test set, input the set into the variational autoencoder model in S2, and use the objective function L to optimize the model training, wherein the objective function is:

[0032]

[0033] Where L represents the objective function, path is the path identifier of the decoder in the model, L(1) is the objective function when Path A and Path B exist at the same time, L(2) is the objective function when only Path A exists, p(X) represents the probability of the real text data X, x = [x1, x2, x3, …, x t ,…,x T ] is a text instance from X, x t represents the words in the text instance x, and denote the likelihood distribution of x learned by Path A and Path B in the decoder, α is the hyperparameter for controlling the word drop ratio in Path B, z denotes the latent variable, represents the posterior probability distribution of the latent variable z corresponding to x learned by the encoder in the model, p(z) represents the prior probability distribution of the latent variable z, using the standard Gaussian distribution N(0,I), represents the parameters of the encoder, θ represents the parameters of the decoder, E.[·] represents the expectation, D KL [·||·] represents KL divergence. In specific applications, the basic KL annealing method is combined to control the KL term weight.

[0034] S3 includes the following steps:

[0035] S31, initializing the parameters, hyperparameters, and Path identifiers of the variational autoencoder, wherein the parameters include encoder parameters and decoder parameters θ, the hyperparameter is α which controls the word discard ratio in Path B of the decoder, and the Path identifier path is set to True. In specific applications, the encoder parameters The decoder parameters θ can be initialized with a uniform distribution, and the word drop ratio α can be determined based on the dataset.

[0036] S32, divide the training set into several small batches of data, use the objective function L to train the optimization model for each small batch of data, and update the Path identifier path according to the Path stop strategy until convergence. The Path stop strategy is to set the Path identifier path to True when the model is initialized, and determine whether the model is close to convergence based on the number of learning rate decays. If so, set the Path identifier path to False.

[0037] S32 includes the following steps:

[0038] ① Input the small batch data into the encoder, for each text instance x = [x1, x2, x3, ..., x t ,…,x T ] to obtain the posterior probability distribution The mean μ x and variance

[0039] ②Using the heavy parameter technology, from the posterior probability distribution Sampling from the , we get the latent variable z.

[0040] ③If path is True, reconstruct x through the decoder's Path A and Path B, and use the gradient descent algorithm to update the parameters according to the objective function L(1) and θ; if path is False, reconstruct x through the decoder's Path A and use the gradient descent algorithm to update the parameters according to the objective function L(2) and θ.

[0041] like Figure 2 As shown, the Path A of the decoder is to convert the text instance x = [x1, x2, x3, ..., x t ,…,x T ] is input into the decoder to reconstruct and decode x; the Path B of the decoder is to randomly convert the text instance x = [x1, x2, x3, ..., x t ,…,x T ] replace the α-ratio words with unregistered words <unk>Get the replaced word sequence Will Input into the decoder to reconstruct and decode x. Path B forces the model to encode more information of the input data in the latent variables, while Path A combined with the Path stopping strategy enables the model to fully utilize the representation capacity of the decoder while avoiding the model from entering the local optimal state when Path B exists.

[0042] In the above method, the parameters are updated according to the objective function. In the process of θ and θ, the modified KL cost annealing method is used. The KL weight is in vector form. During the annealing process, the dimension of the λ ratio in the KL weight is randomly set to 0. The KL weight zeroing method is used to increase the number of active units in the latent variable, thereby further improving the probability density estimation and representation learning effect of the model. In specific applications, the weight zeroing ratio λ can be determined by adjusting the parameters of the data set.

[0043] By adopting the above-mentioned technical scheme disclosed in the present invention, the following beneficial effects are obtained: the method provided by the present invention can prevent the posterior collapse problem of the variational autoencoder in text modeling; compared with the word dropping and history-free decoding methods, the representation ability of the decoder can be more fully utilized; it has better probability density estimation and representation learning effects; compared with the method of forcing the KL term to be greater than a certain positive value, the KL distance between the aggregated posterior probability distribution and the prior probability distribution of the latent variables is smaller, and high-quality and diverse data can be generated from a continuous and smooth latent variable space.

[0044] Those skilled in the art will appreciate that the present application embodiments can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0045] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0046] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0047] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.< / unk> < / unk>

Claims

1. A method for preventing posterior collapse of a variational autoencoder in text modeling, characterized in that: The variational autoencoder model is trained using an objective function, which is: Where L represents the objective function, path is the path identifier of the decoder in the model, L(1) is the objective function when Path A and Path B exist at the same time, L(2) is the objective function when only Path A exists, p(X) represents the probability of the real text data x, x = [x1, x2, x3, ..., x t , ..., x T ] is a text instance from x, x t represents the words in the text instance x, and denote the likelihood distribution of x learned by Path A and Path B in the decoder, α is the hyperparameter for controlling the word drop ratio in Path B, z denotes the latent variable, represents the posterior probability distribution of the latent variable z corresponding to x learned by the encoder in the model, p(z) represents the prior probability distribution of the latent variable z, using the standard Gaussian distribution N(0, I), represents the parameters of the encoder, θ represents the parameters of the decoder, E.[·] represents the expectation, D KL [·||·] represents KL divergence.

2. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 1, characterized in that: The steps include: S1, obtain the text dataset and preprocess the data. S2, constructing a variational autoencoder model, wherein the variational autoencoder model includes an encoder and a decoder, and the encoder and the decoder adopt an autoregressive neural network. S3, divide the text dataset described in S1 into a training set, a validation set, and a test set, input the variational autoencoder model described in S2, and use the objective function L to optimize the model training.

3. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 2, characterized in that: S3 includes the following steps: S31, initializing the parameters, hyperparameters, and Path identifiers of the variational autoencoder, wherein the parameters include encoder parameters and decoder parameter θ, the hyperparameter is α which controls the word discard ratio in Path B of the decoder, and the Path identifier path is set to True. S32, dividing the training set into several small batches of data, using the objective function L to train the optimization model for each small batch of data, and updating the Path identifier path according to the Path stop strategy until convergence.

4. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 3, characterized in that: S32 includes the following steps: ① Input the small batch data into the encoder, for each text instance x = [x1, x2, x3, ..., x t , ..., x T ] to obtain the posterior probability distribution The mean μ x and variance ②Using the heavy parameter technology, from the posterior probability distribution Sampling from the , we get the latent variable z. ③If path is True, reconstruct x through the decoder's PathA and Path B, and use the gradient descent algorithm to update the parameters according to the objective function L(1) and θ; if path is False, reconstruct x through the decoder's PathA and use the gradient descent algorithm to update the parameters according to the objective function L(2) and θ.

5. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 3, characterized in that: The Path stopping strategy in S32 is to set the Path identifier path to True when the model is initialized, and determine whether the model is close to convergence based on the number of learning rate decays. If so, set the Path identifier path to False.

6. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 4, characterized in that: The decoder's PathA is to convert the text instance x=[x1, x2, x3, ..., x t , ..., x T ] is input into the decoder to reconstruct and decode x.

7. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 4, characterized in that: Path B of the decoder is to randomly replace α proportion of the words in the text instance x = [x1, x2, x3,..., x t ,..., x T with the out-of-vocabulary word unified symbol <unk> to obtain the replaced word sequence Input it into the decoder and perform reconstruction decoding on x.

8. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 4, characterized in that: In the above method, the parameters are updated according to the objective function. In the process of θ and θ, a modified KL annealing method (KLcost annealing) is adopted. The KL weight is in vector form, and the dimension of the λ ratio in the KL weight is randomly set to 0 during the annealing process.

Images