A method to prevent posterior collapse in variational autoencoders for text modeling
By optimizing the objective function and decoder path of the variational autoencoder, the posterior collapse problem of the variational autoencoder in text modeling is solved, and the model's representation ability and generation effect are improved.
Patent Information
- Application Number
- CN202411458817.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-08-29
- Estimated Expiration
- 2044-10-18
AI Technical Summary
Existing variational autoencoders have posterior collapse problems in text modeling, especially word discarding and historyless decoding methods fail to fully utilize the decoder's representation ability, affecting the model's performance in data density estimation and new data generation.
The objective function is used to train the variational autoencoder, combined with the decoder paths of Path A and Path B, and optimize the model parameters and hyperparameters through word discarding and modified KL annealing methods, and update the decoder's representation ability using re-parameter technology and gradient descent algorithm to prevent posterior collapse.
Effectively prevent posterior collapse, improve the decoder's representation ability, improve probability density estimation and generation quality, and better data diversity.
Smart Images
Figure CN119416744B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of variational autoencoders, and in particular to a method for preventing a posterior collapse problem from occurring in text modeling of a variational autoencoder. Background Art
[0002] Variational autoencoders are a highly sought-after generative model with widespread applications in many challenging areas, such as natural language processing. A key component of a variational autoencoder is a latent variable with distributional properties. This latent variable allows the model to model high-level grammatical and semantic features of text, and new text can be generated by sampling from the prior probability distribution of the latent variable and then decoding it. However, the information about the input text stored in the latent variable is closely related to the representation power of the variational autoencoder decoder. When the decoder's representation power is sufficient to model the data, the model ignores the latent variable, resulting in a posterior probability distribution that is almost identical to the prior probability distribution. This problem is often referred to as posterior collapse.
[0003] The problem of posterior collapse was first detailed in the paper "Generating Sentences from a Continuous Space" by Bowman et al. This paper used a long short-term memory network (LSTM) as the decoder of a variational autoencoder to model text and proposed two methods to prevent posterior collapse. The first method is KL (Kullback-Leibler divergence) cost annealing, which adds a coefficient that gradually increases from 0 to 1 before the KL term of the objective function. The second method is word dropout and historyless decoding, which removes some input words used as conditioning in the decoder. Although some subsequent work has found that KL annealing is not suitable for more complex text datasets and large decoders, and that the optimal annealing pattern varies across datasets and model architectures, this method has remained widely used and serves as the foundation for many other methods. However, word dropout and historyless decoding methods have received less attention, primarily because they fail to fully utilize the decoder's representational power, negatively impacting the model's performance in areas such as data density estimation and generating new data. Therefore, further research and improvement are needed. Summary of the Invention
[0004] The purpose of the present invention is to provide a method for preventing posterior collapse of variational autoencoders in text modeling, in order to address the problem that existing word-dropping and history-free decoding methods fail to fully utilize the decoder representation capability.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0006] 1. A method for preventing posterior collapse of a variational autoencoder in text modeling, characterized in that a variational autoencoder model is trained using an objective function, wherein the objective function is:
[0007]
[0008] Where L represents the objective function, path is the path identifier of the decoder in the model, L(1) is the objective function when Path A and Path B exist at the same time, L(2) is the objective function when only Path A exists, p(X) represents the probability of the real text data X, x = [x1, x2, x3, ..., x t ,…,x T ] is a text instance from X, x t represents the word in the text instance x, and Represent the likelihood distribution of x learned by Path A and Path B in the decoder, α is the hyperparameter of the control word drop ratio in Path B, z represents the latent variable, represents the posterior probability distribution of the latent variable z corresponding to x learned by the encoder in the model, p(z) represents the prior probability distribution of the latent variable z, using the standard Gaussian distribution N(0,I), represents the parameters of the encoder, θ represents the parameters of the decoder, E.[·] represents the expectation, D KL [·||·] represents the KL divergence.
[0009] 2. The method for preventing posterior collapse in a variational autoencoder in text modeling according to claim 1, comprising the steps of:
[0010] S1, obtain text dataset and preprocess the data;
[0011] S2, constructing a variational autoencoder model, wherein the variational autoencoder model includes an encoder and a decoder, and the encoder and the decoder adopt an autoregressive neural network;
[0012] S3, divide the text dataset described in S1 into a training set, a validation set, and a test set, input the variational autoencoder model described in S2, and use the objective function L to train and optimize the model.
[0013] 3. The method for preventing posterior collapse in text modeling using a variational autoencoder according to claim 2, wherein S3 comprises the following steps:
[0014] S31, initializing the parameters, hyperparameters, and Path identifiers of the variational autoencoder, including the encoder parameters and decoder parameters θ, the hyperparameter is α that controls the word discard ratio in Path B of the decoder, and the Path identifier path is set to True;
[0015] S32, dividing the training set into several small batches of data, using the objective function L to train the optimization model for each small batch of data, and updating the Path identifier path according to the Path stop strategy until convergence.
[0016] 4. The method for preventing posterior collapse in text modeling using a variational autoencoder according to claim 3, wherein S32 comprises the following steps:
[0017] ① Input the small batch data into the encoder, for each text instance x=[x1,x2,x3,…,x t ,…,x T ]Get the posterior probability distribution The mean μ x and variance
[0018] ②Using the heavy parameter technology, from the posterior probability distribution Sampling in the middle, we get the latent variable z;
[0019] ③ If path is True, reconstruct x through Path A and Path B of the decoder, and use the gradient descent algorithm to update the parameters according to the objective function L(1) and θ; if path is False, reconstruct x through the decoder's Path A and use the gradient descent algorithm to update the parameters according to the objective function L(2) and θ.
[0020] 5. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 3, wherein the Path stopping strategy in S32 is to set the Path identifier path to True when the model is initialized, and determine whether the model is close to convergence based on the number of learning rate decays. If so, set the Path identifier path to False.
[0021] 6. The method for preventing posterior collapse in text modeling of a variational autoencoder according to claim 4, wherein the decoder path A is to convert the text instance x = [x1, x2, x3, ..., x t ,…,x T ] is input into the decoder to reconstruct and decode x.
[0022] 7. The method for preventing posterior collapse in text modeling of a variational autoencoder according to claim 4, wherein the decoder's Path B is to randomly replace the text instance x=[x1, x2, x3, ..., x t ,…,x T ] The words with a ratio of α in the list are replaced with the unified symbols of unregistered words. <unk>Get the replaced word sequence Will Input into the decoder and reconstruct and decode x.
[0023] 8. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 4, wherein the updating of parameters according to the objective function In the process of θ and θ, a modified KL cost annealing method is used. The KL weight is in vector form, and the dimension of the λ ratio in the KL weight is randomly set to 0 during the annealing process.
[0024] The beneficial effects of the present invention are: it can prevent the posterior collapse problem of variational autoencoders in text modeling; compared with word dropping and history-free decoding methods, it can more fully utilize the representation ability of the decoder; it has better probability density estimation and representation learning effects; the KL distance between the aggregated posterior probability distribution and the prior probability distribution of the latent variables is smaller, and high-quality and diverse data can be generated from a continuous and smooth latent variable space. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is a flow chart of the method for preventing posterior collapse of the variational autoencoder in text modeling according to the present invention.
[0026] Figure 2 This is a diagram of the model structure for preventing posterior collapse of the variational autoencoder in text modeling according to the present invention. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0028] like Figure 1 As shown, the technical solution for preventing the posterior collapse problem of the variational autoencoder in text modeling according to the embodiment of the present invention includes the following steps:
[0029] S1, obtain the text dataset and preprocess the data.
[0030] S2, build a variational autoencoder model, such as Figure 1 As shown, the variational autoencoder model includes an encoder and a decoder, and the encoder and the decoder adopt an autoregressive neural network.
[0031] S3, divide the text dataset described in S1 into a training set, a validation set, and a test set, input the data into the variational autoencoder model described in S2, and use the objective function L to optimize the model training. The objective function is:
[0032]
[0033] Where L represents the objective function, path is the path identifier of the decoder in the model, L(1) is the objective function when Path A and Path B exist at the same time, L(2) is the objective function when only Path A exists, p(X) represents the probability of the real text data X, x = [x1, x2, x3, ..., x t ,…,x T ] is a text instance from X, x t represents the word in the text instance x, and Represent the likelihood distribution of x learned by Path A and Path B in the decoder, α is the hyperparameter of the control word drop ratio in Path B, z represents the latent variable, represents the posterior probability distribution of the latent variable z corresponding to x learned by the encoder in the model, p(z) represents the prior probability distribution of the latent variable z, using the standard Gaussian distribution N(0,I), represents the parameters of the encoder, θ represents the parameters of the decoder, E.[·] represents the expectation, D KL [·||·] represents the KL divergence. In specific applications, the KL term weight is controlled by combining it with the basic KL annealing method.
[0034] S3 includes the following steps:
[0035] S31, initializing the parameters, hyperparameters, and Path identifiers of the variational autoencoder, including the encoder parameters And decoder parameters θ, the hyper parameter is α which controls the word discard ratio in Path B of the decoder, and the Path identifier path is set to True. In specific applications, the encoder parameters The decoder parameters θ can be initialized with a uniform distribution, and the word drop ratio α can be determined by adjusting the parameters of the dataset.
[0036] S32, dividing the training set into several small batches of data, training the optimization model using the objective function L for each small batch of data, and updating the path identifier path according to the path stopping strategy until convergence. The path stopping strategy is to set the path identifier path to True when the model is initialized, determine whether the model is close to convergence based on the number of learning rate decays, and if so, set the path identifier path to False.
[0037] S32 includes the following steps:
[0038] ① Input the small batch data into the encoder, for each text instance x=[x1,x2,x3,…,x t ,…,x T ]Get the posterior probability distribution The mean μ x and variance
[0039] ②Using the heavy parameter technology, from the posterior probability distribution Sampling from the , we get the latent variable z.
[0040] ③ If path is True, reconstruct x through Path A and Path B of the decoder, and use the gradient descent algorithm to update the parameters according to the objective function L(1) and θ; if path is False, reconstruct x through the decoder's Path A and use the gradient descent algorithm to update the parameters according to the objective function L(2) and θ.
[0041] like Figure 2 As shown, the decoder Path A is to convert the text instance x = [x1, x2, x3, ..., x t ,…,x T ] is input into the decoder to reconstruct and decode x; the decoder's Path B is to randomly convert the text instance x=[x1,x2,x3,…,x t ,…,x T ] The words with a ratio of α are replaced by unregistered words <unk>Get the replaced word sequence Will Input to the decoder to reconstruct and decode x. Path B forces the model to encode more information about the input data in the latent variables, while Path A combined with the Path stopping strategy enables the model to fully utilize the decoder's representation capabilities while avoiding the model entering the local optimal state when Path B exists.
[0042] In the above method, the parameters are updated according to the objective function. During the annealing process, a modified KL cost annealing method (KL cost annealing) is used. The KL weights are in vector form, and the dimension of the λ ratio in the KL weights is randomly reset to 0 during the annealing process. This KL weight zeroing method increases the number of active units in the latent variable, further improving the model's probability density estimation and representation learning. In specific applications, the weight zeroing ratio λ can be determined by adjusting the parameters of the dataset.
[0043] By adopting the above-mentioned technical solution disclosed in the present invention, the following beneficial effects are obtained: the method provided by the present invention can prevent the posterior collapse problem of the variational autoencoder in text modeling; compared with the word dropping and history-free decoding methods, it can make more full use of the representation ability of the decoder; it has better probability density estimation and representation learning effects; compared with the method of forcing the KL term to be greater than a certain positive value, the KL distance between the aggregated posterior probability distribution and the prior probability distribution of the latent variable is smaller, and high-quality and diverse data can be generated from a continuous and smooth latent variable space.
[0044] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0045] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0046] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0047] The above content is only for explaining the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.< / unk> < / unk>
Claims
1. A method for preventing posterior collapse in a variational autoencoder in text modeling, characterized in that: The variational autoencoder model is trained using the objective function, which is: Where L represents the objective function, path is the path identifier of the decoder in the model, L(1) is the objective function when Path A and Path B exist at the same time, L(2) is the objective function when only Path A exists, and p(X) represents the probability of the real text data x, x = [x1, x2, x3, ..., x t ,...,x T ] is a text instance from x, x t represents the word in the text instance x, and Represent the likelihood distribution of x learned by Path A and Path B in the decoder, α is the hyperparameter of the control word drop ratio in Path B, z represents the latent variable, represents the posterior probability distribution of the latent variable z corresponding to x learned by the encoder in the model, p(z) represents the prior probability distribution of the latent variable z, using the standard Gaussian distribution N(0, I), represents the parameters of the encoder, θ represents the parameters of the decoder, E.[·] represents the expectation, D KL [·||·] represents the KL divergence.
2. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 1, characterized in that The steps include: S1, obtain text dataset and preprocess the data; S2, constructing a variational autoencoder model, wherein the variational autoencoder model includes an encoder and a decoder, and the encoder and the decoder adopt an autoregressive neural network. S3, divide the text dataset described in S1 into a training set, a validation set, and a test set, input the variational autoencoder model described in S2, and use the objective function L to train and optimize the model.
3. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 2, characterized in that S3 includes the following steps: S31, initializing the parameters, hyperparameters, and Path identifiers of the variational autoencoder, including the encoder parameters and decoder parameters θ, the hyperparameter is α that controls the word discard ratio in Path B of the decoder, and the Path identifier path is set to True; S32, dividing the training set into several small batches of data, using the objective function L to train the optimization model for each small batch of data, and updating the Path identifier path according to the Path stop strategy until convergence.
4. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 3, wherein: S32 includes the following steps: ① Input the small batch data into the encoder, for each text instance x = [x1, x2, x3, ..., x t ,...,x T ]Get the posterior probability distribution The mean μ x and variance ②Using the heavy parameter technology, from the posterior probability distribution Sampling in the middle, we get the latent variable z; ③ If path is True, reconstruct x through the decoder's PathA and PathB, and use the gradient descent algorithm to update the parameters according to the objective function L(1) and θ; if path is False, reconstruct x through the decoder's PathA and use the gradient descent algorithm to update the parameters according to the objective function L(2) and θ.
5. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 3, wherein: The Path stopping strategy described in S32 is to set the Path identifier path to True when the model is initialized, and determine whether the model is close to convergence based on the number of learning rate decays. If so, set the Path identifier path to False.
6. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 4, characterized in that: The PathA of the decoder is to convert the text instance x=[x1, x2, x3, ..., x t ,...,x T ] is input into the decoder to reconstruct and decode x.
7. The method for preventing posterior collapse of a variational autoencoder in text modeling according to claim 4, wherein: The decoder's Path B is to randomly convert the text instance x=[x1, x2, x3, ..., x t ,...,x T ] in which the words with a ratio of α are replaced by the unregistered word unified symbol <unk> to obtain the replaced word sequence Will Input into the decoder to reconstruct and decode x.
8. The method for preventing posterior collapse in a variational autoencoder in text modeling according to claim 4, wherein: In the above method, the parameters are updated according to the objective function. In the process of θ and θ, the modified KL cost annealing method is adopted. The KL weight is in vector form, and the dimension of the λ ratio in the KL weight is randomly set to 0 during the annealing process.