A semi-supervised synonymous text rewriting method based on variational autoencoder

Abstract

The application discloses a semi-supervised synonymous text rewriting method based on a variational autoencoder, and comprises the following steps: S1, data collection; S2, model training; S3, text synonymous rewriting by using the obtained model. The scheme improves the performance of text rewriting by using an improved version of a differential autoencoder, and reduces the demand for the amount of artificially labeled data.

Description

Technical Field

[0001] This invention relates to the field of text rewriting, and in particular to a semi-supervised synonym rewriting method based on variational autoencoders. Background Technology

[0002] Paraphrasing has many important business applications. For example, for media professionals, paraphrasing can improve the speed and richness of content creation. In the foreign language education industry, paraphrasing plays a significant role in helping students learn more diverse grammatical structures and improve their reading comprehension. Summary of the Invention

[0003] The present invention mainly provides a high-performance semi-supervised synonym rewriting method based on variational autoencoder.

[0004] The present invention addresses the aforementioned technical problems primarily through the following technical solution: a semi-supervised synonym rewriting method based on a variational autoencoder, comprising the following steps:

[0005] S1, Data Collection;

[0006] S2, Model Training;

[0007] S3. Use the obtained model to perform text paraphrasing.

[0008] As a preferred option, step S1 specifically involves:

[0009] S101. Collect a sufficient number (more than 10,000 sentences) of unlabeled sentences, and the unlabeled sentences and subsequent application scenarios should come from the same field;

[0010] S102. Labeling Synonyms: First, randomly select a portion of sentences from the collected unlabeled sentences, and then have data labelers manually rewrite the selected sentences. Then, import the unlabeled sentences collected in the previous stage into a full-text search engine as a database, and then find sentences in the database that have the same meaning as the rewritten sentences to obtain synonyms, thereby obtaining training data.

[0011] Preferably, step S2 specifically involves optimizing the following loss function using the SGD optimization method for unlabeled data:

[0012]

[0013] In the formula, θ and φ are encoder parameters, τ is decoder parameter, λ is a constant, and q Ф (z|x,y) represents the output probability of the encoder model, where y represents the classification label, and p τ (x bow|z) represents the output probability of the decoder model, and z is a spatial vector;

[0014] For data with synonymous rewrites, the second loss function is optimized using the SGD optimization method. The second loss function is:

[0015]

[0016] x' is a paraphrase of x, D KL For KL divergence, the subscript "bow" indicates that sentence x is represented using a bag-of-words approach;

[0017] Training is completed by alternately optimizing two loss functions.

[0018] Preferably, the first loss function is obtained by combining the loss function of the standard variational autoencoder model and the bag-of-words function. The loss function of the standard variational autoencoder model is:

[0019]

[0020] The bag-of-words function is:

[0021]

[0022] w i Given a list of the i-th words in text x, f τ It is the second encoder, used to transform the vector space z to the vector space. V represents the vocabulary, and T represents the number of words.

[0023] Preferably, step S3 specifically involves: given any sentence, encoding it using an encoder to obtain z, then sampling q(z|x) and inputting it into a decoder. The decoder's decoding strategy includes:

[0024] The first method: randomly sample from the softmax output layer (i.e., the last layer of the decoder);

[0025] The second method involves sorting the scores output by the softmax output layer from largest to smallest and then sampling the top K words.

[0026] The substantial effect of this invention is that it proposes a novel text rewriting technique that improves the performance of text rewriting through an improved version of the differential autoencoder and reduces the amount of data required for manual annotation. Attached Figure Description

[0027] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0028] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0029] Example:

[0030] Data collection:

[0031] 1. Collect a large number of unlabeled sentences (more than 10,000): First, we collect a large number of sentences from the Internet or historical data. These sentences do not need to be manually labeled, as long as they come from the same field as the application scenarios later.

[0032] 2. Synonym Labeling: For a small subset of data, we manually labeled whether two sentences were synonyms. To obtain potentially synonymous sentences, we used two methods for data collection. The first method involved randomly selecting a portion of sentences and having data labelers manually rewrite them. The second method involved importing the data collected in the previous stage into a full-text search engine and then searching the database for sentences with equivalent meanings to obtain synonyms.

[0033] Model training:

[0034] After obtaining the above two types of data, the next step is to train the improved version of the VAE (Variational Autoencoder) model of this invention.

[0035] First, we choose the structure of the VAE model. This invention does not impose any restrictions on this step, and various text encoders and decoders can be freely selected, such as LSTM (Long-short Term Memory) or Transformer.

[0036] After selecting the model architecture, we use the data collected in the previous step to train the model.

[0037] The standard VAE loss function ELBO can therefore be expressed as:

[0038]

[0039] However, the above function does not perform well in modeling text data due to the well-known posterior collapse phenomenon. Therefore, we propose a simple and effective auxiliary objective: bag-of-words loss. The idea is to introduce an auxiliary loss function that requires the decoder network to predict the bag of words in the response x. Suppose x = [w1, ... w2]... TLet be a list of T words in text x. Bag-of-words prediction employs conditional independence, which forces latent variables to capture global information about the target response. In the VAE and fully latent action cases, we add a separate decoder fτ to obtain samples of z from the recognition network, thereby independently predicting the probabilities of words in x, specifically, it can be represented as:

[0040]

[0041] By combining our proposed bag-of-words function with the ordinary VAE loss, we can obtain a new loss function:

[0042]

[0043] In fact, fτ can be implemented as a multilayer fully connected neural network (MLP). We will demonstrate that bag-of-words loss in the new loss function is very effective in preventing back-end collapse.

[0044] Based on the loss function described above, we employ the following process to train and utilize the text paraphrasing model. For unlabeled data, we optimize the loss function using the SGD (Stochastic Gradient Descent) optimization method. For supervised data, i.e., data with paraphrasing, we optimize the loss function using SGD as follows:

[0045]

[0046] In other words, we replace the original function's x with a paraphrased sentence x' as the target of the decoder. By alternately optimizing the two loss functions, we complete the training of the VAE model.

[0047] Finally, we can use the VAE model obtained above to perform text paraphrasing. Given any sentence, we first encode it using an encoder to obtain z, and then sample q(z|x) and input it to the decoder to obtain the paraphrased version of the input sentence. The decoder's decoding strategy includes:

[0048] 1. Randomly sample from the softmax output layer.

[0049] 2. Sample the top K words in the softmax output layer.

[0050] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0051] Although this paper makes frequent use of terms such as annotation, synonym, and loss function, the possibility of using other terms is not excluded. These terms are used merely to facilitate the description and explanation of the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A semi-supervised synonym rewriting method based on variational autoencoder, characterized in that, Includes the following steps: S1, Data Collection; S2. Train the model using the collected data; S3. Use the obtained model to perform text paraphrasing; Step S1 is as follows: S101. Collect a sufficient number of unlabeled sentences, and ensure that the unlabeled sentences and subsequent application scenarios come from the same domain; S102, Labeling Synonyms: First, randomly select a portion of sentences from the collected unlabeled sentences, and then have data labelers manually rewrite the selected sentences; then import the unlabeled sentences collected in the previous stage into a full-text search engine as a database, and then find sentences in the database that have the same meaning as the rewritten sentences to obtain synonyms, thereby obtaining training data; Step S2 specifically involves optimizing the first loss function using the SGD optimization method for unlabeled data. The first loss function is: In the formula, x represents the sentence, θ and φ are encoder parameters, τ is decoder parameters, λ is a constant, and q Ф (z|x,y) represents the output probability of the encoder model, where y represents the classification label, and p τ (x bow |z) represents the output probability of the decoder model, and z is a spatial vector; For data with synonymous rewrites, the second loss function is optimized using the SGD optimization method. The second loss function is: x' is a paraphrase of x, D KL For KL divergence, the subscript "bow" indicates that sentence x is represented using a bag-of-words approach; Training is completed by alternately optimizing two loss functions; The first loss function is obtained by combining the loss function of the standard variational autoencoder model and the bag-of-words function. The loss function of the standard variational autoencoder model is: The bag-of-words function is: w i Given a list of the i-th words in text x, f τ It is the second encoder, used to transform the vector space z to the vector space. V represents the vocabulary.

2. The semi-supervised synonym rewriting method based on variational autoencoder according to claim 1, characterized in that, Step S3 specifically involves encoding an arbitrary sentence using an encoder to obtain z, then sampling q(z|x) and inputting it into a decoder. The decoder's decoding strategy includes: The first method: randomly sample from the softmax output layer; The second method involves sorting the scores output by the softmax output layer from largest to smallest and then sampling the top K words.

Images