A Problem Expansion Method Based on Conditional Variational Autoencoder
By using BERT and Transformer to generate extended problem sentences based on the conditional variational autoencoder CVAE, the problem of insufficient diversity in problem expansion in the prior art is solved, and efficient diversity generation and sample expansion are achieved.
Patent Information
- Application Number
- CN202111654035.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-30
- Publication Date
- 2025-07-25
- Estimated Expiration
- 2041-12-30
AI Technical Summary
The existing variational autoencoder VAE and cluster search Beam Search cannot effectively realize the diversity of problem expansion, and the traditional methods are inefficient and cannot meet the needs of problem expansion in intelligent question answering.
The conditional variational autoencoder CVAE is used to obtain the mean μ and variance σ2 using BERT, construct the mask z, and combine the Transformer's decoder to perform self-attention and ordinary attention calculations to generate multiple extended problem sentences.
Diversity extended question sentences were generated, increasing sample size, saving labor costs, fluency and diversity reached 95% and 85%.
Smart Images
Figure CN114330664B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to natural language processing technology in the field of computers, and particularly to a question expansion method based on a conditional variational autoencoder. Background Art
[0002] For generation tasks in natural language processing, such as intelligent question answering, neural machine translation, sentence paraphrasing, etc., there are usually problems of few training samples and poor generation diversity. Questions play an important role in intelligent question answering. Because in an open intelligent question answering system, the analysis and paraphrase expansion of questions is the key first step. Usually in this step, in order to improve the recall rate, the given standard questions will be expanded with various paraphrases for recombination. However, the expanded questions do not have good effects. Generally, the number of questions used as training samples is very small. If the traditional manual method is used for question expansion, the efficiency is low, the intensity is high, and the cycle is long, which obviously cannot meet the requirements of question expansion. Therefore, it is necessary to study question expansion technology, expand the existing standard questions, generate multiple similar expanded questions, so as to perform data augmentation, increase the number of training samples, and lay a solid foundation for the subsequent training and improvement of the model.
[0003] Question expansion is a key issue in natural language processing. The traditional solution is as follows:
[0004] Use an end-to-end model, namely Seq2seq for generation, such as using a variational autoencoder (VAE) to implement.
[0005] Use beam search to increase diversity.
[0006] The traditional variational autoencoder (VAE) model is a deep generative latent variable model that allows learning non-linear representations of high-dimensional inputs. It is achieved by learning the latent representation or hidden code z of the input x, so that the original input x can be well reconstructed from the hidden code z. The encoder of the VAE learns the hidden code z through a deterministic encoding function, which is actually a posterior distribution on the hidden code z. The posterior distribution is usually considered to be a Gaussian distribution. The VAE encourages the posterior distribution to be close to the prior distribution, which can be regarded as a standard normal distribution. The two sets of parameters of the normal distribution are the mean μ and the variance σ^2. The decoder of the VAE takes the hidden code z as the input and generates an observed value x. However, the actual effect is not ideal.
[0007] Specifically, the main reasons for the unsatisfactory question expansion effect are as follows:
[0008] The combination of the traditional variational autoencoder (VAE) and LSTM can generate free text, but it is not applicable to the expansion of given questions.
[0009] Using only beam search can generate multiple sentences and increase a certain degree of diversity. However, the sentences generated by beam search have a high degree of similarity and cannot meet the requirements of question expansion.
[0010] The technical problem addressed by the present invention is: the diversity of question expansion in generation tasks, that is, using a variational autoencoder (VAE) cannot achieve the expansion or paraphrasing of specific questions; using beam search cannot meet the requirements of question expansion. Summary of the Invention
[0011] In view of the problems existing in the prior art, the present invention provides a question expansion method based on a conditional variational autoencoder that can paraphrase or expand the original standard question.
[0012] The object of the present invention is achieved by the following technical solutions.
[0013] A question expansion method based on a conditional variational autoencoder (CVAE) includes the following steps:
[0014] Step 1): The encoding end of the CVAE uses BERT to obtain the mean μ and variance σ 2 ;
[0015] Step 2): Use the obtained mean μ and variance σ 2 to construct the masked z;
[0016] Step 3): The decoding end of the CVAE uses a Transformer, and the decoder of the Transformer is used for predictive output. The standard question is passed into the Transformer encoder to obtain the output;
[0017] Step 4): Combine an embedded representation of an expanded question with the masked z, pass it into the decoder of the Transformer for self-attention calculation, and then perform ordinary attention calculation with the output of the Transformer encoder;
[0018] Step 5): Expand the question sentences generated by the Transformer decoder to generate any number of k question sentences.
[0019] The specific steps implemented in the said Step 1) include:
[0020] Step 11): The input is a standard question and an expanded question of this question. Through embedding lookup, the word embedding representation, position embedding representation, and sentence embedding representation of the input are obtained;
[0021] Step 12) Input all the obtained embedding representations into BERT together. That is, through 12 layers of encoders, the self-attention mechanism is adopted in each layer of the encoder to learn information, and then the information learned previously is processed by a fully connected layer and passed to the next layer of the encoder; BERT adds a [CLS] flag at the beginning of the sentence, and the [CLS] of the last layer serves as the semantic information of the entire sequence or the whole sentence; the information of [CLS] is linearly transformed to obtain the mean μ and variance σ. 2 .
[0022] The specific steps implemented in the said Step 2) include:
[0023] Step 21) The mean μ and variance σ are obtained from Step 1). 2 , sample a z from N(μ, σ 2 ), which is equivalent to randomly sampling a β from N(0, 1), and then use the reparameterization trick to make z = μ + β * σ, thus obtaining the masked z.
[0024] Step 22) The masked z is merged with the input of the Transformer decoder, and the merged information is passed to the Transformer decoder together as the input.
[0025] The specific steps implemented in the said Step 3) include:
[0026] Step 31) The input of the Transformer encoder is a standard question, and its word embedding representation is obtained through embedding lookup, and then the position information is added and passed into the encoder.
[0027] Step 32) After passing through the self-attention mechanism, and then processed by the residual network and layer normalization, the output of the multi-head attention is obtained.
[0028] Step 33) After passing through the fully connected layer, the output of the encoder is obtained.
[0029] Step 34) The input of the Transformer decoder is an extended question, and its word embedding representation is obtained through embedding lookup, and then the position information is added to obtain the embedding representation information of the extended question.
[0030] The specific steps implemented in the said Step 4) include:
[0031] Step 41) Merge the embedding representation information of the extended question obtained in the previous step with the masked z obtained by Module B, and pass the merged information into the Transformer decoder.
[0032] Step 42) After passing through the self-attention mechanism, and then processed by the residual network and layer normalization, the output of the multi-head attention is obtained.
[0033] Step 43) Through the ordinary attention mechanism, and then after processing by the residual network and layer normalization, the output of the ordinary attention is obtained;
[0034] Step 44) Through the fully connected layer, the output of the Transformer decoder is obtained.
[0035] The specific steps implemented in the said step 5) include:
[0036] Step 51) When the Transformer decoder predicts the output of the first one, perform TopK processing to obtain the k words with the highest probabilities;
[0037] Step 52) Respectively use the k words as the starting words of the predicted sentence, and then sequentially pass through the decoder of the Transformer to predict and output the subsequent words, and finally obtain k extended problem sentences.
[0038] Compared with the prior art, the advantages of the present invention are as follows: Through operation, the present invention can effectively expand existing standard problems, generate k different extended problems, improve the diversity of samples, and greatly save labor costs. Since there is no good evaluation index for generation tasks such as problem expansion for the time being, an experimental method of manually evaluating the effect is adopted, and the evaluation is mainly divided into two aspects: fluency and diversity. Through experiments, the fluency is 95% and the diversity is 85%, thus verifying the effectiveness and rationality of the present invention. Description of the Drawings
[0039] Figure 1 It is a schematic diagram of the modules of the present invention.
[0040] Figure 2 It is a flowchart of the present invention. Detailed Embodiment
[0041] The present invention will be described in detail below in conjunction with the drawings in the specification and specific embodiments.
[0042] As Figure 1 shown, Module A: The encoding end of the CVAE uses BERT to replace the LSTM of the traditional VAE to obtain the mean μ and variance σ 2 ;
[0043] Module B: Use the mean μ and variance σ obtained by Module A 2 to construct the latent code z.
[0044] Module C: The decoding end of the CVAE uses Transformer, and the decoder of Transformer is used for predictive output. The encoder and decoder of Transformer respectively replace the LSTM of a traditional VAE.
[0045] Module D: Expand the question sentences generated by the Transformer decoder to generate any number of k question sentences.
[0046] As Figure 2 shown, the workflow of the present invention includes the following steps:
[0047] Step 1) Input the samples, namely the standard question and the expanded question, into Module A. After being processed by BERT, the outputs, namely the mean μ and the variance σ, are obtained. 2 ;
[0048] Step 2) Sample the outputs of Step 1 using the reparameterization trick to obtain the sampling result, namely the mask z.
[0049] Step 3) Input the standard question of the sample into the encoder of Module C to obtain the encoder output.
[0050] Step 4) Combine the embedded representation of an expanded question of the sample with the output mask z of Step 2, input it into the decoder of Module C. After one self-attention calculation, then perform a general attention calculation with the output of Step 3 to obtain the predicted output of the decoder.
[0051] Step 5) After being processed by Module D, multiple expanded questions are obtained.
[0052] The specific steps implemented in the above-mentioned Step 1) include:
[0053] Step 11) The input is a standard question and an expanded question of this question. Through embedding lookup, the word embedding representation, position embedding representation, and sentence embedding representation of the input are obtained.
[0054] Step 12) Input all the obtained embedding representations into BERT, that is, through 12 layers of encoders. In each layer of the encoder, the self-attention mechanism is adopted to learn information, and then the information learned before is processed by a fully connected layer and passed to the next layer of the encoder; BERT adds a [CLS] flag at the beginning of the sentence, and the [CLS] of the last layer serves as the semantic information of the entire sequence or the whole sentence; the information of [CLS] is linearly transformed to obtain the mean μ and the variance σ. 2 .
[0055] The specific steps implemented in the above-mentioned Step 2) include:
[0056] Step 21) The mean μ and the variance σ are obtained from Step 1. 2 , sample a z from N(μ, σ 2 ), which is equivalent to randomly sampling a β from N(0, 1), and then using the reparameterization trick to make z = μ + β * σ, so as to obtain the mask z.
[0057] Step 22) The mask z is merged with the input of the decoder of the Transformer, and the merged information is transmitted to the decoder of the Transformer as the input together.
[0058] The specific steps implemented in the said step 3) include:
[0059] Step 31) The input of the Transformer encoder is a standard problem. Its word embedding representation is obtained through embedding lookup, and then position information is added and transmitted to the encoder.
[0060] Step 32) Through the self-attention mechanism, and then processed by the residual network and layer normalization to obtain the output of the multi-head attention.
[0061] Step 33) Through the fully connected layer, the output of the encoder is obtained.
[0062] Step 34) The input of the Transformer decoder is an extended problem. Its word embedding representation is obtained through embedding lookup, and then position information is added to obtain the embedding representation information of the extended problem.
[0063] The specific steps implemented in the said step 4) include:
[0064] Step 41) The embedding representation information of the extended problem obtained in the previous step is merged with the mask z obtained by module B, and the merged information is transmitted to the decoder of the Transformer.
[0065] Step 42) Through the self-attention mechanism, and then processed by the residual network and layer normalization to obtain the output of the multi-head attention.
[0066] Step 43) Through the ordinary attention mechanism, and then processed by the residual network and layer normalization to obtain the output of the ordinary attention.
[0067] Step 44) Through the fully connected layer, the output of the Transformer decoder is obtained.
[0068] The specific steps implemented in the said step 5) include:
[0069] Step 51) When the Transformer decoder predicts the output of the first one, TopK processing is performed to obtain the k words with the highest probability.
[0070] Step 52) Respectively use the k words as the starting words of the predicted sentence, and then sequentially pass through the decoder of the Transformer to predict and output the subsequent words, and finally obtain k extended problem sentences.
[0071] Next, for the above modules and steps, combined with the corresponding legends, the following will be elaborated in detail.
[0072] Module A: Use BERT at the encoding end of the conditional variational autoencoder to obtain the mean μ and variance σ 2 .
[0073] For the sake of explanation, take the sample ["What is the acquisition channel of e-invoices? What ways can be used to obtain e-invoices?"] as an example. The first sentence is the standard question and the second sentence is the extended question.
[0074] The constructed input format is: [CLS] Standard question [SEP] Extended question [SEP]
[0075] Find the index ids corresponding to the characters in the standard question and the extended question in the vocabulary. The maximum length is 80, and those less than 80 are padded with 0.
[0076] Step A1) Perform an embedding lookup on the index ids through the embedding information of the entire vocabulary to obtain all the word embedding representations of the sample;
[0077] Step A2) Process the output of the previous step. Add the word embedding representations with the position embedding and sentence embedding information and input them into the 12-layer encoder together. The self-attention mechanism is adopted in each layer of the encoder to learn information, and then the information learned previously is processed through a fully connected layer and passed to the next layer of the encoder, layer by layer until the last layer. The [CLS] of the last layer serves as the semantic information of the entire sequence or the whole sentence. Since [CLS] is just a flag without obvious semantic information and, compared with other input words, more fairly integrates the semantic information of each input word, so choosing to use [CLS] also better represents the semantic of the whole sentence. The information of [CLS] is linearly transformed to obtain the mean μ and variance σ 2 .
[0078] Module B: According to the obtained mean μ and variance σ 2 Construct the latent code z, and then combine it with the input of the decoder of the Transformer in Module C.
[0079] Step B1) The mean μ and variance σ are obtained from Module A 2 , sample a z from N(μ, σ ２ ), which is equivalent to randomly sampling a β from N(0, 1), and then using the reparameterization trick to make z = μ + β * σ, so as to obtain the masked z;
[0080] Step B2) The masked z is merged with the input of the Transformer decoder, where the input of the decoder is the extended question[[“ <s>What methods can be used to obtain electronic invoices?] The embedded representation is added to the location information.[" <s>"] is the predicted starting word. The merged information is used as input and fed into the decoder of the Transformer for prediction, and the best result of the current word is output.
[0081] Module C: The decoding end of the CVAE uses the Transformer, and the decoder of the Transformer is used for prediction output. The encoder and decoder of the Transformer respectively replace the LSTM of a traditional VAE.
[0082] Step C1) The input to the Transformer encoder is the standard question [What is the acquisition channel of the electronic invoice? [SEP]], where [SEP] is the separator. The word embedding representation of the input is obtained through embedding lookup, and then position information is added and fed into the encoder;
[0083] Step C2) Self-attention mechanism operations are required. After each self-attention calculation, the output of the multi-head attention is obtained after processing with a residual network and layer normalization, and then the output of the encoder is obtained through a fully connected layer;
[0084] Step C3) Self-attention mechanism operations need to be performed 4 times, that is, step 2 is repeated 4 times;
[0085] Step C4) The input to the Transformer decoder is the extended question <s>”]What ways can be used to obtain e-invoices?]. Among them [" <s>"] is the starting word of the prediction. Its word embedding representation is obtained through an embedding lookup, and then position information is added to obtain the embedding representation information of the extended question;
[0086] Step C5) Merge the embedding representation information of the extended question obtained in the previous step with the mask z obtained by Module B, and the merged information is passed into the decoder of the Transformer.
[0087] Step C6) After 4 self-attention mechanism calculations, the output of the multi-head attention is obtained after being processed by the residual network and layer normalization;
[0088] Step C7) After 4 general attention mechanism calculations, the output of the general attention is obtained after being processed by the residual network and layer normalization;
[0089] Step C8) After each attention calculation, it passes through a fully connected layer to obtain the output of the Transformer decoder.
[0090] Module D: Expand the question sentence generated by the Transformer decoder to generate any number of k question sentences.
[0091] The maximum length of the generated extended question is 40, so the Transformer decoder needs to predict the generated words in sequence, [" <s>"] is the predicted starting word. Based on the existing information, the Transformer decoder predicts the first character, then adds the information of the first character to the known information and continues to predict the next character until the prediction is completed.
[0092] In step D1), when the Transformer decoder predicts the first character, it performs TopK processing, that is, selects the k characters with the highest prediction probabilities as the output results.
[0093] In step D2), each of the k characters is used as the first valid character of the predicted sentence. According to the existing information, the Transformer decoder predicts the next character respectively, then adds the information of the next character to the known information and continues to predict the next character until the prediction is completed. In this way, k extended sentences are obtained.
[0094] Experimental effect
[0095] Through operation, the present invention can effectively expand the existing standard problems and generate k different extended problems.
[0096] The following table is a partial example of problem expansion:
[0097]
[0098] It improves the diversity of samples and greatly saves labor costs. Since there is no good evaluation index for generation tasks such as problem expansion for the time being, an experimental method of manually evaluating the effect is adopted. The evaluation is mainly divided into two aspects: fluency and diversity. Through experiments, the fluency is 95% and the diversity is 85%, thus verifying the effectiveness and rationality of the present invention.
[0099] The present invention uses a conditional variational autoencoder (CVAE) to implement the function of problem expansion, providing a solution to the problem of few training samples in natural language processing text generation tasks. The conditional variational autoencoder (CVAE) is an architecture deformed based on the variational autoencoder (VAE), which plays a strengthening role for the sequence-to-sequence Seq2seq model. However, the conditional variational autoencoder (CVAE) is different from the traditional VAE and unconditional sentence generation models. Conditional restrictions are imposed on both the encoding end and the decoding end of the VAE on the input sentence, that is, restricted to the original standard problem, so that the original standard problem can be paraphrased or expanded.< / s> < / s> < / s> < / s> < / s>
Claims
1. A problem expansion method based on conditional variational autoencoder, characterized in that: It includes the following steps: Step 1): The encoding end of the CVAE uses BERT to obtain the mean μ and variance σ 2 ; Step 2): Use the obtained mean μ and variance σ 2 Construct the mask z; Step 3): The decoder of the CVAE uses Transformer. The decoder of Transformer is used for predictive output. The standard question is passed into the Transformer encoder to obtain the output; Step 4): The embedding representation of an extended question is merged with the mask z and passed into the decoder of Transformer for self-attention calculation, and then ordinary attention calculation is performed with the output of the Transformer encoder; Step 5): The question sentence generated by the Transformer decoder is extended to generate any number of k question sentences.
2. The problem extension method based on conditional variational autoencoder according to claim 1, wherein: The specific steps implemented in the said Step 1) include: Step 11): The input is a standard question and an extended question of this question. Through embedding lookup, the word embedding representation, position embedding representation, and sentence embedding representation of the input are obtained; Step 12) Input all the obtained embedding representations into BERT together, that is, through 12 layers of encoders. The self-attention mechanism is adopted in each layer of the encoder to learn information, and then the information learned previously is processed by a fully connected layer and passed to the next layer of the encoder; BERT adds a [CLS] flag at the beginning of the sentence, and the [CLS] of the last layer serves as the semantic information of the entire sequence or the whole sentence; the information of [CLS] is linearly transformed to obtain the mean μ and variance σ 2 .
3. The problem expansion method based on conditional variational autoencoder according to claim 1, wherein: The specific steps implemented in the said Step 2) include: Step 21) The mean μ and variance σ are obtained from Step 1) 2 , sample a z from N(μ, σ 2 ), which is equivalent to randomly sampling a β from N(0, 1), and then using the reparameterization trick to make z = μ + β * σ, thus obtaining the mask z; Step 22): The mask z is merged with the input of the Transformer decoder, and the merged information is passed into the Transformer decoder together as the input.
4. The problem expansion method based on conditional variational autoencoder according to claim 1, wherein: The specific steps implemented in the said Step 3) include: Step 31): The input of the Transformer encoder is a standard question. Its word embedding representation is obtained through embedding lookup, and then position information is added and passed into the encoder; Step 32): Through the self-attention mechanism, and then the output of the multi-head attention is obtained after being processed by the residual network and layer normalization; Step 33): Through the fully connected layer, the output of the encoder is obtained; Step 34): The input of the Transformer decoder is an extended question. Its word embedding representation is obtained through embedding lookup, and then position information is added to obtain the embedding representation information of the extended question.
5. The problem expansion method based on conditional variational autoencoder according to claim 1, characterized in that: The specific steps implemented in the said Step 4) include: Step 41): The embedding representation information of the extended question obtained in the previous step is merged with the mask z obtained by module B, and the merged information is passed into the Transformer decoder; Step 42): Through the self-attention mechanism, and then the output of the multi-head attention is obtained after being processed by the residual network and layer normalization; Step 43): Through the ordinary attention mechanism, and then the output of the ordinary attention is obtained after being processed by the residual network and layer normalization; Step 44): Through the fully connected layer, the output of the Transformer decoder is obtained.
6. The problem extension method based on conditional variational autoencoder according to claim 1, wherein: The specific steps implemented in the said Step 5) include: Step 51): When the Transformer decoder predicts the output of the first one, TopK processing is performed to obtain the k words with the highest probability; Step 52): Respectively take the k words as the starting words of the predicted sentence, and then sequentially pass through the Transformer decoder to predict and output the subsequent words, and finally obtain k extended question sentences.