A paraphrase sentence generation method based on template sentence enhancement

Abstract

The application discloses a kind of based on template sentence enhancement's paraphrase sentence generation method, steps are as follows: step 1: obtain original sentence, and obtain the target paraphrase sentence of original sentence in parallel corpus, use the method of rule to retrieve 1 sentence from parallel predata with the highest similarity with target paraphrase sentence as optimal sentence as reference;Step 2: different part-of-speech special symbols are used to replace noun, verb, adjective and adverb in optimal sentence to construct template sentence;Step 3: original sentence and template sentence are spliced through special symbol and joint, and the spliced sentence is used as the input of paraphrase generation model, and the paraphrase generation model outputs several paraphrase sentences;Step 4: the similarity of several paraphrase sentences and original sentence in grammatical structure and semantics is calculated, and the highest similarity paraphrase sentence is used as the optimal paraphrase sentence of original sentence.The application avoids the information dependence of model training phase in template sentence by masking the words of related part-of-speech in template sentence.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and specifically to a method for generating paraphrased sentences based on template sentence enhancement. Background Technology

[0002] Paraphrasing is an important research direction in the field of natural language processing, with wide-ranging applications such as machine translation, automatic question answering, and information retrieval. It is also a valuable method for data augmentation. Paraphrasing refers to generating paraphrased sentences with the same semantics but different grammatical structures from a given sentence. For humans, paraphrasing involves two main considerations: first, the semantic meaning expressed by the original sentence; and second, the grammatical structure used to paraphrase it while preserving the semantic meaning. These same questions apply to pre-trained models, which must not only understand the semantics of the text but also be instructed on how to express this information. Natural language generation methods simulate human paraphrasing behavior, that is, first understanding the semantic information of the sentence and then expressing it in a different syntactic form.

[0003] Early research on paraphrasing primarily employed rule-based and dictionary-based methods, which suffer from poor scalability and limited variety in generated paraphrases. Furthermore, paraphrasing generated by statistical machine translation methods often exhibits significant semantic differences from the original sentences. While neural networks and pre-trained models have demonstrated powerful advantages in semantic learning and feature representation, the generated paraphrases still lack grammatical diversity. Summary of the Invention

[0004] This invention provides a method for generating paraphrased sentences based on template sentence enhancement, in order to solve the problem of the lack of diversity in the grammatical structure of paraphrased sentences generated in existing paraphrasing techniques.

[0005] This invention provides a method for generating paraphrased sentences based on template sentence enhancement, comprising the following steps:

[0006] Step 1: Obtain the original sentence and the target paraphrase of the original sentence in the parallel corpus. Using the target paraphrase as a reference, use the rule-based method to retrieve the sentence with the highest similarity to the target paraphrase from the parallel corpus as the optimal sentence.

[0007] Step 2: Perform part-of-speech analysis on the words in the optimal sentence, and use different special symbols for part-of-speech to replace nouns, verbs, adjectives and adverbs in the optimal sentence to construct template sentences;

[0008] Step 3: Connect the original sentence and the template sentence by splicing special symbols, and use the spliced ​​sentence as the input to the paraphrase generation model. The paraphrase generation model outputs several paraphrased sentences.

[0009] Step 4: Calculate the similarity between several paraphrases and the original sentence in terms of grammatical structure and semantics, and select the paraphrases with the highest similarity as the optimal paraphrases of the original sentence.

[0010] Furthermore, in step 3, the paraphrase generation model is an improved paraphrase generation model, specifically:

[0011] The token vector, position vector, paragraph vector, and edit vector are used together as word vectors; each encoder layer is equipped with a multi-head attention layer, layer normalization layer, feed-forword layer, and layer normalization layer.

[0012] Furthermore, in step 3, the original sentence and the template sentence are connected by special symbols and used as an editing vector.

[0013] Furthermore, the word vector dimension in the paraphrase generation model is 768; the number of encoders is 12.

[0014] Furthermore, each Multi-head Attention layer in the paraphrasing generation model has 12 attention heads.

[0015] Furthermore, the objective function of the restatement generation model is:

[0016]

[0017] In the formula, x is the original sentence; e is the template sentence; y is the generated restatement sentence; θ is the model parameter; and T is the number of generated words.

[0018] The beneficial effects of this invention are:

[0019] The present invention replaces words with relevant parts of speech in the template sentence with corresponding special symbols. While preserving the complete grammatical structure of the template sentence, it does not affect the generation ability of the pre-trained model. It follows the idea of ​​controlling the paraphrase generation with template sentences. By masking words with relevant parts of speech in the template sentence, it avoids the model's dependence on information in the template sentence during the training stage.

[0020] The paraphrase generation model of this invention incorporates an edit vector at the input end to enhance the model's learning of the features of the paraphrase formed by editing the original sentence. By adopting the idea of ​​editing vectors, the model can pay more attention to the important semantic information in the original sentence. Attached Figure Description

[0021] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0022] Figure 1 This is a flowchart of a specific embodiment of the present invention;

[0023] Figure 2 This is a framework diagram of the paraphrase generation model in a specific embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram illustrating the construction of template sentences in a specific embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the construction of edit vectors in a specific embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, this embodiment of the invention provides a method for generating paraphrased sentences based on template sentence enhancement, including the following steps:

[0028] Step 1: Obtain the original sentence and the target paraphrase of the original sentence in the parallel corpus. Using the target paraphrase as a reference, use the rule-based method to retrieve the sentence with the highest similarity to the target paraphrase from the parallel corpus as the optimal sentence.

[0029] The following rules must be followed when searching for template sentences:

[0030] The candidate template sentence is different from both the original sentence and the reference paraphrase sentence;

[0031] The length difference between the candidate template sentence and the reference paraphrase sentence shall not exceed 2;

[0032] The BLEU score of the candidate template sentence and the reference paraphrase sentence is less than 0.6;

[0033] The candidate template sentence and the reference paraphrase sentence have the lowest translation edit rate (TER).

[0034] like Figure 3 As shown, the original sentence "What is the best way to maintain a relationship" has the highest similarity among the sentences retrieved from the parallel corpus as "How do I train muscle effectively?".

[0035] Step 2: Perform part-of-speech analysis on the words in the optimal sentence, and use different special symbols for part-of-speech to replace nouns, verbs, adjectives and adverbs in the optimal sentence to construct template sentences;

[0036] Special symbols for parts of speech can be used, but are not limited to, replacing adjectives with "[ADJ]", adverbs with "[ADV]", nouns with "[NOUN]", and verbs with "[VERB]".

[0037] like Figure 3 As shown, after selecting the sentence with the highest similarity, "How do I train muscle effectively," in step 1, the template sentence constructed in step 2 is "How do I [VERB][NOUN][ADV]."

[0038] The current mainstream approach is to parse the template sentence into a syntactic parsing tree to guide the generation of restatements. However, during model training, a complete syntactic parsing tree can introduce noisy data, while a truncated syntactic parsing tree template can result in incomplete restatements. Therefore, the completeness of the syntactic parsing tree, the parts of speech of words, and the dependencies between words are crucial for generating restatements. The method proposed in this invention can solve this problem by masking words with relevant parts of speech in the template sentence, thus avoiding complete reliance on information from the template sentence during the model training phase.

[0039] Step 3: Connect the original sentence and the template sentence by splicing special symbols, and use the spliced ​​sentence as the input to the paraphrase generation model. The paraphrase generation model outputs several paraphrased sentences.

[0040] In this step, n paraphrases are generated based on the original sentence and the template sentence as input to the model, resulting in a reference paraphrase similar to the final result "How do I maintain a relationship?"

[0041] Among them, "[SEP]" can be used as a special symbol for splicing, but is not limited to it.

[0042] like Figure 2As shown, the input to the paraphrase generation model is a combination of the original sentence, the template sentence, and the reference paraphrase sentence. A separator “[SEP]” is added between the original sentence and the template sentence, and between the template sentence and the reference paraphrase sentence. The concatenated sentence is then input into the UniLM pre-trained model. In the input layer, token vectors, segment vectors, and edit vectors are used together as word vectors. The UniLM model directly transforms Seq2Seq into a sentence completion task. For example, given the input sentence “Where is the capital of China” and the target sentence “The capital of China is Beijing”, the UniLM model concatenates the input and target sentences into the form “[CLS]Where is the capital of China[SEP]The capital of China is Beijing” for training. When predicting the result, it only needs to input “[CLS]Where is the capital of China[SEP]” to predict “The capital of China is Beijing” word by word until “[SEP]” appears. During the training phase, the attention matrix uses a special mask pre-training objective. The input part of the model's attention is bidirectional, while the output part's attention is unidirectional, which also satisfies the Seq2Seq requirement.

[0043] The paraphrase generation model is an improved paraphrase generation model, specifically:

[0044] The paraphrase generation model is constructed from the open-source UniLM pre-trained model and fine-tuned based on paraphrase parallel corpora. The word vector dimension in the paraphrase generation model is 768, and the number of encoders is 12.

[0045] In this method, token vectors, position vectors, paragraph vectors, and edit vectors are used together as word vectors. Each encoder layer consists of a multi-head attention layer, a layer normalization layer, a feed-forword layer, and a layer normalization layer stacked together. Each multi-head attention layer has 12 attention heads.

[0046] The original sentence and the template sentence are connected by special symbols and then used as an editing vector.

[0047] like Figure 4As shown, the edit vector refers to the editing operations performed when the original sentence is transformed into a reference restatement by adding, deleting, and replacing words. These editing operations are represented by edit vectors. Edit vectors are mainly used to enhance the language model by concentrating probabilities on a few important values. The edit vector z is represented by the word vectors inserted and deleted during the transformation of the original sentence x into the reference restatement y. The word vectors are 300-dimensional GloVe word vectors. Formally, I = x\y is defined as the set of words added to y, and D = y\x is the set of words deleted from y. The difference between x and y is represented by the following formula, where Φ(w) represents the word vector of word w. This indicates vector concatenation.

[0048]

[0049] The paraphrasing generation model employs a bundle search-based decoding strategy to ensure the fluency and diversity of the generated text. Specifically, at each time step, only the top K words with the highest probabilities are sampled, ultimately returning a set of decoded sequences whose sum of elements is at most p. In the first time step, the k words with the highest current conditional probabilities are selected as candidate output sequences. In subsequent time steps, based on the output sequence from the previous time step, the k words with the highest conditional probabilities are selected from all sequences. Finally, the optimal sequence is chosen from the k candidates and combined to form a set of decoded sequences whose sum of elements is at most p.

[0050] Step 4: Calculate the similarity between several paraphrases and the original sentence in terms of grammatical structure and semantics, and select the paraphrases with the highest similarity as the optimal paraphrases of the original sentence.

[0051] like Figure 2 In the process, the similarity between the original sentence "What is the best way to maintain a relation?" and the reference paraphrase is calculated, and the paraphrase with the highest similarity is "How do I maintain this relation?".

[0052] The paraphrase generation model generates m paraphrases from the original sentence. This requires selecting paraphrases from these m paraphrases that are semantically and grammatically similar to the original sentence and the template sentence. Simply finding the optimal result based on word alignment between the paraphrases and the reference paraphrases does not reflect the diversity of the paraphrases' grammatical structures and their semantic consistency with the original sentence. Therefore, it is necessary to calculate the similarity between the paraphrases and the reference paraphrases. This similarity calculation involves encoding both the m paraphrases and the reference paraphrases into vectors and using cosine similarity to calculate the semantic similarity between the pairs of sentences. The cosine similarity formula is as follows:

[0053]

[0054] Where A and B represent vectors, i is the i-th vector value, and n represents the number of vector elements.

[0055] The syntactic diversity of the paraphrased sentences generated by the paraphrasing model specifically refers to calculating the tree-edit distance (TED) between m paraphrased sentences and the reference paraphrased sentence. A smaller TED value indicates higher syntactic similarity. The TED value not only measures the syntactic structural consistency between the generated paraphrased sentence and the template sentence from a syntactic structure perspective, but also evaluates the syntactic structural diversity of the generated paraphrased sentences.

[0056] To verify the performance of the template-based paraphrase generation model described in this application, this embodiment trains the model on the paraphrase parallel corpora Quora and ParaNMT-small, and compares it with other published paraphrase models on the metrics ROUGE-1 / 2, BLEU, METEOR, TED-R, and TED-E. TED-R refers to the TED value between the syntactic component analysis trees of the paraphrase and the reference paraphrase, and TED-E refers to the TED value between the syntactic component analysis trees of the paraphrase and the template sentence. A smaller TED value indicates higher syntactic similarity. The training, validation, and test sets of the Quora and ParaNMT-small datasets are shown in Table 1.

[0057] Dataset Name training set Validation set test set Quora 140K 3K 3K ParaNMT-small 500K 0.5K 0.8K

[0058] Table 1

[0059] Through specific experiments, the hardware environment was set to Nvidia GeForce GTX 1080Ti, and the model framework was Tensorflow. The Microsoft UniLM model was used to train the retelling generation task model. During the model training phase, the batch size was set to 64, the Dropout rate to 0.1, and the learning rate to 1e-5. The Adam optimizer was used to optimize the model. Table 2 shows the comparison results of different models on the Quora and ParaNMT-small datasets:

[0060]

[0061] Table 2

[0062] The paraphrase generation model includes different modules, including template sentences, edit vectors, and template sentences. To verify the impact of these modules on the paraphrase generation results, ablation experiments were conducted on these modules on the Quora dataset in actual experiments. The results are shown in Table 3.

[0063]

[0064] Table 3

[0065] Through practical experiments and verification, examples of paraphrasing generated by the template-based paraphrasing generation model are shown below.

[0066] Table 4 shows the original sentence, e the template sentence, y the generated restatement sentence, and y' the reference restatement sentence:

[0067]

[0068] Table 4

[0069] Compared with existing technologies, this application achieves more diverse grammatical structures for generating paraphrases by sharing model implementation parameters and using different template sentences to guide the generation of paraphrases.

[0070] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A paraphrase sentence generation method based on template sentence enhancement, characterized in that, Includes the following steps: Step 1: Obtain the original sentence and the target paraphrase of the original sentence in the parallel corpus. Using the target paraphrase as a reference, use the rule-based method to retrieve the sentence with the highest similarity to the target paraphrase from the parallel corpus as the optimal sentence. Step 2: Perform part-of-speech analysis on the words in the optimal sentence, and use different special symbols for part-of-speech to replace nouns, verbs, adjectives and adverbs in the optimal sentence to construct template sentences; Step 3: Concatenate the original sentence and the template sentence using special symbols to create an edit vector. Use the concatenated sentence as input to the paraphrase generation model. The paraphrase generation model outputs several paraphrased sentences. Specifically, this is an improved paraphrase generation model: The token vector, position vector, paragraph vector, and edit vector are used together as word vectors; each encoder layer consists of a multi-head attention layer, a layer normalization layer, a feed-forword layer, and a layer normalization layer. Each multi-head attention layer has 12 attention heads. The objective function of the generative model is as follows: ； In the formula, x is the original sentence; e is the template sentence; y is the generated paraphrase sentence; is the model parameter; T is the number of generated words; Step 4: Calculate the similarity between several paraphrases and the original sentence in terms of grammatical structure and semantics, and select the paraphrases with the highest similarity as the optimal paraphrases of the original sentence.

2. The template sentence-based paraphrase sentence generation method of claim 1, wherein, The word vector dimension in the paraphrasing generation model is 768; the number of encoders is 12.

Images