A Keyword-Based Controllable Text Retelling Generation Method and Apparatus

Abstract

This invention relates to a keyword-based controllable text paraphrasing generation method and apparatus. The method includes: extracting keywords from the text to be paraphrased to obtain a keyword list; sequentially splitting each keyword in the keyword list into individual characters, and using each individual character as a token to obtain a token list; inputting the text to be paraphrased into an mT5 model for iterative text paraphrasing generation to obtain a paraphrased text set; wherein each round of iterative text paraphrasing generation includes: inputting the input text into the mT5 model to obtain the output text; performing a beam search on the output text to obtain a first result text; selecting the first token not added to the input text from the token list and adding it to the end of the input text to obtain a second result text; combining the first result text and the second result text to obtain a generated result set; and performing a beam search on the generated result set, using the search result as the input text for the next round of iteration.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to a method and apparatus for generating controllable text paraphrasing based on keywords. Background Technology

[0002] Text paraphrasing methods essentially involve generating similar sentences from given text, thereby training a model to achieve the desired generation. Many methods exist for this purpose. The mainstream generation or decoding methods are mainly four types: Greedy search, Beam search, Top-K sampling, and Top-P sampling.

[0003] Greedy search simply selects the word with the highest probability as the output word when generating the next word. The method is simple, but it is prone to the problem of generating duplicate words.

[0004] Beam Search can solve the problem of repetition by retaining the n beams with the highest probabilities at each time step and ultimately selecting the hypothesis with the highest overall probability, thus reducing the risk of missing hidden high-probability words. Beam Search works well if the generated length is more or less predictable, such as in machine translation or summarization. However, in open-ended generation scenarios, Beam Search performs poorly because the generated length can vary significantly.

[0005] Random sampling involves randomly sampling when generating the next token. This gives the generated text randomness and prevents mechanical repetition, but it can lead to disjointed text context and strange sentences.

[0006] Top-K sampling improves upon random sampling by addressing the issues of incoherent context and strange sentence generation that arise with random sampling. It selects the K tokens with the highest probabilities to form a probability distribution and then performs multinomial sampling. This ensures coherent sentences without generating strange or repetitive ones. However, fixing the value of K can lead to the selection of tokens with low or unusual probabilities. A K value that is too small results in poor sentence diversity, while a K value that is too large leads to incoherent sentences, similar to random sampling.

[0007] To address the K-value selection issue in Top-K sampling, Top-P sampling can accumulate probabilities from largest to smallest, dynamically selecting tokens. Whenever the accumulated probability of a token exceeds a P-value, these selected tokens are grouped into a new probability distribution and sampled using a multinomial distribution. This process continues until the next token, and so on, for decoding and sentence generation. A combination of Top-K and Top-P sampling can be used for generation / decoding.

[0008] However, current text paraphrasing still suffers from problems such as uncontrollable generated text and loss or error of key information. Specifically, the main reasons for poor generation results are:

[0009] (1) Key information is difficult to identify, making it difficult to extract keywords;

[0010] (2) The generation method is too simple, resulting in problems such as incoherent sentences, repeated sentences, and misunderstandings.

[0011] (3) It is uncontrollable or cannot be generated by combining key information according to human ideas; the controllable paraphrase generation task requires manual annotation or manual feature design. Therefore, it is also a challenge to select appropriate text for learning and achieve controllable paraphrase generation without relying on human intervention. Summary of the Invention

[0012] The purpose of this invention is to provide a keyword-based controllable text paraphrase generation method and apparatus, which aims to achieve controllable paraphrase generation tasks by extracting keywords and intervening in the generation process without relying on manual annotation.

[0013] Firstly, a keyword-based controllable text paraphrasing generation method is provided, including:

[0014] Obtain the corpus, use the corpus to train the mT5 model, and obtain the trained mT5 model for text paraphrasing;

[0015] Obtain the text to be paraphrased, extract keywords from the text to obtain a keyword list, and then break down each keyword in the keyword list into individual characters, using each individual character as a token to obtain a token list;

[0016] The text to be paraphrased is input into the trained mT5 model, and iterative text paraphrasing is performed to generate a set of paraphrased texts. Each round of iterative text paraphrasing includes:

[0017] Input the input text into the mT5 model to obtain the output text;

[0018] Perform a beam search on the output text to obtain the first result text;

[0019] Select the first token in the token list that has not been added to the input text, and add it to the end of the input text to get the second result text;

[0020] Combine the first result text with the second result text to obtain the generated result set;

[0021] Perform a beam search on the generated result set and use the search results as the input text for the next iteration.

[0022] In one possible implementation, the method of training the mT5 model using a corpus to obtain the trained mT5 model further includes:

[0023] The text data in the corpus is preprocessed to obtain standard sentences and their corresponding paraphrased sentences;

[0024] The standard sentence is input into the encoder of the mT5 model to obtain the hidden state information;

[0025] The hidden state information is input into the decoder of the mT5 model, and the restatement is used as a label to iteratively train the mT5 model to obtain the trained mT5 model.

[0026] In one possible implementation, the method for extracting keywords from the text to be restated to obtain a keyword list further includes:

[0027] Obtain the corpus needed for keyword extraction;

[0028] Inverse document frequency (IDF) calculations were performed on the corpus to obtain an IDF table;

[0029] The text to be paraphrased is segmented into words, and the segmentation results are filtered using a preset filtering scheme to obtain a list of candidate keywords.

[0030] Calculate the term frequency (TF) of each candidate keyword in the candidate keyword list in the text to be paraphrased;

[0031] Multiply the IDF value of each candidate keyword in the candidate keyword list by its TF value to obtain the TF-IDF value;

[0032] Sort each candidate keyword in the candidate keyword list from largest to smallest according to its TF-IDF value, select the top K candidate keywords, and arrange the K candidate keywords according to their order in the text to be paraphrased to obtain the keyword list, where K is a preset positive integer.

[0033] In one possible implementation, the method of filtering the word segmentation results using a preset filtering scheme to obtain a candidate keyword list further includes:

[0034] For the word list in the word segmentation results, only nouns, verbs and place names are retained, and the remaining words are removed to obtain a candidate keyword list.

[0035] In one possible implementation, the method for calculating the inverse document frequency (IDF) of the corpus to obtain an IDF table further includes:

[0036] Each text in the corpus is segmented into words. For each word, the IDF value of the word is obtained by dividing the total number of documents in the corpus by the number of documents in which the word appears.

[0037] Use the median of the IDF values ​​of all words as the IDF value for words that have not appeared.

[0038] Secondly, a keyword-based controllable text paraphrasing generation device is provided, comprising:

[0039] The model training unit is used to acquire a corpus and train the mT5 model using the corpus to obtain a trained mT5 model for text paraphrasing. The keyword extraction unit acquires the text to be paraphrased, extracts keywords from it, obtains a keyword list, and then breaks down each keyword into individual characters, treating each character as a token to obtain a token list. The paraphrased text generation unit inputs the text to be paraphrased into the trained mT5 model and iteratively generates paraphrased text, resulting in a set of paraphrased texts. Each iteration of text paraphrasing generation is implemented through the following modules:

[0040] The result generation module is used to input the input text into the mT5 model and obtain the output text; the result collection module is used to perform a beam search on the output text to obtain the first result text; the token addition module is used to select the first token that has not been added to the input text from the token list and add it to the end of the input text to obtain the second result text; the result processing module is used to combine the first result text and the second result text to obtain the generated result set; the beam search module is used to perform a beam search on the generated result set and use the search result as the input text for the next iteration.

[0041] This invention proposes a keyword-based controllable text paraphrase generation method and apparatus, which has the following advantages: Keyword extraction is performed using TF-IDF, resulting in simple computation and good performance. The mT5 model is used to generate paraphrased sentences. During the generation process, a beam search is performed by controlling the keywords to generate valid sentences that must contain those keywords. This combines human prior knowledge with the model's generative capabilities, effectively improving the controllability of the model's generation. Attached Figure Description

[0042] Figure 1 This is a flowchart of a keyword-based controllable text paraphrase generation method disclosed in an embodiment of the present invention;

[0043] Figure 2 This is a flowchart of a keyword extraction method disclosed in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the first round result of the iterative text paraphrase generation process in one embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of the second round result of the iterative text paraphrase generation process in one embodiment of the present invention;

[0046] Figure 5 This is a schematic block diagram of a keyword-based controllable text paraphrase generation device disclosed in an embodiment of the present invention. Detailed Implementation

[0047] Before providing a detailed description of the embodiments of this application, the terms used in the embodiments of this application will be explained first.

[0048] (1) Word frequency

[0049] Term frequency (TF) is the frequency with which a word appears in a document. For a given word, the TF is calculated as (the number of times the word appears in a document) / (the total number of words in the document).

[0050] (2) Inverse document frequency

[0051] Inverse Document Frequency (IDF) is calculated as (total number of documents in the corpus) / (number of documents containing a particular word). IDF reflects the distinctiveness of a word among documents. If a word appears only in one document, it means that this word can distinguish that document from others. In other words, IDF reflects the uniqueness of a word.

[0052] (3) Term frequency-inverse document frequency technique

[0053] Term Frequency-Inverse Document Frequency (TF-IDF) is a commonly used weighting technique in information retrieval and text mining. It is used to assess the importance of a word to a document within a document set or corpus. A word's importance increases proportionally to its frequency of occurrence in a document, but decreases inversely proportionally to its frequency of occurrence in the corpus. If a word is relatively rare but appears frequently in an article, it is likely to reflect the characteristics of that article and is precisely the keyword we need.

[0054] (4) mT5 model

[0055] The mT5 (Multilingual T5) model is a multilingual version of the T5 (Text-to-Text Transfer Transformer) model, including Chinese. As the original generative Seq2seq pre-trained language model, it uses the complete Transformer and constructs both unsupervised and supervised text generation and training tasks. Structurally, mT5 differs from T5 in that it uses the T5.1.1 scheme, where the first layer with ReLU activation is replaced with a GELU-activated gated linear unit. Furthermore, T5.1.1 modifies the embedding layer. Dropout is removed during pre-training and only used during fine-tuning.

[0056] (5) Beam Search

[0057] Beam search is an improvement on greedy search, expanding the search space. Beam search has a hyperparameter `beamsize` (bundle width), set to `k`. In the first step, it selects the `k` words with the highest conditional probabilities as candidate output words. In each subsequent step, based on the output sequence of the previous step, it selects the `k` words with the highest conditional probabilities from all combinations as candidate output sequences for that step, always maintaining `k` candidates. Finally, it selects the optimal result from these `k` candidates.

[0058] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0059] Figure 1 This is a flowchart of a keyword-based controllable text paraphrase generation method disclosed in an embodiment of the present invention.

[0060] In step S110, a corpus is obtained, and the mT5 model is trained using the corpus to obtain a trained mT5 model for text paraphrasing.

[0061] Specifically, the text data in the corpus is preprocessed into two columns: the first column contains standard sentences, and the second column contains paraphrased sentences corresponding to the first column. Then, "paraphrase:" (i.e., "paraphrase: standard sentence") is added to the beginning of all sentences in the first column. The paraphrased sentences in the second column remain unchanged and do not need to be modified. This is used to invoke the text paraphrasing function of the mT5 model.

[0062] The standard sentence is input into the encoder of the mT5 model to obtain the hidden state information. Then, the hidden state information is input into the decoder of the mT5 model for decoding. The restatement sentence is used as the label to obtain the loss output. The mT5 model is iteratively trained in the direction of reducing the loss, and finally the trained mT5 model is obtained.

[0063] In step S120, the text to be paraphrased is obtained, and keywords are extracted from the text to obtain a keyword list. Each keyword in the keyword list is then broken down into individual characters, and each individual character is used as a token to obtain a token list. The purpose of keyword extraction is to ensure that the model does not lose this important information during the generation process, or that the generated information will definitely contain these keywords.

[0064] Specifically, the first step is to obtain the corpus needed for keyword extraction. Each text in the corpus is then segmented into words, and the IDF value of each word is calculated. Each word and its IDF value are then combined to form an IDF table.

[0065] Then as Figure 2 As shown, the text to be paraphrased is segmented into words. Only nouns, verbs, and place names are retained from the segmentation results, and the remaining words are removed to obtain a candidate keyword list. The term frequency (TF) value of each candidate keyword in the candidate keyword list is calculated in the text to be paraphrased. The IDF value of each candidate keyword in the candidate keyword list is multiplied by the TF value to obtain the TF-IDF value. For words not found in the IDF table, the median of the IDF values ​​of all words is used as the IDF value of that word. Each candidate keyword in the candidate keyword list is sorted in descending order of its TF-IDF value. The top K candidate keywords are selected and arranged in order of their appearance in the text to be paraphrased to obtain the keyword list, where K is a preset positive integer.

[0066] Then iterate through the keyword list, break down each keyword into individual characters, treat each individual character as a token, and form a token list. Let n be the number of tokens in the token list.

[0067] Optionally, the above word segmentation process uses the jieba word segmentation tool.

[0068] In step S130, the text to be paraphrased is input into the trained mT5 model to perform iterative text paraphrasing generation, resulting in a paraphrased text set of the text to be paraphrased. Each round of iterative text paraphrasing generation includes the following steps S131 to S135.

[0069] For beam search, a positive integer beamsize is preset as the number of results to be retained in the search.

[0070] In step S131, the input text is input into the mT5 model to obtain the output text.

[0071] In step S132, a beam search is performed on the output text to obtain the first result text.

[0072] Specifically, for each input text, mT5 outputs several texts, and selects the top beamsize results with the highest probability from these texts. The beamsize results corresponding to each input text are then combined to obtain the first result text.

[0073] In step S133, select the first token that has not been added to the input text from the token list and add it to the end of the input text to obtain the second result text.

[0074] Specifically, add a STEP for each piece of text. i The token is a tag, where i is an integer and 0 ≤ i ≤ n, and n is the number of tokens in the token list. STEP i This indicates that i tokens have been added to this text. Initially, each text has a STEP... i The value of 'i' in the markers is set to 0. For each input text, access its STEP. i The token is marked, the value of i is read, and then the (i+1)th token is selected from the token list and added to the end of this input text, while STEP is applied. i Increment the value of i in the token by 1, and combine all the input text with the token added into the second result text.

[0075] In step S134, the first result text and the second result text are combined to obtain the generated result set.

[0076] In step S135, a beam search is performed on the generated result set, and the top beamsize results with the highest probability are selected and retained as search results. The search results are then used as the input text for the next iteration.

[0077] It can be understood that the search result of step S135 in the last round of iteration will be directly output and will not be used as the input text for the next round of iteration.

[0078] Figure 3 This is a schematic diagram of the results of the first round of the iterative text paraphrase generation process in an embodiment disclosed by the present invention. In this embodiment, text paraphrasing is performed on the text to be paraphrased "What's the weather like in Beijing today". In the keyword extraction step, two words, "Beijing" and "weather", are extracted as keywords, and then they are split into four single characters, "北", "京", "天", "气", as tokens to form a token list {"北", "京", "天", "气"}. At the same time, the beam size of beam search is set to 3.

[0079] As Figure 3 shown, taking the character "今" as the input for the first round of iteration, beam search is performed on the output result of the mT5 model, and the first 3 single characters with the highest probabilities, namely "天", "晚", "年", are retained, and then combined with the character "今" to form "今天", "今晚", and "今年", as the first result text {"今天", "今晚", "今年"}.

[0080] Then for the input text "今", access its STEP i mark, and read that i = 0, then add the first token "北" in the token list to the end of "今" to form "今北", and at the same time increase the value of i in its STEP i mark by 1, and then obtain the second result text {"今北"}. Combine the first result text and the second result text to obtain the generated result set {"今天", "今晚", "今年", "今北"}. Perform beam search on the generated result set, and retain the first 3 texts with the highest probabilities to obtain the search result {"今天", "今晚", "今北"}, as the input text for the second round.

[0081] Figure 4 This is a schematic diagram of the results of the second round of the iterative text paraphrase generation process in an embodiment disclosed by the present invention. As Figure 4 shown, taking {"今天", "今晚", "今北"} as the input for the second round of iteration, input "今天" into the mT5 model, perform beam search on the output result, and retain the first 3 single characters with the highest probabilities, "是", "的", "我", and then combine them with "今天" to form "今天是", "今天的", and "今天我". Similarly, input "今晚" and "今北" into the mT5 model, perform beam search on the output result, retain the first 3 single characters with the highest probabilities, and then combine them to obtain the first result text {"今天是", "今天的", "今天我", "今晚又", "今晚的", "今晚要", "今北方", "今北面", "今北边"}.

[0082] Then, for the input texts {"today", "tonight", "north today"}, access their STEP i markers. For "today", when reading i = 0, add the 1st token "north" in the token list to the end of "today" to form "today north", and at the same time increase the i value in its STEP i marker by 1 to become 1; for "tonight", when reading i = 0, add the 1st token "north" in the token list to the end of "tonight" to form "tonight north", and at the same time increase the i value in its STEP i marker by 1 to become 1; for "north today", when reading i = 1, add the 2nd token "beijing" in the token list to the end of "north today" to form "north today beijing", and at the same time increase the i value in its STEP i marker by 1 to become 2. Then obtain the second result texts {"today north", "tonight north", "north today beijing"}. Combine the first result texts with the second result texts to obtain the generated result set {"today is", "today's", "today me", "tonight again", "tonight's", "tonight will", "north today", "north today side", "north today direction", "today north", "tonight north", "north today beijing"}. Perform beam search on the generated result set, retain the first 3 texts with the highest probabilities, and obtain the search result {"north today beijing", "today north", "today is"} as the input text for the third round.

[0083] And so on. In the last round of iteration, retain the first 3 texts with the highest probabilities and output them.

[0084] Figure 5 It is a schematic block diagram of a keyword-based controllable text paraphrase generation device disclosed in an embodiment of the present invention. The device 500 includes:

[0085] A model training unit 510, configured to obtain a corpus, use the corpus to train the mT5 model, and obtain a trained mT5 model for text paraphrase; a keyword extraction unit 520, configured to obtain a text to be paraphrased, extract keywords from the text to be paraphrased to obtain a keyword list, and sequentially split each keyword in the keyword list into single characters, and use each single character as a token to obtain a token list; a paraphrase text generation unit 530, configured to input the text to be paraphrased into the trained mT5 model to perform iterative text paraphrase generation to obtain a set of paraphrase texts of the text to be paraphrased, where any round of iterative text paraphrase generation is implemented through the following modules:

[0086] The result generation module 531 is used to input the input text into the mT5 model to obtain the output text; the result collection module 532 is used to perform a beam search on the output text to obtain the first result text; the token addition module 533 is used to select the first token that has not been added to the input text from the token list and add it to the end of the input text to obtain the second result text; the result processing module 534 is used to combine the first result text and the second result text to obtain the generated result set; the beam search module 535 is used to perform a beam search on the generated result set and use the search result as the input text for the next iteration.

[0087] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A keyword-based controllable text paraphrase generation method, characterized in that, The method includes: Obtain a corpus, preprocess the text data in the corpus to obtain standard sentences and their corresponding paraphrased sentences; The standard sentence is input into the encoder of the mT5 model to obtain the hidden state information; The hidden state information is input into the decoder of the mT5 model, and the paraphrased sentences are used as labels to iteratively train the mT5 model to obtain the trained mT5 model, which is used for text paraphrasing. Obtain the text to be paraphrased, extract keywords from the text to obtain a keyword list, and then break down each keyword in the keyword list into individual characters, using each individual character as a token to obtain a token list; The text to be paraphrased is input into the trained mT5 model, and iterative text paraphrasing is performed to generate a set of paraphrased texts. Each round of iterative text paraphrasing includes: Input the input text into the mT5 model to obtain the output text; Perform a beam search on the output text to obtain the first result text; Select the first token in the token list that has not been added to the input text, and add it to the end of the input text to get the second result text; Combine the first result text with the second result text to obtain the generated result set; Perform a beam search on the generated result set and use the search results as the input text for the next iteration.

2. The method according to claim 1, characterized in that, The method for extracting keywords from the text to be restated to obtain a keyword list further includes: Obtain the corpus needed for keyword extraction; Inverse document frequency (IDF) calculations were performed on the corpus to obtain an IDF table; The text to be paraphrased is segmented into words, and the segmentation results are filtered using a preset filtering scheme to obtain a list of candidate keywords. Calculate the term frequency (TF) of each candidate keyword in the candidate keyword list in the text to be paraphrased; Multiply the IDF value of each candidate keyword in the candidate keyword list by its TF value to obtain the TF-IDF value; Sort each candidate keyword in the candidate keyword list from largest to smallest according to its TF-IDF value, select the top K candidate keywords, and arrange the K candidate keywords according to their order in the text to be paraphrased to obtain the keyword list, where K is a preset positive integer.

3. The method according to claim 2, characterized in that, The method for filtering the word segmentation results using a preset filtering scheme to obtain a list of candidate keywords further includes: For the word list in the word segmentation results, only nouns, verbs and place names are retained, and the remaining words are removed to obtain a candidate keyword list.

4. The method according to claim 2, characterized in that, The method for calculating the inverse document frequency (IDF) of the corpus to obtain an IDF table further includes: Each text in the corpus is segmented into words. For each word, the IDF value of the word is obtained by dividing the total number of documents in the corpus by the number of documents in which the word appears. Use the median of the IDF values ​​of all words as the IDF value for words that have not appeared.

5. A keyword-based controllable text paraphrasing generation device, comprising: The model training unit is used to acquire a corpus, preprocess the text data in the corpus, and obtain standard sentences and their corresponding paraphrased sentences. The standard sentence is input into the encoder of the mT5 model to obtain the hidden state information; the hidden state information is input into the decoder of the mT5 model, and the paraphrased sentence is used as a label to iteratively train the mT5 model to obtain the trained mT5 model, which is used for text paraphrasing. The keyword extraction unit is used to obtain the text to be paraphrased, extract keywords from the text to obtain a keyword list, and then split each keyword in the keyword list into individual characters, and use each individual character as a token to obtain a token list. The paraphrasing unit is used to input the text to be paraphrased into the trained mT5 model, perform iterative text paraphrasing generation, and obtain a set of paraphrased texts of the text to be paraphrased. Each round of iterative text paraphrasing generation is implemented through the following modules: The results generation module is used to input the input text into the mT5 model and obtain the output text; The result collection module is used to perform a beam search on the output text to obtain the first result text; The token addition module is used to select the first token in the token list that has not been added to the input text and add it to the end of the input text to obtain the second result text; The result processing module is used to combine the first result text and the second result text to obtain a set of generated results. The beam search module is used to perform beam search on the generated result set and use the search results as input text for the next iteration.

Images