Text enhancement method based on dual diversity generation

Abstract

The invention discloses a text enhancement method based on dual diversity generation. The text enhancement method specifically comprises the following steps: firstly, carrying out word segmentation on text data and carrying out word vector representation through an ELMo model; performing smooth inverse frequency weighting on the obtained word vector to obtain word embedding and sentence embedding, and better capturing a central topic; the maximum boundary correlation between word embedding and sentence embedding is calculated, keyword diversity is introduced, Top K keywords are selected as finally extracted keywords according to a sorting result, and the semantic repetition problem of the extracted keywords is solved; the extracted keywords are mapped back to the position of the original text, the rest part is replaced with the MASK token, and meanwhile, a draft is constructed by adding a label as a prompt to the front of the text; and inputting the draft into a text enhancement model GENIUS to generate a new sample, and introducing sample diversity. The sample quality of text enhancement is improved, and the problem that the enhanced sample lacks diversity is solved.

Description

Technical Field

[0001] The present invention relates to a text enhancement method in the technical field of natural language processing, and in particular to a text enhancement method based on dual diversity generation. Background Art

[0002] Text classification refers to the task of annotating an existing text according to a defined category label. By automatically classifying the text into different categories through text classification, it can assist decision-making for humans and improve work efficiency. With the explosive growth of social information, the categories of text classification are also increasingly showing a huge trend, and most categories have only a small number of samples, and these categories are crucial for text classification tasks. Therefore, the present invention combines text enhancement technology to solve the above problems. In order to improve the performance and effect of text classification, text enhancement becomes a necessary means when the sample size is small. Text enhancement is to increase the diversity and quantity of training data by performing various transformations and expansions on the original text data, and improve the generalization ability of the model.

[0003] The two mainstream methods are rewriting-based text enhancement and generation-based text enhancement. Rewriting-based text enhancement usually does not change the sentence much, and the generated text is close to the original text in meaning and form. Wei J et al. (Wei J, Zou K. Eda: Easy data augmentation techniques for boosting performance on text classification tasks [J]. arXi v preprint arXiv: 1901.11196, 2019.) proposed EDA (Easy Data Augmentation), which is a widely used text enhancement method. It contains four basic operations: synonym replacement, random insertion, random exchange, and random deletion. The synonym replacement uses WordNet to replace the original word with its synonym. Thakur et al. (Thakur N, Reimers N, Daxenberger J, et al. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks [J]. arXiv preprint arXiv: 2010.08240, 2020.) proposed a simple and efficient text augmentation strategy called Augmented SBERT, which uses a cross encoder to mark input sentence pairs to increase the training data of the dual encoder. Zhang et al. (Zhang L, Yang Z, Yang D. TreeMix: Composition alconstituency-based data augmentation for natural language understanding [J]. arXiv preprint arXiv: 2205.06153, 2022.) proposed a synthetic data augmentation method TreeMix for natural language understanding, which uses constituency parse trees to decompose sentences into constituent substructures and uses Mix up technology to recombine them to generate new sentences. Generation-based methods usually make greater changes to sentences and generate better sentence diversity.Anaby-Tavor et al. (Anaby-Tavor A, Carmeli B, Goldbraich E, et al. Don’t have enough data? Deep learning to the rescue! [C] / / Proceedings of the AAAIconference on artificial intelligence. 2020, 34(05): 7383-7390.) proposed a text enhancement method called LAMBDA, which is based on GPT-2 fine-tuning on the training set and generates new samples based only on label prompts, which makes the generated enhanced text have great diversity, but the data quality cannot be guaranteed. Chen et al. (Chen Y, Liu Y. Rethinking data augmentation in text-to-text paradigm [C] / / Proceedings of the 29th international conference on computation allinguistics. 2022: 1157-1162.) proposed a two-stage enhancement framework built by integrating T5 and BART. One stage is the fine-tuning stage, in which the pre-trained language model can adapt well to downstream classification, and the other stage is the generation stage, in which the fine-tuned model is used to generate new samples to improve performance. Dai et al. (Dai H, Liu Z, Liao W, et al. Chataug: Leveraging chatgpt for text data augmentation [J]. arXiv preprint arXiv: 2302.13007, 2023, 1 (2).) proposed a text enhancement method AugGPT based on CharGPT, which rewrites each sentence in the training sample into multiple samples with similar concepts but different semantics. However, most of the above methods are either too conservative, making only minor modifications to the original text and limiting the diversity of samples; or too radical, allowing the model to generate completely new samples, which has a greater risk of generating bad samples.

[0004] In view of the above problems, the present invention proposes a text enhancement method based on dual diversity generation DDG (DualDiversity Generation). This method uses a draft-driven generation model to balance the two extremes. The draft is constructed by keyword extraction, which retains the core semantics. At the same time, the diversity of keywords is considered when extracting keywords, which solves the semantic duplication problem of the extracted keywords. First, the word is converted into a word vector through a word embedding model. However, most high-frequency words are unimportant in the text, so the word embedding and sentence embedding are calculated by smooth inverse frequency weighting, and then the maximum boundary correlation between the word embedding and the sentence embedding is calculated to sort and select keywords. The maximum boundary correlation combines the correlation between words and sentences and the concept of diversity between words in a controllable way, solves the problem of keyword semantic duplication, effectively improves the quality of text generated by the draft-driven generation model, and lays a solid data foundation for subsequent text classification tasks.

[0005] In summary, the text enhancement method based on dual diversity generation is an innovative research issue with important research significance. Summary of the invention

[0006] In text enhancement, most traditional methods only make minor modifications to the original text, which limits the diversity of enhanced samples. In response to this, the present invention proposes a text enhancement method based on a dual diversity draft, which solves the above problems through dual diversity, namely keyword diversity and sample diversity. First, a draft is constructed by extracting keywords, and then input into the text enhancement model to generate new samples for text enhancement. When extracting keywords, keyword diversity is considered to solve the semantic duplication problem of the extracted keywords; when generating new samples, sample diversity is considered to improve the sample quality of text enhancement. The effect after text enhancement is evaluated by the text classification effect. The experimental results show that the method proposed in the present invention has achieved significant results in solving the problem of lack of diversity in enhanced samples.

[0007] The text enhancement method based on dual diversity generation specifically includes:

[0008] (1) Keyword extraction module

[0009] First, the text in the text dataset is segmented, and then the segmented results are vectorized through the pre-trained model to obtain word vectors. The smoothed inverse frequency weight of each word is calculated, and then the obtained word vectors are weighted averaged to obtain word embedding and sentence embedding. The maximum boundary correlation between word embedding and sentence embedding is calculated and sorted, and the Top K keywords are selected as the final extracted keywords based on the sorting results.

[0010] (2) New sample generation module

[0011] The keywords extracted by the keyword extraction method are mapped back to the original text, and the remaining parts are replaced with MASKtoken. At the same time, the label is added as a prompt before the text, so that the draft is constructed. The prompt is equivalent to a topic prompt, which makes the generated samples closer to the set label, providing better data quality for subsequent text classification. Then the draft is input into the text enhancement model GENIUS to generate new samples, and finally the generated new samples are added to the original dataset to obtain the text enhancement dataset.

[0012] The specific process of the method includes the following steps:

[0013] S1 segments the text into words, and then uses the ELMo model to represent the word vectors of the results.

[0014] S2 performs smoothed inverse frequency (SIF) weighting on word vector representations, which can better capture the central topic of the text and better filter common words.

[0015] After smoothing the inverse frequency weighting, S3 obtains the final extracted keywords by calculating and sorting the similarity of word embeddings and sentence embeddings. Considering that the extracted keywords are too similar in semantics, which will affect the samples generated by the text enhancement model to be too simple, and will also cause the truly important keywords to be ignored because the keywords with similar semantics are ranked higher, the maximum margin relevance (MMR) algorithm is used to calculate the similarity. After the similarity calculation, the MMR values ​​need to be sorted from large to small to select the Top K keywords.

[0016] After keyword extraction, S4 needs to construct a draft to input into the text enhancement model to generate new samples. The keywords extracted by the keyword extraction method are mapped back to the original text, and the remaining parts are replaced with MASK tokens. At the same time, the label is added as a prompt before the text, so that the draft is constructed. The prompt is equivalent to a topic prompt, which makes the generated samples closer to the set label, providing better data quality for subsequent text classification.

[0017] S5 inputs the draft into the text enhancement model GENIUS to generate new samples, and finally adds the generated new samples to the original dataset to obtain the text enhancement dataset. The new samples generated by the proposed method not only retain the core semantics of the samples by constructing the draft through keyword extraction, but also make the samples more diverse through dual diversity. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Algorithm structure diagram of the text enhancement method based on dual diversity generation

[0019] Figure 2 Comparison of experimental results for complaint and report dataset

[0020] Figure 3 Comparative experimental results for the Chinese text classification dataset Thucnews

[0021] Figure 4 1. Setup for Ablation Experiments

[0022] Figure 5 Ablation test results for each module DETAILED DESCRIPTION

[0023] In order to make the technical solution of the present invention clearer, the specific implementation methods in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0024] The present invention proposes a text enhancement method based on dual diversity generation. The specific algorithm structure diagram is as follows: Figure 1 As shown. For the text dataset where x i is the i-th text in the text dataset, y i is the i-th label in the text dataset, and m represents the total number of text datasets. Expanded to text augmentation dataset Where m′≥m, m′ represents the total number of datasets after text enhancement.

[0025] The text enhancement method based on dual diversity generation specifically includes:

[0026] (1) Keyword extraction module

[0027] First, the text dataset The text in x i Perform word segmentation and obtain where w j For x i The result after word segmentation, k is the number of words after word segmentation. Then the result after word segmentation is vectorized through the pre-trained model to obtain the word vector The word embedding is obtained by calculating the smoothed inverse frequency weight of each word and then taking the weighted average of the obtained word vectors. and sentence embeddings s . Calculate the maximum boundary correlation between word embedding and sentence embedding and sort them. Select the Top K keywords as the final extracted keywords based on the sorting results.

[0028] (2) New sample generation module

[0029] Map the keywords extracted by the keyword extraction method back to the original text, and replace the remaining parts with MASKtoken. At the same time, add the label as a prompt before the text, so that the draft is constructed. The prompt is equivalent to a topic prompt, which makes the generated sample closer to the set label, providing better data quality for subsequent text classification. Then input the draft into the text enhancement model GENIUS to generate new samples n is the text enhancement multiple, n is 16 in the complaint and report dataset, and n is 4 in the Chinese text classification dataset Thucnews. Finally, the generated new samples are added to the original dataset to obtain the text enhancement dataset.

[0030] The specific process of the method includes the following steps:

[0031] S1 performs word segmentation on the text, and then uses the ELMo model to represent the word vectors after the word segmentation. Unlike traditional word vectors that use fixed vectors, ELMo is a dynamic word vector representation method that dynamically adjusts the word vector according to the context. The word vector obtained by ELMo has three layers. The bottom character encoding layer Char-CNN is used to generate static word vectors. The first and second layers are both bidirectional LSTM. The word vectors encoded by the first layer of bidirectional LSTM contain more grammatical information, and the word vectors encoded by the second layer of bidirectional LSTM contain more context-related semantic information. The word vectors of these three layers can be weighted and then input into different downstream tasks. The whole process allows the model to dynamically adjust the word vector representation according to the use of each word in different contexts, so as to better capture the semantic information and context of the words.

[0032] S2 Since most high-frequency words in a text are irrelevant to the topic of the text, the word vector is smoothed and weighted to better capture the central topic of the text and better filter common words. The smoothed inverse frequency weight of each word is calculated according to the word frequency, and then the word vector is weighted to obtain the word embedding v w ′ and sentence embedding v s The Smoothed Inverse Frequency (SIF) weights are calculated as follows:

[0033]

[0034] Among them, w is a word in the sentence, f w is the statistical probability of a word appearing in a large corpus, that is, the probability that the frequency of the word accounts for the total word frequency. The hyperparameter b is 2.7×10 -4 .

[0035] Word Embeddingsv w ′ and sentence embedding vs The calculation is as follows:

[0036] v w ′=SIF(w)v w (2)

[0037]

[0038] Among them, v w is the word vector of each word obtained by the ELMo model, and |s| represents the number of words in the entire text.

[0039] After smoothing the inverse frequency weighting, S3 obtains the final extracted keywords by calculating and sorting the similarity of word embeddings and sentence embeddings. Considering that the extracted keywords are too similar in semantics, which will affect the samples generated by the text enhancement model to be too single, and will also cause the truly important keywords to be ignored because the keywords with similar semantics are ranked higher, the maximum margin relevance (MMR) algorithm is used to calculate the similarity. The MMR algorithm was originally designed for the field of information retrieval. By calculating the similarity between the text and the searched document, the documents are sorted to help users find relevant and diverse documents in the search results. The MMR algorithm is introduced into the keyword extraction method, thereby introducing keyword diversity. By calculating the MMR value between word embedding and sentence embedding, the Top K keywords are selected as the final extracted keywords. The extracted keywords combine the relevance between words and sentences and the diversity between words, thereby solving the problem of semantic repetition. The MMR calculation method is as follows:

[0040]

[0041] where w i and w j denote words i and j respectively, W denotes a word set, and denote the word vectors of word i and j respectively, and η is the maximum marginal correlation coefficient, where η is 0.5. When η=1, MMR calculates the standard correlation, and when η=0, MMR calculates the maximum diversity. The value of is between 0 and 1. The closer it is to 1, the more relevant the word is to the text topic. On the contrary, the closer the value is to 0, the less relevant the word is to the topic. The value of is also between 0 and 1. The larger the value, the more similar the semantics of the two words are; the smaller the value, the less related the two words are.

[0042] After similarity calculation, the MMR values ​​need to be sorted from large to small to select the Top K keywords. K is calculated as follows:

[0043] K=μ·|s| (7)

[0044] Where μ represents the ratio of the expected number of keywords to the number of words after text segmentation, and here μ is 0.7.

[0045] After keyword extraction, S4 needs to construct a draft to input into the text enhancement model to generate new samples. The keywords extracted by the keyword extraction method are mapped back to the original text, and the remaining parts are replaced with MASK tokens. At the same time, the label is added as a prompt before the text, so that the draft is constructed. The prompt is equivalent to a topic prompt, which makes the generated samples closer to the set label, providing better data quality for subsequent text classification.

[0046] S5 inputs the draft into the text enhancement model GENIUS to generate new samples, and finally adds the generated new samples to the original dataset to obtain the text enhancement dataset. Using the draft-driven generation model GENIUS as a text enhancement model introduces sample diversity. The GENIUS model is a conditional text generation pre-training model. It is initialized based on the BART

[15] model and can complete the text according to the given draft to generate a complete and fluent text, such as keywords, phrases, fragments, and other information. This is similar to when humans write, they first draft a framework and then complete the entire article. Unlike BERT, BART, and T5 series, the mask ratio of GENIUS in the pre-training stage is as high as 80%, while BERT, BART, and T5 series can only mask a small part. GENIUS masks unimportant parts, while BERT, BART, and T5 series all use random masking. Therefore, applying the GENIUS model to the field of text enhancement can generate more diverse text. The new samples generated by the proposed method not only retain the core semantics of the samples by constructing drafts through keyword extraction, but also make the samples have a higher degree of diversity through double diversity.

[0047] The present invention uses two data sets to verify the effect of the proposed method, one is the complaint and report data set, and the other is the Chinese text classification data set Thucnews. The complaint and report data set contains 96,589 complaint and report texts and 844 label categories. The Thucnews data set is generated by filtering and screening the historical data of Sina News RSS subscription channel from 2005 to 2011, from which 1,000 samples are extracted for text enhancement. The text length is between 20 and 30, and there are 10 categories in total.

[0048] In the experiment, the complaint and report dataset is divided into a head dataset and a tail dataset. The head dataset contains 93,151 complaint and report texts, and the tail dataset contains 3,438 complaint and report texts. The label categories of the head dataset account for 32% of the total categories, and the label categories of the tail dataset account for 68% of the total categories. First, text enhancement is performed on the tail dataset, and then the enhanced data is merged with the original dataset to perform the text classification task to verify the text enhancement effect.

[0049] In the text classification task, both datasets are divided into training set, validation set and test set in the ratio of 8:1:1.

[0050] like Figure 2 As shown in the figure, in order to verify the effectiveness of DDG compared with other text enhancement methods, the text classification effects of different text enhancement methods on the complaint and report dataset after enhancement are compared. The keyword ratio is set to 0.7 and the text enhancement multiple is 16. It can be seen that DDG has achieved the best performance in the three text classification models. Compared with GENIUS, the Micro-F1 value has increased by 0.51%, 1.54%, and 3.07%, respectively, and the Macro-F1 value has increased by 2.89%, 3.79%, and 3%, respectively. EDA, BERT, C-BERT, and ChineseBERT do not make much changes to the original text when performing text enhancement. Therefore, the generated new samples lack diversity and are very similar to the original text. For text classification, the model may not be able to learn enough different patterns or features, which will cause the model to perform poorly on unseen data and have limited generalization capabilities. The GENIUS model can balance diversity and core semantics well, but when it is applied to the text enhancement task for keyword extraction, the traditional statistical method TF-IDF is used to calculate the importance of a word based on statistical information. These statistical information are difficult to reflect the complex relationship between words in the text. There is also a problem of semantic duplication in the extracted keywords. The semantic similarity of keywords will affect the samples generated by the text enhancement model to be too single, and the truly important keywords will be ignored because the keywords with similar semantics are ranked higher. The proposed method can solve the above problems well, and thus can achieve a large performance gain.

[0051] like Figure 3As shown in the figure, in order to verify the performance of DDG on general datasets, the Chinese text classification dataset Thucnews was selected for text enhancement, and the performance of the proposed text enhancement method was verified by the text classification effect. The keyword ratio was set to 0.7 and the text enhancement factor was set to 4. It can be seen that DDG achieved the best effect on the Thucnews dataset. Compared with GENIUS, the Micro-F1 values ​​increased by 0.42%, 0.83%, and 3.32%, respectively, and the Macro-F1 values ​​increased by 0.35%, 0.67%, and 3.36%, respectively. In the classification results of the BiLSTM model, the Micro-F1 and Macro-F1 values ​​of EDA, BERT, C-BERT, and ChineseBERT are very low. This is because the Thucnews dataset is mostly short texts, and BiLSTM is better at processing long text sequences. The EDA, BERT, C-BERT, and ChineseBERT methods only made small modifications to the original text, and the text length did not change much. Therefore, these methods cannot train the BiLSTM model well when the training samples are limited. The GENIUS and DDG methods can generate diverse long texts and can still train the BiLSTM model well when the training samples are limited.

[0052] like Figure 4 and Figure 5 As shown in the figure, in order to verify the effectiveness of smooth inverse frequency weighted SIF and maximum boundary correlation MMR in DDG in text enhancement, ablation experiments were conducted on SIF and MMR on the complaint and report dataset with GENIUS as the benchmark. E1 represents the experiment on the benchmark model GENIUS, extracting keywords according to the TF-IDF value; E2 represents the use of the GENIUS model and SIF, smoothing the inverse frequency weighting of the word vector, and using the cosine distance to calculate the similarity, sorting and selecting keywords; E3 is the method proposed in the present invention, using GENIUS, SIF and MMR, and the difference from E2 is the similarity calculation method. This experiment uses the MMR algorithm to calculate the similarity. By comparing E1 and E2, it can be seen that by smoothing the inverse frequency weighting of the word vector to integrate additional features into external knowledge, the classification performance has been improved; by comparing E2 and E3, it can be seen that the superiority of the MMR algorithm compared to the cosine distance in the similarity calculation method further improves the Micro-F1 value and Macro-F1 value. In summary, each component of the method proposed in the present invention has shown a large performance gain.

[0053] This paper studies the problem of lack of diversity in enhanced samples and proposes a text enhancement method based on dual diversity drafts. New samples are generated based on keyword diversity and sample diversity, which not only retains the core semantics of the text, but also makes the samples more diverse. The quality of the enhanced samples is greatly improved, laying a good data foundation for downstream tasks. By using the proposed method DDG, the text classification effect has been significantly improved on the complaint and report dataset and the Chinese text classification dataset Thucnews. The experimental results show the feasibility of using DDG to solve the problem of lack of diversity in enhanced samples.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A text enhancement method based on dual diversity generation, characterized by: For text datasets where x i is the i-th text in the text dataset, y i is the i-th label in the text dataset, and m represents the total number of text datasets; Through text enhancement, the text dataset Expanded to text augmentation dataset Where m′≥m, m′ represents the total number of datasets after text enhancement; The method specifically comprises the following steps: (1) Keyword extraction module First, the text dataset The text in x i Perform word segmentation and obtain where w j For x i The result after word segmentation, k is the number of words after word segmentation; the result after word segmentation is then vectorized through the pre-training model to obtain the word vector The word embedding is obtained by calculating the smoothed inverse frequency weight of each word and then taking the weighted average of the obtained word vectors. and sentence embeddings s ; Calculate the maximum boundary correlation between word embedding and sentence embedding and sort them, and select the Top K keywords as the final extracted keywords based on the sorting results (2) New sample generation module The keywords extracted by the keyword extraction method are mapped back to the original text, and the remaining parts are replaced with MASKtoken. At the same time, the label is added as a prompt before the text, so that the draft is constructed. The prompt is equivalent to a topic prompt, which makes the generated sample closer to the set label and provides better data quality for subsequent text classification. The draft is then input into the text enhancement model GENIUS to generate new samples. n is the text enhancement multiple. In the complaint and report dataset, n is 16, and in the Chinese text classification dataset Thucnews, n is 4. Finally, the generated new samples are added to the original dataset to obtain the text enhancement dataset. The specific process of the method includes the following steps: S1 performs word segmentation on the text, and then uses the ELMo model to represent the word vectors of the segmented text. S2 calculates the smoothed inverse frequency weight of each word according to the word frequency, and then weights the obtained word vector to obtain the word embedding v w ′ and sentence embedding v s ; The smoothed inverse frequency weight is calculated as follows: Among them, w is a word in the sentence, f w is the statistical probability of a word appearing in a large corpus, that is, the probability that the frequency of the word accounts for the total word frequency. The hyperparameter b is 2.7×10 -4 ; Word Embeddingsv w ′ and sentence embedding v s The calculation is as follows: v w ′=SIF(w)v w (2) Among them, v w is the word vector of each word obtained by the ELMo model, and |s| represents the number of words in the entire text; S3 selects the top K keywords as the final extracted keywords by calculating the MMR value between word embedding and sentence embedding; the MMR calculation method is as follows: where w i and w j denote words i and j respectively, W denotes a word set, and Represent the word vectors of words i and j respectively, η is the maximum boundary correlation coefficient, here η is 0.5; After similarity calculation, the MMR values ​​need to be sorted from large to small to select the Top K keywords. K is calculated as follows: K=μ·|s| (7) Where μ represents the ratio of the expected number of keywords to the number of words after text segmentation, where μ is 0.7; After keyword extraction, S4 needs to construct a draft and input it into the text enhancement model to generate new samples; map the keywords extracted by the keyword extraction method back to the position of the original text, and replace the remaining part with MASK token; at the same time, add the label as a prompt before the text; S5 inputs the draft into the text enhancement model GENIUS to generate new samples, and finally adds the generated new samples to the original dataset to obtain the text enhancement dataset.

Images