A word-based short text summarization method

Abstract

This invention belongs to the field of text summarization extraction technology in NLP, and discloses a word-based short text summarization extraction method, including an extraction model and a word order model. The word order model steps are as follows: Step 1: Data labeling; S1.1: Select 500,000 short dialogue texts from a telemarketing scenario, and manually review and correct the word order of each sentence; S1.2: Use jieba word segmentation, with the original word order labeled as 1, then enumerate all word combinations for each sentence, labeled as 0, and finally manually review and correct all label data; Step 2: Data preprocessing. This invention uses a dilated convolutional neural network (DCNN) in the extraction model network structure, with the dilation coefficient set in the order of 1, 2, 4, 1, enabling the model to capture as much text information as possible, and adds a word order model, improving the effectiveness and continuity of the summary, thus perfectly solving the problem of incoherence in extraction-based summarization.

Description

Technical Field

[0001] This invention belongs to the field of text summarization extraction technology in NLP, specifically a word-based short text summarization extraction method. Background Technology

[0002] The goal of text summarization is to compress, summarize, and conclude long texts to form short texts with a concise and comprehensive meaning. Depending on the number of documents, text summarization tasks can be divided into single-document summarization and multi-document summarization. Based on the summarization method, text summarization can be broadly categorized into extractive and abstractive methods. The former directly extracts text from the original document to form a summary, while the latter generates a summary word by word. Comparatively, extractive methods, due to their inherent characteristics, sometimes fail to concisely and coherently summarize the content of the original text. Summary of the Invention

[0003] The purpose of this invention is to provide a word-based short text summarization method to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a word-based short text summarization method, comprising an extraction model and a word order model, wherein the word order model steps are as follows:

[0005] Step 1: Data Labeling 1;

[0006] S1.1: Select 500,000 short dialogue texts from a telemarketing scenario and manually review and correct the word order of each sentence;

[0007] S1.2: Use jieba word segmentation. The original word order is labeled as 1. Then, for each sentence, all word combinations are enumerated and labeled as 0. Finally, all label data is manually reviewed and corrected.

[0008] Step 2: Data Preprocessing 1;

[0009] Step 3: Model Building 1, reducing the dimensionality of high-dimensional data to 2 dimensions;

[0010] Step 4: Model Evaluation 1;

[0011] S4.1: During training, after each 100 batches of training data are completed, a validation set is run once, and the F1 score and loss are calculated for the predicted and true labels of the validation set.

[0012] S4.2: When the F1-score does not improve after 10 batches, the model training is terminated early. At this time, the last version of the model is considered to be the optimal model. The optimal model is used to run the test set data and calculate the F1-score. The current F1-score value is the score of the model.

[0013] The model extraction steps are as follows:

[0014] Step 1: Data Annotation 2. Select 200,000 short dialogue texts from the telemarketing scenario and manually summarize concise summary texts based on the original content.

[0015] Step 2: Data Preprocessing II;

[0016] Step 3: Model Construction 2, using a dilated convolutional neural network with dilation coefficients set in the order of 1, 2, 4, 1 to capture data information;

[0017] Step 4: Model Evaluation II;

[0018] S4.1: Set a threshold to determine whether the current word is part of the extracted summary content. If the score is greater than the threshold, the word is considered to have been extracted; otherwise, it is discarded.

[0019] S4.2: Enumerate all combinations of words that are greater than the threshold and input them into the word order model. Then, take the word sequence with the highest score and concatenate them in order as the summary.

[0020] S4.3: Calculate the rogue score for the abstract and the original text, and use the average of all rogue scores to evaluate the quality of the current model. The closer the rogue score is to 1, the better.

[0021] Preferably, the specific steps of the data preprocessing are as follows:

[0022] Step 1: Construct a dictionary by using all the deduplicated words from the training set as the dictionary;

[0023] Step 2: Unify the text dimension. Use the <PAD> symbol to pad all the text to a uniform max_length dimension. Here, we choose the number of words with the most words in the training set as the max_length.

[0024] Step 3: Represent the training data as an index according to the dictionary, loop through each sentence of text, check if it exists in the dictionary, if it exists, get the index in the dictionary, otherwise get the index of <UNK>, where <UNK> represents all out-of-vocabulary words;

[0025] Step 4: Generate a data iterator, divide the data into multiple batches and feed them into the model for training. Each batch contains 64 data points. During the process of generating the iterator, the data is also added to the GPU.

[0026] Preferably, the model constructs a network structure as follows:

[0027] First layer: Embedding

[0028] Second layer: BILSTM()

[0029] Third layer: Linear();

[0030] Embedding is the vector representation layer, the second layer is a bidirectional long short-term memory neural network (BILSTM), and Linear() is the linear layer.

[0031] Preferably, the specific steps of the second data preprocessing are as follows:

[0032] Step 1: Find the word index of the manual summary in the original text. First, perform jieba word segmentation on the summary and the original text respectively. Then, find the center position of the summary in the original text according to the sliding window method. Match the summary from the center position to both sides. If the match is successful, record it as 1, otherwise record it as 0. Finally, the summary tag corresponding to each sentence of the original text will be generated.

[0033] Step 2: Construct the dictionary and embedding. Select the first 200,000 words from the Tencent word vector dictionary and the words in the training set after deduplication from the Tencent word vector dictionary to form the dictionary. For words in the training set, if they appear in the Tencent word vector dictionary, select the word vector from it; otherwise, randomly initialize them as 200-dimensional word vectors.

[0034] Step 3: Unify the text dimension. After word segmentation, use the <PAD> symbol to pad all the text to a unified max_length dimension. Here, the number of words with the most words in the training set is selected as max_length.

[0035] Step 4: Generate data iterators. For deep learning, the data needs to be divided into multiple batches and fed into the model for training. Each batch contains 32 data points. During the generation of iterators, the data is simultaneously added to the GPU.

[0036] Preferably, the model constructs a network structure as follows:

[0037] First layer: Embedding

[0038] Second layer: Linear()

[0039] Third layer: Dropout(0.5)

[0040] Fourth layer: DCNN (dilation_rate=1)

[0041] Fifth layer: Dropout(0.5)

[0042] Sixth layer: DCNN (dilation_rate=2)

[0043] Seventh layer: Dropout(0.5)

[0044] Eighth layer: DCNN (dilation_rate=4)

[0045] Ninth layer: Dropout(0.5)

[0046] Tenth layer: DCNN (dilation_rate=1)

[0047] Eleventh layer: Dropout(0.5)

[0048] Twelfth layer: Linear()

[0049] Thirteenth layer: Sigmoid();

[0050] Embedding is the vector representation layer; Linear() is the fully connected layer; Dropout is the deactivation layer. Dropout randomly selects 50% of the network nodes to deactivate. Each layer of DCNN is followed by a deactivation layer; DCNN is a dilated convolutional neural network, and dilation_rate is the dilation rate.

[0051] The beneficial effects of this invention are as follows:

[0052] This invention uses a dilated convolutional neural network (DCNN) in the extraction model network structure, with the dilation coefficients set in the order of 1, 2, 4, 1, enabling the model to capture as much text information as possible. It also adds a word order model, which improves the effectiveness and continuity of the summary, thus perfectly solving the problem of incoherence in extraction summarization. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the network structure of the structural word order model of this invention;

[0054] Figure 2 This is a schematic diagram of the network structure of the structure extraction model of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] like Figures 1 to 2 As shown, this embodiment of the invention provides a word-based short text summarization method, including an extraction model and a word order model. The steps of the word order model are as follows:

[0057] Step 1: Data Labeling 1;

[0058] S1.1: Select 500,000 short dialogue texts from a telemarketing scenario and manually review and correct the word order of each sentence;

[0059] S1.2: Use jieba word segmentation. The original word order is labeled as 1. Then, for each sentence, all word combinations are enumerated and labeled as 0. Finally, all label data is manually reviewed and corrected.

[0060] Step 2: Data Preprocessing 1;

[0061] Step 3: Model Building 1, reducing the dimensionality of high-dimensional data to 2 dimensions;

[0062] Step 4: Model Evaluation 1;

[0063] S4.1: During training, after each 100 batches of training data are completed, a validation set is run once, and the F1 score and loss are calculated for the predicted and true labels of the validation set.

[0064] S4.2: When the F1-score does not improve after 10 batches, the model training is terminated early. At this time, the last version of the model is considered to be the optimal model. The optimal model is used to run the test set data and calculate the F1-score. The current F1-score value is the score of the model.

[0065] The steps for extracting the model are as follows:

[0066] Step 1: Data Annotation 2. Select 200,000 short dialogue texts from the telemarketing scenario and manually summarize concise summary texts based on the original content.

[0067] Step 2: Data Preprocessing II;

[0068] Step 3: Model Construction 2, using a dilated convolutional neural network with dilation coefficients set in the order of 1, 2, 4, 1 to capture data information;

[0069] Step 4: Model Evaluation II;

[0070] S4.1: Set a threshold to determine whether the current word is part of the extracted summary content. If the score is greater than the threshold, the word is considered to have been extracted; otherwise, it is discarded.

[0071] S4.2: Enumerate all combinations of words that are greater than the threshold and input them into the word order model. Then, take the word sequence with the highest score and concatenate them in order as the summary.

[0072] S4.3: Calculate the Rogue score for the abstract and the original text, and use the average of all Rogue scores to evaluate the quality of the current model. The closer the Rogue score is to 1, the better.

[0073] The annotation results of Data Annotation 2 are two columns of data, one column of text, and one column of annotation summary (as shown in the table below);

[0074]

[0075] Furthermore, by using a Dilated Convolution Neural Network (DCNN) with different dilation rates in the extracted model's network structure, the model can capture as much textual information as possible. The word order model is a Bidirectional Long Short-Term Memory (LSTM) model, whose main function is to score the sequences entering the model. The higher the score, the better the word order, and the more coherent the sentence. Finally, the highest-scoring word combination is selected as the summary of the current text, thus perfectly solving the problem of incoherence in extractive summarization.

[0076] The specific steps of data preprocessing are as follows:

[0077] Step 1: Construct a dictionary by using all the deduplicated words from the training set as the dictionary;

[0078] Step 2: Unify the text dimension. Use the <PAD> symbol to pad all the text to a uniform max_length dimension. Here, we choose the number of words with the most words in the training set as the max_length.

[0079] Step 3: Represent the training data as an index according to the dictionary, loop through each sentence of text, check if it exists in the dictionary, if it exists, get the index in the dictionary, otherwise get the index of <UNK>, where <UNK> represents all out-of-vocabulary words;

[0080] Step 4: Generate a data iterator, divide the data into multiple batches and feed them into the model for training. Each batch contains 64 data points. During the process of generating the iterator, the data is also added to the GPU.

[0081] By adding data to the GPU, an acceleration effect is achieved.

[0082] The network structure for model construction is as follows:

[0083] First layer: Embedding

[0084] Second layer: BILSTM()

[0085] Third layer: Linear();

[0086] Embedding is the vector representation layer, the second layer is a bidirectional long short-term memory neural network (BILSTM), and Linear() is the linear layer.

[0087] Through network structure (see Figure 1 In this model, Embedding is the vector representation layer; the second layer is a bidirectional long short-term memory neural network (BILSTM), which can capture the dependencies between words over a long distance and reduce the dimensionality of high-dimensional data to 2-dimensionality through Linear().

[0088] The specific steps of data preprocessing are as follows:

[0089] Step 1: Find the word index of the manual summary in the original text. First, perform jieba word segmentation on the summary and the original text respectively. Then, find the center position of the summary in the original text according to the sliding window method. Match the summary from the center position to both sides. If the match is successful, record it as 1, otherwise record it as 0. Finally, the summary tag corresponding to each sentence of the original text will be generated.

[0090] Step 2: Construct the dictionary and embedding. Select the first 200,000 words from the Tencent word vector dictionary and the words in the training set after deduplication from the Tencent word vector dictionary to form the dictionary. For words in the training set, if they appear in the Tencent word vector dictionary, select the word vector from it; otherwise, randomly initialize them as 200-dimensional word vectors.

[0091] Step 3: Unify the text dimension. After word segmentation, use the <PAD> symbol to pad all the text to a unified max_length dimension. Here, the number of words with the most words in the training set is selected as max_length.

[0092] Step 4: Generate data iterators. For deep learning, the data needs to be divided into multiple batches and fed into the model for training. Each batch contains 32 data points. During the generation of iterators, the data is simultaneously added to the GPU.

[0093] The purpose of using center position matching is to make the summary index in the matched original text more compact, thereby increasing the model recognition rate. In order to unify the dimension with Tencent word vectors, since words in the training set do not appear in Tencent word vector dictionary, they are randomly initialized to 200-dimensional word vectors. By selecting the maximum number of words as max_length, all data can be covered.

[0094] The network structure for model construction two is as follows:

[0095] First layer: Embedding

[0096] Second layer: Linear()

[0097] Third layer: Dropout(0.5)

[0098] Fourth layer: DCNN (dilation_rate=1)

[0099] Fifth layer: Dropout(0.5)

[0100] Sixth layer: DCNN (dilation_rate=2)

[0101] Seventh layer: Dropout(0.5)

[0102] Eighth layer: DCNN (dilation_rate=4)

[0103] Ninth layer: Dropout(0.5)

[0104] Tenth layer: DCNN (dilation_rate=1)

[0105] Eleventh layer: Dropout(0.5)

[0106] Twelfth layer: Linear()

[0107] Thirteenth layer: Sigmoid();

[0108] Embedding is the vector representation layer; Linear() is the fully connected layer; Dropout is the deactivation layer, where 50% of the network nodes are randomly selected to be deactivated, and each layer of DCNN is followed by a deactivation layer; DCNN is a dilated convolutional neural network, and dilation_rate is the dilation rate.

[0109] Through network structure (see Figure 2 In this model, Linear() is a fully connected layer used for dimensionality reduction. 50% of the network nodes are randomly selected to be deactivated through the deactivation layer. To prevent overfitting, a dilated convolutional neural network (DCNN) is chosen as the base model. Compared to a regular convolutional neural network, a DCNN has a wider field of view and can capture data information from a greater distance. The dilation_rate is set to 1, 2, and 4. Since the data is short text, setting the dilation_rate too high has little effect. The twelfth layer, Linear(), is used for final classification. Because this task involves multi-label classification, the sigmoid() function is used for scoring.

[0110] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0111] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A word-based short text summarization method, comprising an extraction model and a word order model, characterized in that: The word order modeling steps are as follows: Step 1: Data Labeling 1; S1.1: Select 500,000 short dialogue texts from a telemarketing scenario and manually review and correct the word order of each sentence; S1.2: Use jieba word segmentation. The original word order is labeled as 1. Then, for each sentence, all word combinations are enumerated and labeled as 0. Finally, all label data is manually reviewed and corrected. Step 2: Data Preprocessing 1; Step 3: Model Building 1, reducing the dimensionality of high-dimensional data to 2 dimensions; Step 4: Model Evaluation 1; S4.1: During training, after each 100 batches of training data are processed, a validation set is run once, and the F1-score and loss are calculated for the predicted and true labels of the validation set. S4.2: When the F1-score does not improve after 10 batches, the model training is terminated early. At this time, the last version of the model is considered to be the optimal model. The optimal model is used to run the test set data and calculate the F1-score. The current F1-score value is the score of the model. The model extraction steps are as follows: Step 1: Data Annotation 2. Select 200,000 short dialogue texts from the telemarketing scenario and manually summarize concise summary texts based on the original content. Step 2: Data Preprocessing II; Step 3: Model Construction 2, using a dilated convolutional neural network with dilation coefficients set in the order of 1, 2, 4, 1 to capture data information; Step 4: Model Evaluation II; S4.1: Set a threshold to determine whether the current word is part of the extracted summary content. If the score is greater than the threshold, the word is considered to have been extracted; otherwise, it is discarded. S4.2: Enumerate all combinations of words that are greater than the threshold and input them into the word order model. Then, take the word sequence with the highest score and concatenate them in order as the summary. S4.3: Calculate the Rogue score for the abstract and the original text, and use the average of all Rogue scores to evaluate the quality of the current model. The closer the Rogue score is to 1, the better. The model constructs a network structure as follows: First layer: Embedding Second layer: BILSTM() Third layer: Linear(); Embedding is the vector representation layer, the second layer is a bidirectional long short-term memory neural network (BILSTM), and Linear() is the linear layer.

2. The word-based short text summarization method according to claim 1, characterized in that: The specific steps of the data preprocessing are as follows: Step 1: Construct a dictionary by using all the segmented and deduplicated words from the training set as the dictionary; Step 2: Unify text dimensions, using... <pad> The symbol pads all text to a uniform max_length dimension, where the number of words with the most occurrences in the training set is chosen as max_length.< / pad> Step 3: Represent the training data as an index based on the dictionary, iterate through each sentence of text, check if it exists in the dictionary, and if it does, retrieve the index from the dictionary; otherwise, retrieve the index from the dictionary. <unk>index, <unk> This refers to all out-of-vocabulary words;< / unk> < / unk> Step 4: Generate a data iterator, divide the data into multiple batches and feed them into the model for training. Each batch contains 64 data points. During the process of generating the iterator, the data is also added to the GPU.

3. The word-based short text summarization method according to claim 1, characterized in that: The specific steps of the second data preprocessing step are as follows: Step 1: Find the word index of the manual summary in the original text. First, perform jieba word segmentation on the summary and the original text respectively. Then, find the center position of the summary in the original text according to the sliding window method. Match the summary from the center position to both sides. If the match is successful, record it as 1, otherwise record it as 0. Finally, the summary tag corresponding to each sentence of the original text will be generated. Step 2: Construct the dictionary and embedding. Select the first 200,000 words from the Tencent word vector dictionary and the words in the training set after deduplication from the Tencent word vector dictionary to form the dictionary. For words in the training set, if they appear in the Tencent word vector dictionary, select the word vector from it; otherwise, randomly initialize them as 200-dimensional word vectors. Step 3: Unify text dimensions; after word segmentation, use... <pad> The symbol pads all text to a uniform max_length dimension, where the number of words with the most occurrences in the training set is chosen as max_length.< / pad> Step 4: Generate data iterators. For deep learning, the data needs to be divided into multiple batches and fed into the model for training. Each batch contains 32 data points. During the process of generating iterators, the data is also added to the GPU.

4. The word-based short text summarization method according to claim 1, characterized in that: The model constructs a network structure as follows: First layer: Embedding Second layer: Linear() Third layer: Dropout(0.5) Fourth layer: DCNN (dilation_rate=1) Fifth layer: Dropout(0.5) Sixth layer: DCNN (dilation_rate=2) Seventh layer: Dropout(0.5) Eighth layer: DCNN (dilation_rate=4) Ninth layer: Dropout(0.5) Tenth layer: DCNN (dilation_rate=1) Eleventh layer: Dropout(0.5) Twelfth layer: Linear() Thirteenth layer: Sigmoid(); Embedding is the vector representation layer; Linear() is the fully connected layer; Dropout is the deactivation layer, where 50% of the network nodes are randomly selected to be deactivated, and each layer of DCNN is followed by a deactivation layer; DCNN is a dilated convolutional neural network, and dilation_rate is the dilation rate.

Images