Summary source detection method based on text similarity and deep learning

Abstract

A kind of summary source detection method based on text similarity and deep learning, respectively extract the repeat rate feature based on N-gram of to-be-processed document and its summary, the feature based on the part-of-speech of Rouge score and the deep learning feature extracted by neural network, then the features extracted are merged to obtain the classification result using logistic regression classifier.The present application according to the characteristics of the two existing automatic abstract generation techniques, respectively with the method of text similarity and deep learning to extract corresponding features, accurately carry out summary source detection by using the machine learning model of these two features.

Description

Technical Field

[0001] This invention relates to a technique in the field of natural language processing, specifically a method for detecting the source of a summary based on text similarity and deep learning. Background Technology

[0002] The task of abstract source detection involves determining whether a given original text and its corresponding abstract were generated by a model or created manually. Currently, one approach to this problem is to treat it as a sequence classification problem, solving it by fine-tuning existing language models. Another approach utilizes existing techniques for detecting AI-generated content, such as probabilities of the content being generated within existing language models or the RoBERTa language model detector proposed by OpenAI. However, given the diverse range of automatic abstract generation techniques, none of these approaches have achieved satisfactory results in experiments for this task. Summary of the Invention

[0003] This invention addresses the problem that existing techniques that rely solely on deep learning methods to construct similarity features are ineffective for source detection in abstracts. It proposes a source detection method based on text similarity and deep learning. By leveraging the characteristics of the two main categories of existing automatic abstract generation techniques, this invention extracts relevant features using both text similarity and deep learning methods. Then, it uses machine learning models based on these two feature sets to accurately detect the source of the abstract.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a method for detecting the source of a summary based on text similarity and deep learning. The method extracts N-gram-based repetition rate features, part-of-speech-based Rouge score features, and deep learning features extracted by a neural network from the document to be processed and its summary. The extracted features are then merged and a logistic regression classifier is used to obtain the classification result.

[0006] This invention relates to a system for implementing the above method, comprising: a text similarity feature extraction module, a deep learning feature extraction module, and a machine learning module, wherein: the text similarity feature extraction module extracts features from the word repetition rate and word part-of-speech similarity of the document to be processed and its summary; the deep learning feature extraction module constructs features from the document to be processed and its summary using a built-in neural network; and the machine learning module predicts the classification result of the summary source based on the features extracted by the first two modules. Technical effect

[0007] This invention designs features based on text similarity and deep learning for two common automatic summarization methods, respectively. The complementary nature of these two features improves accuracy in summarization source detection. Furthermore, the invention's framework integrates several statistical measures as features, thus offering good interpretability. Additionally, the model involved in this invention has a small number of parameters, making it lightweight. Attached Figure Description

[0008] Figure 1 This is a flowchart of the present invention;

[0009] Figure 2 Flowchart for text similarity feature extraction;

[0010] Figure 3 This is a schematic diagram of a deep learning model network. Detailed Implementation

[0011] like Figure 1 As shown in the figure, this embodiment involves a method for detecting the source of a summary based on text similarity and deep learning, which includes the following steps:

[0012] Step 1) Extract the corresponding N-gram-based repetition rate features for the given document and summary, specifically as follows: Figure 2 As shown, after preprocessing the text information of the document and summary to remove punctuation and extra spaces, the text is segmented to obtain tokens. Specifically, the preprocessing of the text information of the document and summary to remove punctuation and extra spaces involves: first, removing stop words from the segmentation results; then, stemming is performed on each token in the text to ensure that different forms of the same token can be restored to the same stem; and finally, the set S consisting of n tokens (N-Gram) in document D and summary S is calculated respectively. D (n) = set(N-Gram(D,n)) and S S (n) = set(N-Gram(S, n)), which takes the proportion of the number of elements in the intersection of the two sets as the corresponding feature. Specifically: Where: N-Gram(·, n) in the formula represents taking the N-Gram tokens of the input text. For example, N-Gram(['I', 'am', 'happy'], 2) results in ['I', 'am'] and ['am', 'happy']. set(·) represents creating a set of the input content, ∩ represents taking the intersection of two sets, and |·| represents counting the number of elements in the input.

[0013] In the actual implementation, the 10 values ​​[Precision(1), Recall(1), ..., Precision(5), Recall(5)] are selected as the repetition rate features based on N-Gram.

[0014] Step 2) Calculate text similarity features based on Rouge scores for parts of speech.

[0015] The main intuition for this part is that some abstracts don't completely copy the words from the original text, but instead use synonyms. Since synonym substitution doesn't change the part of speech (PS) of the words before and after the substitution, the similarity between the two versions will be relatively high from a PS perspective. Therefore, this feature introduces the idea of ​​Part-of-Speech (POS).

[0016] Step 2 specifically includes:

[0017] a) Use the pos_tag tool in the nltk library to perform part-of-speech tagging on the preprocessed lexical units, and convert the document and summary into two sequences D' and S' composed of part-of-speech tags.

[0018] b) Calculate the Rouge-1, Rouge-2, and Rouge-L scores between the two sequences, specifically: Wherein: LCS is the length of the longest common subsequence between two strings, which uses, but is not limited to, the technique described by Lin, CY in "Rouge: A package for automatic evaluation of summaries" (In Text summarizationbranches out (pp. 74-81)).

[0019] In actual implementation, nine values—Recall, Precision, and F1—from Rouge-1, Rouge-2, and Rouge-L are selected as features for this part, and the Rouge library in Python is used for calculation.

[0020] Step 3) Construct as follows Figure 3 The neural network shown extracts deep learning features for document D and its corresponding summary S;

[0021] The neural network comprises: an Albert word segmenter, an Albert model, two Bi-LSTMs, and two fully connected layers. The Albert word segmenter segments the input text to obtain a word sequence. The Albert model obtains word vectors corresponding to each word based on the word sequence and, through several Transformer modules, obtains a hidden state sequence composed of vector representations of word elements containing semantic information of the context. The hidden state sequences of the document and corresponding summary output by the Albert model are input into the two Bi-LSTMs respectively, and corresponding vector representations of the document and summary are created. The first fully connected layer performs feature integration processing on the merged vector representations through linear operations to obtain a 50-dimensional vector. The second fully connected layer performs dimensionality reduction processing based on the output information of the previous fully connected layer after passing through the Tanh activation function, obtaining a binary classification result.

[0022] The corresponding vector representation refers to the Bi-LSTM layer processing the hidden states of the input sequence and outputting the last hidden state as the vector representation of the text.

[0023] The merged vector representation refers to: concat[d; s; abs(ds)], where: d and s are the corresponding vector representations of the document and the summary, respectively; concat[] means concatenating several vectors; and abs is the absolute value function.

[0024] In practical applications, the network needs to be trained on the training dataset using the cross-entropy loss function. During training, the weights of the open-source language model Albert are kept fixed, and only the weights of the Bi-LSTM and fully connected layers are updated. The learning rate is set to 1e-4, and the training is performed for 3 epochs. After training, for a document D and its corresponding summary S, the vector representation of the output after passing through the first fully connected layer and the Tanh activation function of the network can be taken as the corresponding deep learning feature.

[0025] Step 4) After concatenating the three vector features obtained in steps 1-3 into a 69-dimensional vector, this vector is used to train a logistic regression classifier with L2 regularization based on sklearn. In the online stage, the trained logistic regression classifier is used for real-time classification.

[0026] The training described refers to using LogisticRegression from sklearn, employing L2 regularization as a penalty, with the reciprocal of the regularization strength C = 100, the maximum number of iterations max_iter = 500, and other parameters using default values; the loss function used during training is... Where: y i For sample X iThe corresponding true 0 / 1 label, in reality, sample X i The 69-dimensional vector obtained after concatenating the features from steps 1-3 is used by the logistic regression model to predict the value of sample X. i The probability that its corresponding label is 1 w and w0 are the parameters of the logistic regression model.

[0027] Through practical experiments, the proposed method was successfully trained on a summary source detection dataset (based on the CNN / DailyMail public dataset and incorporating various automatic summary generation methods) to obtain the final model feature1+feature2. This dataset includes a training set (6400 entries), a validation set (800 entries), and a test set (800 entries). The results below show a comparison of the proposed method's performance with other methods on the test set. Specifically, the other methods used as baselines include:

[0028] The method based on the language model gpt2-xl: the calculation method of -logP is as follows In other words, the higher the probability that the language model considers the summary content to be present, the lower the corresponding value. Rank is the average ranking of the true next word within the probability distribution of the model's predicted next word; logrank is the result of taking the logarithm of the rank value. The entropy is calculated as follows: This refers to the average entropy used when predicting each term in the summary. After calculating the specific values, these methods set a threshold; results exceeding this threshold are considered to indicate that the summary was manually written. This threshold is selected based on the data from the training set.

[0029] openai-detector: A detector based on the RoBERTa language model proposed by OpenAI.

[0030] albert_ft treats the task as a sequence classification problem, organizing training data in the format of "[CLS] document [SEP] summary [SEP]", and obtaining the classification result using a fully connected layer based on the representation corresponding to [CLS]. This method first trains the model with the training data, and then performs inference on the test set.

[0031] Table 1 Comparison Results of Examples

[0032] As shown in Table 1, the method proposed in this invention can significantly improve accuracy compared to existing technologies. Furthermore, given that this method incorporates multiple features, ablation experiments were also conducted, and the results are shown in Table 2.

[0033] Table 2 Comparison results of ablation experiments

[0034] The results show that using either text similarity-based features (feature1) or deep learning-based features (feature2) alone is not as effective as combining the two features for classification, thus proving the effectiveness of feature fusion.

[0035] As shown in Table 3, this method effectively combines the advantages of both text similarity and deep learning features. Specifically, text similarity features excel at identifying machine-generated summaries based on word similarity to the original text, while deep learning features can identify human-written summaries at the semantic level. Through this fusion, this invention can help achieve more accurate source detection for summaries.

[0036] Table 3 Effects of the Examples

[0037] In summary, this invention achieves an accuracy of 93.0%, which is a significant improvement over existing methods. It also shows a significant improvement over the 86.2% accuracy achieved using text similarity features alone and the 83.8% accuracy achieved using deep learning features alone, demonstrating the effectiveness of merging the two feature sets.

[0038] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for detecting the source of summaries based on text similarity and deep learning, characterized in that, The N-gram-based repetition rate feature, the part-of-speech-based Rouge score feature, and the deep learning feature extracted by the neural network are extracted from the document to be processed and its summary respectively. The extracted features are then merged and a logistic regression classifier is used to obtain the classification result. The N-gram-based repetition rate feature is obtained as follows: After preprocessing the text information of the document and summary by removing punctuation and extra spaces, the text is segmented to obtain word units. Specifically, the preprocessing of the document and summary text information by removing punctuation and extra spaces involves: first removing stop words from the segmentation results, then stemming each word unit in the text to ensure that different forms of the same word unit can be restored to the same stem; and then calculating the repetition rate feature for each document. and summary The set consisting of n words and N-grams and The proportion of the number of elements in the intersection of the two sets to the total number of elements in each set is used as the corresponding feature, specifically: , , where: in the formula This indicates that the N-Gram words in the input text are extracted. This indicates that the input content is created as a set. This means taking the intersection of two sets. This represents the number of elements in the input. The features of the Rouge score based on part-of-speech tagging are obtained in the following way: a) Use the pos_tag tool from the nltk library to perform part-of-speech tagging on the preprocessed tokens, transforming the document and summary into two sequences composed of part-of-speech tags. ; b) Calculate the Rouge-1, Rouge-2, and Rouge-L scores between the two sequences; The deep learning features extracted by the neural network are obtained through the following method: constructing a neural network for a document... and corresponding summary Extracting deep learning features; The neural network comprises: an Albert word segmenter, an Albert model, two Bi-LSTMs, and two fully connected layers. The Albert word segmenter segments the input text to obtain a word sequence. The Albert model obtains word vectors corresponding to each word based on the word sequence and, through several Transformer modules, obtains a hidden state sequence composed of vector representations of word elements containing semantic information of the context. The hidden state sequences of the document and corresponding summary output by the Albert model are input into the two Bi-LSTMs respectively, and corresponding vector representations of the document and summary are created. The first fully connected layer performs feature integration processing on the merged vector representations through linear operations to obtain a 50-dimensional vector. The second fully connected layer performs dimensionality reduction processing based on the output information of the previous fully connected layer after passing through the Tanh activation function, obtaining a binary classification result.

2. The summary source detection method based on text similarity and deep learning according to claim 1, characterized in that, Select [ These 10 values ​​serve as N-Gram-based repetition rate features.

3. The summary source detection method based on text similarity and deep learning according to claim 1, characterized in that, simultaneously Nine values—Recall, Precision, and F1—from Rouge-1, Rouge-2, and Rouge-L were selected as features for this part, and the calculations were performed using the rouge library in Python.

4. The summary source detection method based on text similarity and deep learning according to claim 1, characterized in that, The corresponding vector representation refers to the Bi-LSTM layer processing the hidden states of the input sequence and outputting the last hidden state as the vector representation of the text. The merged vector representation refers to: ,in: and These are the corresponding vector representations of the document and the summary, respectively. concat[] means concatenating several vectors together, and abs is the absolute value function.

5. The summary source detection method based on text similarity and deep learning according to claim 1, characterized in that, First, the neural network is trained on the training dataset using the cross-entropy loss function. During training, the weights of the open-source language model Albert are kept fixed, and only the weights of the Bi-LSTM and fully connected layers are updated. The learning rate is set to 1e-4, and training is performed for 3 epochs. After training, the documents are... and corresponding summary Then, the vector representation of the output after passing through the first fully connected layer and the Tanh activation function of the network can be taken as the corresponding deep learning feature.

6. The summary source detection method based on text similarity and deep learning according to claim 1, characterized in that, The classification results are obtained by concatenating the N-gram-based repetition rate features, the part-of-speech-based Rouge score features, and the deep learning features extracted by the neural network into a 69-dimensional vector. This vector is then used to train a logistic regression classifier with L2 regularization based on sklearn. In the online stage, the trained logistic regression classifier is used for real-time classification. The training described refers to using LogisticRegression from sklearn, with L2 regularization as the penalty, the reciprocal of the regularization strength C=100, the maximum number of iterations max_iter=500, and the other parameters using default values; the loss function used during training is... ,in: For the sample The corresponding real 0 / 1 labels, in actual samples The 69-dimensional vector obtained after concatenating the features from steps 1-3 is used by the logistic regression model to predict the performance of samples. The probability that its corresponding label is 1 , These are the parameters of the logistic regression model.

7. A summary source detection system based on text similarity and deep learning that implements the method of any one of claims 1-6, characterized in that, include: The system comprises a text similarity feature extraction module, a deep learning feature extraction module, and a machine learning module. Specifically, the text similarity feature extraction module extracts features from the document to be processed and its summary in terms of word repetition rate and word part-of-speech similarity; the deep learning feature extraction module constructs features from the document to be processed and its summary using a built-in neural network; and the machine learning module predicts the classification result of the summary source based on the features extracted by the first two modules.

Images