A fake news detection method based on pre-training model fusion

Abstract

The application discloses a false news detection method based on pre-training model fusion, different pre-training models are used to obtain feature representations from different perspectives, similarity measurement is performed on the features through a twin convolution network, classification detection of false news is realized based on a multilayer perception network, and false news detection of short texts is realized. In the actual application of false news detection, the method has the ability of fully extracting different text features, so that the screening accuracy of false news is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of NLP, specifically to a method for detecting fake news based on the fusion of pre-trained models. Background Technology

[0002] The development of social media worldwide has brought unprecedented convenience to news dissemination. The rise of the internet has made every user not only a receiver of news but also a disseminator and publisher. Advances in internet and network technology allow anyone to easily publish any content on online platforms such as social media, blogs, and news commentaries, which also provides fertile ground for the spread of fake news. Due to the importance of fake news detection, many scholars both domestically and internationally have conducted in-depth research in this field. Early research mainly employed machine learning methods, manually constructing features and training classifiers for fake news detection. Features were mainly divided into two categories: features based on the news content itself and features based on social context. Content-based features include the number of characters per word, word frequency, total number of words, etc., while context-based features include registration duration, number of followers, credibility, and stance network, etc. However, machine learning methods rely on manual feature extraction, heavily depending on the skills of feature engineers. Furthermore, manual feature extraction struggles to extract deeper features. With the development of deep learning, it has made significant progress in text classification, possessing the ability to extract deeper features and achieving better classification results. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies in detecting fake news in short texts by providing a fake news detection method based on the fusion of pre-trained models, thereby solving the problems mentioned in the background.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for detecting fake news based on pre-trained model fusion, characterized in that it includes completing the fake news detection and classification according to steps 1 to 5 as follows:

[0005] Step 1: Input news data, including the data statement, metadata, and credit history;

[0006] Step 2: Extract features from the news data in Step 1 using the pre-trained models BERT and SKEP, and then fuse the model features in Step 3.

[0007] Step 3: Input features, perform word vectorization and credit history concatenation respectively. Word vectorization converts text data into word vectors, and credit history concatenation performs statistical analysis of various labels in the speaker's history based on the features.

[0008] Step 4: Extract twin features from the word vectors obtained in Step 3 through the pre-trained model, obtain the similarity vector by feature subtraction, and input the three-dimensional similarity vector into Step 5;

[0009] Step 5: Transform the similarity vector into a one-dimensional vector, and perform classification and prediction using a multilayer perceptron to obtain the final fake news detection result.

[0010] As a preferred embodiment of the present invention: in step 2, the pre-trained model BERT uses a random masking strategy for prediction, and its working strategy is as follows:

[0011] Strategy A: Randomly mask 15% of the tags in BERT;

[0012] Strategy B: Replace 80% of the tags in Strategy A with the [MASK] tag;

[0013] Strategy C: Replace 10% of the tags in Strategy A with random tags;

[0014] Strategy D: The remaining 10% of the tags remain unchanged.

[0015] As a preferred embodiment of the present invention: the pre-trained model SKEP in step 2 includes the following three branches: A, B, and C:

[0016] Branch A: Unsupervised sentiment knowledge mining, which uses a point mutual information-based mining method to mine sentiment knowledge from unlabeled data. The mined sentiment knowledge includes a set of sentiment words and a set of aspect-level sentiment pairs.

[0017] Branch B: Sentiment masking, including sentiment detection and hybrid sentiment masking. First, the input is detected through sentiment word detection and aspect-level sentiment pair detection, and compared with the mined knowledge to discover sentiment knowledge. Then, the input is masked. Hybrid sentiment masking consists of three steps: 1) Aspect-level sentiment masking; 2) Randomly selecting less than 10% of sentiment words for masking; 3) If less than 10% are masked, randomly select more to mask until the masking count reaches 10%.

[0018] Branch C: Pre-training objectives, including: sentiment word prediction, word polarity prediction, and aspect-level word pair prediction.

[0019] As a preferred technical solution of the present invention: the feature of word vectorization in step 3 is: word vector V can be obtained through the BERT pre-trained model. BERT V BERT ∈R batchsize×768 Word vectors V can be obtained through the SKEP pre-trained model. SKEP V SKEP ∈Rbatchsize×768 Where batchsize is the number of input data in a single training run, and 768 is the feature dimension.

[0020] As a preferred technical solution of the present invention: the credit history vector CH referred to in step 3 is: CH = {19, 32, 34, 58, 33}, where the value of each element corresponds to the number of five types of tags "pants on fire", "false", "barely true", "half true" and "mostly true" in a news release by a certain spokesperson.

[0021] As a preferred technical solution of the present invention: the model structure of the twin convolutional neural network in step 4 is composed of two branches, namely the BERT-CNN model and the SKEP-CNN model with the same network structure. The same branch includes, in sequence, a first one-dimensional convolutional layer, a first pooling layer, a second one-dimensional convolutional layer, and a second pooling layer from its word vector to the similarity direction.

[0022] As a preferred embodiment of the present invention: the calculation formula for the multilayer sensor in step 5 is:

[0023]

[0024] Where, x i Given the input of the i-th feature of a sample, w ij and w jp σ represents the connection weights between the input layer and the hidden layer, and between the hidden layer and the output layer, respectively, where σ is the activation function.

[0025] The fake news detection method based on pre-trained model fusion described in this invention has the following technical advantages compared with existing technologies:

[0026] The beneficial effects of this invention are as follows: This invention proposes a fake news detection method based on pre-trained model fusion. By using different pre-trained models to obtain feature representations in different directions, and using a Siamese convolutional network to measure the similarity of features, it achieves fake news detection in short texts. Experimental results on the public dataset LIAR show that this method significantly improves the detection capability compared to a single pre-trained model. Attached Figure Description

[0027] Figure 1 This is a flowchart of a fake news detection method based on pre-trained model fusion as described in this invention;

[0028] Figure 2 A flowchart of a BERT pre-trained model Masked LM task provided by the present invention;

[0029] Figure 3 A diagram illustrating the structure of a twin neural network model for the method provided in this invention;

[0030] Figure 4 A structural diagram of a multilayer perceptron for the method provided by this invention;

[0031] Figure 5 A confusion matrix for the experimental cases provided in this invention; Detailed Implementation

[0032] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0033] Example: This invention provides a technical solution: a fake news detection method based on pre-trained model fusion, which completes fake news detection and classification according to steps 1 to 5:

[0034] Experimental group: Please refer to Figure 1 A fake news detection method based on pre-trained model fusion was proposed. The effectiveness of the model was evaluated on the LIAR dataset through a series of experiments, including the following steps:

[0035] Perform step 1 and enter the news data, including the data statement, metadata, and credit history.

[0036] Perform step 2, processing the news data from step 1 as follows: Figure 3 Feature extraction is performed on the pre-trained models BERT and SKEP shown, and the model features are then fused in step 3.

[0037] Step 3 involves inputting features and performing word vectorization and concatenating credit history. Word vectorization converts text data into word vectors, while concatenating credit history performs statistical analysis of various labels related to the speakers' history.

[0038] Perform step 4, and then process the two pre-trained word vectors from step 3 using methods such as... Figure 4 The twin convolutional neural network shown extracts twin features and obtains a similarity vector by feature subtraction. The structure of the convolutional neural network is shown in Table 1.

[0039] Table 1. Structure of Siamese Convolutional Neural Network

[0040]

[0041] Step 5: Transform the similarity vector into a one-dimensional vector, by means of... Figure 5 The multilayer perceptron shown is used for classification and prediction to obtain the final fake news detection result.

[0042] The LIAR dataset contains nearly 13,000 short, hand-annotated phrases in various contexts, obtained from the website websitepolifact.com. It includes data collected over a decade and is labeled as: pants onfire, false, barely true, half true, mostly true, and true. The label distribution is fairly balanced: aside from 1050 cases of "pants onfire," each label has over two thousand examples. For each news item, in addition to the news phrase, speaker-related metadata was considered, including speaker's name, speech topic, occupation, state of the speech, political party, and credibility history. Table 2 presents the statistical results of the LIAR dataset.

[0043] Table 2 shows the statistical results of the LIAR dataset provided by Wang.

[0044]

[0045]

[0046] To better verify the effectiveness of the model described in this method, all hyperparameters of the model were standardized. The learning rate was set to 1e-05, which determines the step size in each iteration. `batch_size` was set to 64, indicating that 64 samples are input to the network each time. `max_seq_len` was set to 50, indicating that the maximum text length processed by the model is 50 characters; excessively long sentences are truncated, and excessively short sentences are padded. `epochs` was set to 10, representing a total of 10 training epochs.

[0047] Table 3 shows the experimental results of each model on the test set. In this method, the test set accuracy based on the BERT pre-trained model fine-tuned is 0.288, and the test set accuracy based on the SKEP pre-trained model fine-tuned is 0.291. Both pre-trained models exceed the baseline best result of 0.274, demonstrating the significant potential of pre-trained models. Further feature extraction is achieved by adding convolutional neural networks (CNNs) to the pre-trained models. Experimental results show that adding CNNs to both the BERT and SKEP pre-trained models achieves accuracy exceeding that of the original pre-trained models. The accuracy of B-CNN improves from 0.288 (BERT) to 0.294, and the accuracy of S-CNN improves from 0.291 (SKEP) to 0.303, demonstrating the effectiveness of CNNs in high-level feature extraction. Among the five models, BS-CNN achieves the highest accuracy. This model achieves an accuracy of 0.321 on the test set by fusing different pre-trained models to extract different features and then further extracting these features using a Siamese convolutional neural network. Compared to B-CNN and S-CNN, it demonstrates the effectiveness of fusing pre-trained models.

[0048] Table 3 Comparison of Model Accuracy

[0049]

[0050]

[0051] To more effectively evaluate the effectiveness of pre-trained model fusion, more detailed evaluation metrics were used for the five models. Table 4 shows the precision, recall, and F1 score of the five models under various class labels, as well as the precision, recall, and F1 score of the average across the six classes. For the six different classes, the BS-CNN method did not achieve optimal results in every class, but the average results of BS-CNN across the six classes, in terms of precision, recall, and F1, were the highest, at 0.334, 0.336, and 0.323, respectively. On the other hand, for individual models, they performed well in classification of a particular class. For example, the S-CNN model outperformed other models in the mostly true and false classifications, with F1 scores of 0.385 and 0.387, respectively, while the B-CNN model outperformed other models in the barely true and pants-fire classifications, with F1 scores of 0.309 and 0.352, respectively. This indicates that different pre-trained models, due to their ability to extract different features based on different pre-training tasks, demonstrate the effectiveness of fusing pre-trained models.

[0052] Table 4 Evaluation Indicators for Each Model

[0053]

[0054]

[0055] The classification results of the BS-CNN model can be more clearly seen by drawing a confusion matrix of the model's classification results. The BS-CNN confusion matrix is ​​shown in the figure.

[0056] from Figure 5 As can be seen, the more misclassifications occur when the category label is in the middle. For example, 84 news items with the label "false" are classified as "barele-true," while the outermost labels, such as "true" and "pants fire," perform better. To more accurately measure the specific situation of classification errors, the evaluation metric "interval i: number of classification errors" (CFC) is defined. i and the classification error ratio at interval i (RFC) i CFC i The RFC indicates the number of classification error intervals in i. i This represents the percentage of classification errors. (CFC) i and RFC i The expression is as follows:

[0057]

[0058] RFC i =CFC i / total#

[0059] ji≥0,j+i≤5#

[0060] Where L = [true, mostlytrue, halftrue, barelytrue, false, powdersfire] is an array of labels for each class, i is the misclassification interval (e.g., for half true, i = 2, then the labels with an interval of 2 from half true are true and false), TL is the true label, PL is the predicted label, and count calculates the number of samples under a certain condition. After calculation, the RFC can be obtained. 0～5 The results are shown in Table 5. Table 5 displays the detailed results from intervals of 0 to 5. RFC0 accounts for 0.321, meaning the predicted label is the same as the true label, equivalent to an accuracy of 0.321. RFC1+RFC2 is as high as 0.505, with more than half of the misclassifications occurring at intervals of 1 and 2. The experimental results demonstrate the difficulty of multi-classification problems: due to overly fine label divisions, without sufficiently distinct features, text can easily be classified into similar labels.

[0061] Table 5. Classification Error Rate

[0062]

[0063] Experimental Conclusions: For the problem of detecting fake news in short texts, the limited text length and content make feature extraction for classification a significant challenge. This paper addresses this challenge by conducting experiments on LIAR, the largest English short text fake news detection dataset. Building upon pre-trained models, and considering their ability to extract different text features due to varying pre-training tasks, we utilize two pre-trained models, BERT and SKEP, to extract features. These features are then fused to uncover multi-dimensional features, and further extracted using a Siamese convolutional neural network. Experimental results demonstrate that the pre-trained fusion model effectively extracts multi-dimensional features, achieving the best performance on the test set.

[0064] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A method for fake news detection based on pre-trained model fusion, characterized in that: The fake news detection task is completed by the following steps 1 to step 5: Step 1: input news data, including data statements, metadata, and credit history; Step 2: feature extraction of news data in step 1 by pre-training model BERT and SKEP, and feature fusion of extracted features by step 3; The pre-training model BERT in step 2 uses a random mask strategy for prediction, and its working strategy is as follows: Strategy A: randomly mask 15% of the labels in BERT; Strategy B: use [MASK] labels to replace 80% of the labels in strategy A; Strategy C: replace 10% of the labels in strategy A with random labels; Strategy D: the remaining 10% of the labels are not changed; Step 3: input features, respectively word vectorization and splice credit history, word vectorization converts text data into word vectors, and splicing credit history counts various labels of speaker history; Step 4: the word vectors obtained by the pre-training model in step 3 are extracted by twin convolutional neural network, and the similarity vector is obtained by subtracting the features, and the three-dimensional similarity vector is input to step 5; The model structure of the twin convolutional neural network in step 4 is composed of two branches of BERT-CNN model and SKEP-CNN model with the same network structure, which includes first one-dimensional convolutional layer, first pooling layer, second one-dimensional convolutional layer and second pooling layer in turn from the input of word vector to the output of similarity vector; Step 5: transform the similarity vector into a one-dimensional vector, and classify and predict by multilayer perception to obtain the final fake news detection result.

2. The method of claim 1, wherein the method is based on a pre-trained model fusion. The pre-training model SKEP in step 2 includes the following A, B and C branches: Branch A: unsupervised sentiment knowledge mining, which uses a point mutual information-based mining method to mine sentiment knowledge from unlabeled data, and the mined sentiment knowledge includes a set of sentiment words and a set of aspect-level sentiment pairs; Branch B: sentiment masking, including sentiment detection and mixed sentiment masking, first detecting the input by sentiment word detection and aspect-level sentiment pair detection, comparing with the mined knowledge to find the sentiment knowledge; then mask the input, mixed sentiment masking includes three steps: 1) aspect-level sentiment masking; 2) randomly select less than 10% of the sentiment words for masking; 3) if the number of masking is less than 10%, randomly select to mask to meet the masking number of 10%; Branch C: pre-training target, including sentiment word prediction, word polarity prediction and aspect-level word pair prediction.

3. The method of claim 1, wherein the method is based on a pre-trained model fusion. The feature of the word vectorization in the step 3 is that the word vector can be obtained through a BERT pre-training model The word vector can be obtained through a SKEP pre-training model Wherein, the batchsize is the number of input data in a single training, and 768 is the feature dimension.

4. The method of claim 1, wherein the method is based on a pre-trained model fusion. The credit history vector CH referred to in step 3 is: CH = {19, 32, 34, 58, 33}, where each element value corresponds to the number of five types of labels "pants on fire", "false", "barely true", "half true" and "mostly true" in the news published by a certain speaker.

5. The method of claim 1, wherein the method is based on a pre-trained model fusion. The calculation formula of the multilayer perception in step 5 is: wherein, is the input of the i-th feature of a given sample, and are the connection weights between the input layer and the hidden layer and the connection weights between the hidden layer and the output layer, respectively, is an activation function.

Images