A chinese clickbait detection method based on pre-trained language model
By combining a pre-trained Chinese BERT model with full word coverage and Masked Language Model tasks, the low accuracy and data dependency issues of Chinese clickbait detection are solved, achieving efficient clickbait detection results suitable for Chinese news headlines.
Patent Information
- Application Number
- CN202310990840.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-08
- Publication Date
- 2026-02-03
- Estimated Expiration
- 2043-08-08
AI Technical Summary
Existing clickbait detection methods suffer from low detection accuracy, require large amounts of labeled data and feature engineering, and have poor adaptability in Chinese environments. In particular, lexical similarity-based, traditional machine learning, and convolutional neural network models are not effective in Chinese clickbait detection.
We employ a Chinese BERT model based on a pre-trained language model. Through full word coverage and Masked Language Model tasks, we perform feature extraction and classification using a small amount of data. This includes a combination of embedding layers, encoder training, and softmax classifiers. We use a tagger for sentence tagging and encoding, and utilize contextual semantics for prediction.
It improves the accuracy of Chinese clickbait detection to 87.35%, outperforms other methods on a small dataset, reduces training cost and time, and is suitable for Chinese clickbait datasets.
Smart Images
Figure CN117171342B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to the field of natural language processing, in particular to a Chinese clickbait detection method based on a pre-training language model. BACKGROUND
[0002] With the rapid development of the Internet and the fierce competition of various media, in order to obtain more user click volume and browsing volume, the title of network news gradually deviates from its original intention and is seriously inconsistent with its content, and such clickbait phenomenon distorts news facts and causes serious trouble to the public.
[0003] At present, scholars at home and abroad have done a lot of research on clickbait detection, and the main detection algorithms can be divided into algorithms based on lexical similarity, algorithms based on traditional machine learning and algorithms based on deep learning.
[0004] The above three kinds of clickbait detection methods have achieved certain results, but it is still difficult to meet the complex and numerous clickbait detection tasks. Among them, the detection algorithm based on lexical similarity generally has a problem, part of the news title although there is an exaggerated component, but there is still a semantic connection between the words that make up the title and the content, which is easy to mislead the detection result; in addition, this method is also a difficulty for the extraction of the theme sentence of the news, which needs to consume a lot of time resources, and the accuracy is not high; the algorithm based on traditional machine learning can effectively detect clickbait, but when facing different fields, it needs to redefine the features and complete a large amount of feature engineering tasks, and the result is not necessarily accurate when detecting the regular news title, and a large amount of time cost is needed; the method of using a convolutional neural network model to detect false news has a certain accuracy, but a large amount of labeled data is needed for training, and in the case of less training samples in actual application, the clickbait cannot be accurately detected, and these methods cannot be well applied in the field of Chinese clickbait detection. SUMMARY
[0005] The purpose of the application is to overcome the defects in the prior art, provide a Chinese clickbait detection method based on a pre-training language model, extract useful features based on a Chinese BERT pre-training language model based on full word coverage, detect clickbait and improve classification accuracy.
[0006] The purpose of the application is achieved by a Chinese clickbait detection method based on a pre-training language model, comprising the following steps:
[0007] Step 1: processing the input in the pre-training language model, encoding the input and converting it into the encoding format required by the model;
[0008] Step 2: Build the embedding layer of the BERT model to convert the sequence encoding obtained in step 1 into a continuous N-dimensional space vector, and the entire embedding layer includes three components, namely embedding transformation, layer normalization and random inactivation;
[0009] Step 3: Build the encoder and perform nonlinear representation on the hidden state output by the embedding layer in step 2, train the model by pre-training, and extract features using the MLM task;
[0010] Step 4: Classify on the Chinese clickbait dataset using a softmax classifier to obtain the classification result.
[0011] Further, the step 1 specifically comprises:
[0012] Step 1.1: Use the tokenizer to mark the sequence, perform rule-based tokenization on the entire sentence, and take a single Chinese character as the basic unit of text, mark the beginning and separation of the sentence using auxiliary markers [CLS] and [SEP], and mark the end of the entire sequence with [PAD]; For example, when a news title X is input into the BERT model, the BERT model will first mark the sequence using the tokenizer, perform rule-based tokenization on the entire sentence, and take a single Chinese character as the basic unit of text. The above example will be separated into "[CLS] X [SEP] [PAD]".
[0013] Step 1.2: After separation, the BERT model will treat each space-separated character as a token, and then map it to an encoding consisting of integers by the tokenizer, including token encoding, position encoding, sentence position encoding, and attention encoding.
[0014] Further, the step 2 specifically comprises:
[0015] Step 2.1: Embedding transformation is to convert vectors into low-dimensional and dense vectors through linear transformation, assuming that n is the encoding of t, which usually refers to the number of words in the vocabulary, W is a V-row H-column matrix, V represents the size of the vocabulary, and H represents the length of the word embedding. The embedding e of t can be represented as shown in (1),
[0016] e=W n (1),
[0017] At this time, the obtained sequence can be represented as an encoding vector.
[0018] Step 2.2: Layer normalization, assuming that the input word embedding is e = [x0, x1,..., x H-1 ], where x kwhere H is the length of the word embedding, and the layer normalization is obtained as shown in (2),
[0019]
[0020] where y k is the output, μ and σ 2 are the mean and variance, respectively, ε is a very small constant to avoid division by zero when calculating the denominator, sqrt(σ 2 + ε) represents the square root of the variance, α k and β k are the learned parameters to prevent the degradation of the model's representation ability.
[0021] Step 2.3: Random Dropout is a commonly used regularization method in the field of deep learning to prevent model overfitting, and the output value y is shown in (3),
[0022]
[0023] where P ∈ [0, 1] represents the probability of each neuron being randomly deactivated during training, a random number between 0 and 1 is randomly set, x is a one-dimensional component in the word embedding, r ∈ [0, 1] is a random number, and the encoded continuous multi-dimensional space vector is obtained.
[0024] Further, the step 3 specifically comprises:
[0025] Step 3.1: The pre-training method trains the model as a Chinese BERT pre-training language model based on full word coverage, and uses the model to replace all Chinese characters that make up the same word with [MASK] to complete the pre-training task.
[0026] Step 3.2: The MLM task predicts the correct word at this position using the context semantics by randomly covering some words in the sentence, and the loss function is mainly composed of the word-level classification task and the sentence-level classification task of the MLM task. In the classification task, cross entropy (Cross Entropy) is usually used as the loss function, and the loss function is defined as shown in (4),
[0027]
[0028] where N is the number of samples, K is the number of categories, y ik is a binary label representing the probability that the i-th sample belongs to the k-th category, is the output of the model.
[0029] Step 3.3: After pre-training, the BERT model is fine-tuned using Chinese click bait corpus, converted into an input format that the BERT model can read, and then fed into the BERT model for iterative training. After training, the classification effect is detected by testing samples.
[0030] Furthermore, the classification process described in step 4 is as follows:
[0031] Training was performed using a Chinese clickbait corpus, in softmax.
[0032]
[0033] Where P(y=j|x) represents the probability that a sample belongs to class j given input x, and x is the input feature vector. T w represents the transpose of the input vector. j and w k Let these represent the weight vectors for category j and category k, respectively. Indicates x T and w j The inner product is expressed in the form of an exponential function to obtain a non-negative number. K is the number of output nodes, i.e. the number of categories. The softmax function can be used to convert the output values of binary classification into a probability distribution in the range [0, 1].
[0034] Compared with the prior art, the beneficial effects of the present invention are as follows: First, the present invention proposes a Chinese clickbait detection method based on a pre-trained language model, which completes the prediction in the form of a cloze test through the Masked Language Model (MLM) task; Second, by introducing a Chinese pre-trained language model based on full word coverage, a good detection effect can be achieved with a small amount of data, and the effect on the Chinese clickbait dataset can reach 87.35%. Attached Figure Description
[0035] Figure 1 This is a general framework diagram of the detection method of the present invention.
[0036] Figure 2 This is a schematic diagram of the embedded layer in the detection method of the present invention. Detailed Implementation
[0037] A Chinese clickbait detection method based on a pre-trained language model includes the following steps:
[0038] Step 1: Process the input in the pre-trained language model, encode the input, and convert it into the encoding format required by the model;
[0039] Step 2: Construct the embedding layer of the BERT model, converting the sequence encoding obtained in Step 1 into a continuous N-dimensional space vector. The entire embedding layer consists of three components: embedding transformation, layer normalization, and random deactivation.
[0040] Step 3: Construct the encoder and non-linearly represent the hidden states output by the embedding layer in Step 2. Train the model through pre-training and extract its features using the MLM task.
[0041] Step 4: Classify the Chinese click bait dataset using a softmax classifier to obtain the classification results.
[0042] Furthermore, step 1 specifically includes:
[0043] Step 1.1: Use the tokenizer to tokenize the sequence, perform rule-based tokenization on the entire sentence, and use a single Chinese character as the basic unit of the text. Use auxiliary tags [CLS] and [SEP] to mark the beginning and end of the sentence respectively, and use [PAD] to mark the end of the entire sequence. For example, when a news headline X is input into the BERT model, the BERT model will first use the tokenizer to tokenize the sequence, perform rule-based tokenization on the entire sentence, and use a single Chinese character as the basic unit of the text. The above example will be split into "[CLS]X[SEP][PAD]".
[0044] Step 1.2: After the separation is completed, the BERT model treats each space-separated character as a token, and then the tokenizer maps it into an encoding composed of integers, including token encoding, position encoding, sentence position encoding and attention encoding.
[0045] Furthermore, step 2 specifically includes:
[0046] Step 2.1: Embedding transformation is a linear transformation that converts a vector into a low-dimensional and dense vector. Assuming that n is the code of t, usually referring to the number in the vocabulary, and W is a V-row H-column matrix, where V represents the size of the vocabulary and H represents the length of the word embedding, then the embedding e of t can be expressed as shown in (1).
[0047] e = W n (1),
[0048] At this point, the obtained sequence can be represented as an encoding vector.
[0049] Step 2.2: Layer normalization, assuming the input word embeddings are e = [x0, x1, ..., x...]. H-1 ], where x kLet H be the k-th dimension component of e, and H be the length of the word embedding. From this, layer normalization can be obtained, as shown in (2).
[0050]
[0051] Among them, y k The outputs are μ and σ. 2 These are the mean and variance, respectively. ε is a very small constant used to avoid dividing by zero when calculating the denominator. sqrt(σ) 2 +ε) represents the calculation of the square root of the variance, α k and β k These are the learned parameters used to prevent the model's representational power from degrading.
[0052] Step 2.3: Random deactivation is a commonly used regularization method in the field of deep learning to prevent model overfitting. Its output value y is shown in (3).
[0053]
[0054] Where P∈[0,1] represents the probability that each neuron is randomly deactivated during training, and is randomly set to a value between 0 and 1, x is a one-dimensional component in the word embedding, and r∈[0,1] is a random number, thus obtaining the encoded continuous multidimensional space vector.
[0055] Furthermore, step 3 specifically includes:
[0056] Step 3.1: The pre-training method trains the Chinese BERT pre-trained language model based on full word coverage. Using this model, all Chinese characters that make up the same word are replaced with [MASK] to complete the pre-training task.
[0057] Step 3.2: The MLM task randomly masks some words in a sentence and uses the semantics of the context to predict the correct word at that position. The loss function mainly consists of word-level classification and sentence-level classification tasks of the MLM task. In the classification task, cross-entropy is usually used as the loss function. The loss function is defined as shown in (4).
[0058]
[0059] Where N is the number of samples, K is the number of categories, and y ik It is a binary label representing the probability that the i-th sample belongs to the k-th class. It is the output of the model.
[0060] Step 3.3: After pre-training, the BERT model is fine-tuned using Chinese click bait corpus, converted into an input format that the BERT model can read, and then fed into the BERT model for iterative training. After training, the classification effect is detected by testing samples.
[0061] Furthermore, the classification process in step 4 is as follows:
[0062] Training was performed using a Chinese clickbait corpus, in softmax.
[0063]
[0064] Where P(y=j|x) represents the probability that a sample belongs to class j given input x, and x is the input feature vector. T w represents the transpose of the input vector. j and w k Let these represent the weight vectors for category j and category k, respectively. Indicates x T and w j The inner product is expressed in the form of an exponential function to obtain a non-negative number. K is the number of output nodes, i.e. the number of categories. The softmax function can be used to convert the output values of binary classification into a probability distribution in the range [0, 1].
[0065] This invention can be further illustrated by the following experiments:
[0066] To test the effectiveness of the detection method of this invention, prediction results were achieved on a Chinese clickbait dataset. The dataset includes 14,922 Chinese headlines, half of which contain news type information and are clickbait headlines. These news headlines were taken from four representative domestic online news platforms: Tencent News, 163 News, Sohu News, and Sina News, as well as some popular forums and WeChat public accounts. To ensure a realistic measurement of the detection effect, the following four representative evaluation metrics were selected and defined as follows:
[0067]
[0068]
[0069]
[0070]
[0071] Where tp is the number of click decoys correctly predicted by the algorithm; fp is the number of non-click decoys predicted by the algorithm; fn is the number of non-click decoys predicted by the algorithm but which are actually click decoys; tn is the number of non-click decoys correctly predicted by the algorithm; N is the total number of articles predicted; and the F1 score is the harmonic mean of precision and recall.
[0072] To demonstrate the performance of the test results, the detection models CIBTSS (based on lexical similarity algorithm), PBCD (based on machine learning algorithm), TextCNN, and CBCNN (based on convolutional neural network) were selected. The prediction results are shown in Table 1. Table 1 shows that the prediction results of this invention on the Chinese clickbait dataset are 85.13%, 92.17%, and 87.35% in terms of precision, recall, and accuracy, respectively, significantly outperforming the prediction results of the CIBTSS, PCCD, TextCNN, and CBCNN models.
[0073] Table 1. Comparison results of BERT model with four baseline methods (%)
[0074]
[0075] The click decoy detection method of this invention can effectively extract the features of Chinese click decoys and improve the accuracy of dataset classification. By using a fine-tuning method to pre-train the model, more accurate detection results can be obtained with less data, which greatly reduces the training cost of the model and has high practicality, making it suitable for use in text classification.
[0076] This invention is not limited to the above embodiments. Based on the technical solutions disclosed in this invention, those skilled in the art can make some substitutions and modifications to some of the technical features without creative effort, and all such substitutions and modifications are within the protection scope of this invention.
Claims
1. A Chinese clickbait detection method based on a pre-trained language model, characterized in that, Includes the following steps: Step 1: Process the input in the pre-trained language model, encode the input, and convert it into the encoding format required by the model; Step 2: Construct the embedding layer of the BERT model, converting the sequence encoding obtained in Step 1 into a continuous N-dimensional space vector. The entire embedding layer consists of three components: embedding transformation, layer normalization, and random deactivation. Step 3: Construct the encoder and non-linearly represent the hidden states output by the embedding layer in Step 2. Train the model using pre-training and extract its features using the MLM task; specifically including: Step 3.1: The pre-training method trains the Chinese BERT pre-trained language model based on full word coverage. Using this model, all Chinese characters that make up the same word are replaced with [MASK] to complete the pre-training task. Step 3.2: The MLM task randomly masks some words in a sentence and uses the semantics of the context to predict the correct word at that position. The loss function mainly consists of word-level classification and sentence-level classification tasks of the MLM task. In the classification task, cross-entropy is usually used as the loss function. The loss function is defined as shown in (4). （4）， Where N is the number of samples, and K is the number of categories. y ik It is a binary label representing the probability that the i-th sample belongs to the k-th class. It is the output of the model; Step 3.3: After pre-training, the BERT model is fine-tuned using "clickbait" news corpus, converted into an input format that the BERT model can read, and then fed into the BERT model for iterative training. After training, the classification effect is detected by testing samples. Step 4: Classify the Chinese click bait dataset using a softmax classifier to obtain the classification results.
2. The Chinese click decoy detection method based on a pre-trained language model according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Use the tokenizer to tokenize the sequence, perform rule-based tokenization on the entire sentence, and use a single Chinese character as the basic unit of the text. Use auxiliary tags [CLS] and [SEP] to mark the beginning and end of the sentence respectively, and use [PAD] to mark the end of the entire sequence. Step 1.2: After the separation is completed, the BERT model treats each space-separated character as a token, and then the tokenizer maps it into an encoding composed of integers, including token encoding, position encoding, sentence position encoding and attention encoding.
3. The Chinese click decoy detection method based on a pre-trained language model according to claim 1, characterized in that, Step 2 specifically includes: Step 2.1: Embedding transformation is a linear transformation that converts a vector into a low-dimensional and dense vector. Assume... n for t The encoding usually refers to the numbering in a word. W It is V OK H A matrix of columns, V Indicates the size of the vocabulary. H To indicate the length of the word embedding, then... t Embedded e It can be represented as shown in (1), （1）， At this point, the obtained sequence can be represented as an encoding vector; Step 2.2: Layer normalization, assuming the input word embeddings are ,in, x k for e The k The components of dimension H are the length of the word embedding, from which layer normalization can be obtained, as shown in (2). （2）， in, y k It is the output. μ and σ 2 These are the mean and variance, respectively. ε It is a very small constant used to avoid dividing by zero when calculating the denominator. sqrt(σ 2 +ε) This represents the square root of the variance. α k and β k These are the learned parameters used to prevent the model's representational power from degrading. Step 2.3: Random deactivation is a commonly used regularization method in deep learning to prevent model overfitting. Its output value... y As shown in (3), （3）， in, P ∈[0, 1], representing the probability that each neuron is randomly deactivated during training, and is randomly set to a value between 0 and 1. x It is a one-dimensional component in word embedding. r ∈[0,1] is a random number, from which a continuous multidimensional space vector is obtained for encoding.
4. The Chinese click decoy detection method based on a pre-trained language model according to claim 1, characterized in that: The classification process described in step 4 is as follows: Training was performed using a Chinese clickbait corpus, in softmax. （5）， in, P(y=j|x) Indicates the given input x The sample belongs to the category j The probability, x It is the input feature vector. x T This represents the transpose of the input vector. w j and w k Representing categories j and categories k The weight vector, Indicates to x T and w j The inner product is expressed in the form of an exponential function to obtain a non-negative number. K is the number of output nodes, i.e. the number of categories. The softmax function can be used to convert the output values of binary classification into a probability distribution in the range [0, 1].