False comment detection method based on BERT-CNN

Abstract

The invention proposes a BERT-CNN-based false comment detection method, and the method comprises the steps: firstly, capturing deep representation on text data through a BERT model, and enabling the BERT to be capable of capturing different levels of language information from shallow grammar features to deep semantic features through the deep structure of a plurality of Encoder layers of the BERT; and then the contextual information is input into the convolutional neural network, so that the sequence can learn the contextual information more completely; and then learning information of BERT and information of CNN are combined, and classification is realized by using a softmax classifier through full connection processing. According to the method, the BERT model is used, the BERT model abandons a traditional neural network structure, meanwhile, the characteristics of the Transform model are combined, a more excellent result is obtained while the calculation amount is reduced and the parallel efficiency is improved, the problem that current false comment detection is inaccurate is solved, and the accuracy and robustness of detection can be improved.

Description

Technical Field

[0001] The present invention relates to a false comment detection method based on BERT-CNN, and belongs to the fields of natural language processing, deep learning, etc. Background Art

[0002] With the rapid development of B2C e-commerce, consumers rely heavily on online reviews of products before choosing and purchasing them online, which makes product reviews more important for the products.

[0003] False reviews can cause a lot of trouble to consumers, making it impossible for them to get the real information about the product, and thus preventing them from buying the products they really want. If consumers go for after-sales service, it will waste their time, causing a lot of trouble and unnecessary losses. For merchants, false reviews can also have advantages and disadvantages. On the one hand, good reviews written by others hired by merchants will make the merchant's products known to more people, but there will also be malicious bad reviews from peers, which is also a big trouble for merchants. At the same time, real reviews will also promote merchants to upgrade and iterate their products, resulting in the upgrading of production technology, which is very beneficial to the industry. In turn, the upgrading of the industry will further reduce the price of products and benefit consumers. It can be seen that false reviews have a very serious impact on both merchants and consumers. Therefore, the study of false reviews has important social significance and economic value.

[0004] At present, the research on fake reviews mainly includes traditional fake review detection methods, graph structure-based methods, and machine learning-based methods. Traditional fake review research is based on behavioral and linguistic features, which is a relatively early research. The detection method based on graph structure is to connect the relationship between comments, reviewers and products into a graph structure, and regard the detection problem as a classification sorting or joint classification problem. In terms of machine learning, the detection of fake reviews requires manual labeling of training sets to select and extract features. The extraction of high-dimensional features is not in place, and the training model is not good. In essence, it is mainly because the learning ability of neural networks is not good enough, the model is not perfect enough, and the computing efficiency needs to be improved. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of low recognition and poor robustness of online false comments in the prior art, and to provide a false comment detection method based on BERT-CNN.

[0006] Technical Solution

[0007] The present invention uses the BERT model to learn the comment content, extract features, fully learn the sentence information, mine the information, and then perform context learning through CNN to improve the accuracy of false comment detection. The specific scheme is as follows:

[0008] A false review detection method based on BERT-CNN, comprising the following steps:

[0009] S1. Obtain a comment text dataset and preprocess the obtained dataset;

[0010] S2. Input the processed data set into the BERT model. The BERT model consists of multiple Encoder layers, including a position encoding layer, a multi-head attention layer, an addition and normalization layer, and a feedforward network layer. After multiple Encoder layers, the BERT masked language task and the next sentence prediction task are completed to obtain the text word vector trained by BERT.

[0011] S3. Input the text word vector trained by BERT into the convolutional neural network to learn the local semantics of the text;

[0012] S4. Obtain label results through softmax classifier classification.

[0013] Furthermore, in step S1, the preprocessing includes deleting unnecessary symbols, processing the text format, and replacing numbers and characters with unified symbols.

[0014] Further, in step S2, the masked language task refers to randomly masking a portion of words in a sentence, and then simultaneously predicting the masked words using contextual information, so as to better understand the meaning of the words based on the full text;

[0015] The next sentence prediction task is to enable the model to better understand the relationship between sentences.

[0016] Further, in step S2, the position encoding layer adopts different strategies to implement different position encodings. After position encoding, a coding matrix X with the same dimension as the input is obtained. pos , when it is superimposed on the original word embedding, a new word embedding matrix is ​​obtained, as shown in formula (1)

[0017] X embedding =X embedding +X pos (1).

[0018] Further, in step S2, the multi-head attention layer is composed of multiple scaling points and a self-attention mechanism;

[0019] The attention mechanism maps the query and a set of key-value pairs into a high-dimensional space, that is, it performs a weighted sum calculation on the value, where the weight value is calculated by the query and the key;

[0020] For scaled dot product attention, we multiply the query vector (Q) by the key vector (K) and then perform a vector multiplication of size The scaling of k Represents the vector dimension size of the query vector (Q) and the key vector (K); after normalization, it is multiplied with the value vector (V) to obtain the final output, as shown in formula (2):

[0021]

[0022] Q, K, and V use the self-attention method to input their own X embedding To generate, that is, to generate by linear mapping, as shown in equations (3), (4), and (5):

[0023] Q=X embedding *W Q (3)

[0024] K=X embedding *W K (4)

[0025] V=X embedding *W V (5)

[0026] The three weight matrices are all square matrices with dimension Embedding;

[0027] Multi-head attention is to concatenate the output results of multiple scaled dot product attentions, which has multiple groups of W Q , W K , W V The weight matrix of each set of weight matrices is randomly initialized. embedding The vector is cut into multiple parts and multiplied by the weight to form the input vector W i X, as shown in equations (6), (7), and (8), produces Q i , K i , V i , i = 1, ..., h, where h represents the number of split heads, calculate the Attention weight matrix to get each head, and finally merge the heads and pass them through the fully connected layer (W O ) is transformed to obtain the final result, as shown in formula (9) and (10):

[0028] Q i =QW iQ , i = 1, ..., h (6)

[0029] K i =KW i K , i = 1, ..., h (7)

[0030] V i =VW i V , i = 1, ..., h (8)

[0031] head i =Attention(Q i , K i , V i ), i = 1, ..., h (9)

[0032] MultiHead(Q,K,V)=Concat(head1,...,head n )W O (10).

[0033] Further, in step S2, the addition and normalization layer includes an Add&Norm layer, Add adds the input and output of the self-attention layer (residual connection), and LayerNorm performs layer normalization to normalize the hidden layer in the neural network to a standard normal distribution to accelerate convergence.

[0034] Furthermore, in step S2, the feedforward network layer is composed of two fully connected layers plus a ReLU activation function, as shown in formula (11):

[0035] FFN(x)=max(0,xW1+b1)W2+b2 (11)

[0036] Among them, W1 and b1 represent the weight and bias of the first fully connected layer, respectively, and W2 and b2 represent the weight and bias of the second fully connected layer, respectively.

[0037] Further, in step S3, the convolutional neural network includes an embedding layer, a convolution layer, a pooling layer and a fully connected layer;

[0038] In the embedding layer, the text word vectors that have been trained by BERT are combined into an n×m matrix Z=[W1,…,W n ], where W i The word vector output by BERT;

[0039] The convolution layer extracts convolution features from the input text. Each convolution kernel is equivalent to a window. The window is continuously moved to perform semantic learning on the input text and extract different semantic features.

[0040] Convolve the convolution kernel with the input matrix. The convolution formula is shown in formula (12):

[0041] S=f(WZ+b) (12)

[0042] Where S represents the feature matrix after the convolution operation, W represents the weight matrix, and b represents the offset vector;

[0043] After convolution, feature mapping is required. Each convolution layer contains multiple feature maps. The mapping operation is completed by the ReLU function, as shown in formula (13):

[0044] ReLU = max(0, x) (13)

[0045] The function of the pooling layer is to select and filter features, simplify the computational complexity and extract the main features;

[0046] The function of the fully connected layer is to combine the extracted features, use the learned features to complete the learning objectives, and send the output value to the classifier.

[0047] Furthermore, in step S4, the softmax classifier uses the softmax function to perform probability calculation. In order to prevent data overflow, the maximum value among the output values ​​is subtracted from each output value, as shown in equations (14) and (15):

[0048] D=max(Z) (14)

[0049]

[0050] Where Z = [X [CLS] ,Z1,…,Z n ], X [CLS] After BERT training, the output is [Z1,…,Z n ] is the feature matrix extracted by CNN after BERT training, and D is [Z1,…,Z n ];

[0051] Finally, the cross entropy function is used to continuously update and optimize the parameters of the model, as shown in formula (16):

[0052]

[0053] Among them, C represents the size of the training set, y c Represents the label value of the comment, p c Model classification probability for the review.

[0054] Beneficial effects of the present invention:

[0055] The present invention provides a false comment detection method based on BERT-CNN. First, the comment data is preprocessed, and then BERT is used to extract the global semantic features of the text. BERT consists of multiple Encoder layers and can better learn the semantic features of the context than the general neural network model. Then the learned vector matrix is ​​input into CNN for local semantic extraction, and finally the output layer performs combined output. Compared with the prior art, the method of the present invention improves the accuracy of detection and solves the problem of poor accuracy of current false comments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a false comment detection method based on BERT-CNN in Example 1;

[0057] Figure 2 This is the overall model diagram of BERT in Example 1;

[0058] Figure 3 This is the Encoder layer model diagram of BERT in Example 1;

[0059] Figure 4 This is the model design diagram of CNN in Example 1. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0061] Example 1

[0062] like Figure 1 , a fake review detection method based on BERT-CNN, comprising the following steps:

[0063] S1. First, obtain the comment text dataset through crawlers and preprocess the obtained dataset so that the processed data can be better used for feature learning by the BERT model to improve the completion of training;

[0064] The preprocessing specifically includes: deleting unnecessary symbols, processing the text format, and replacing numbers and characters with unified symbols;

[0065] S2. Input the data into the BERT model. The overall model diagram of BERT is as follows Figure 2 As shown in the figure, through the training of the BERT model, the masked language task and the next sentence prediction task of BERT are completed (the masked language task is to randomly mask a part of the words in the sentence, and then use the context information to predict the masked words at the same time, so as to better understand the meaning of the words according to the full text; the next sentence prediction task is to enable the model to better understand the relationship between sentences.), and the text word vector trained by BERT is obtained;

[0066] Among them, the basic structure of BERT is composed of multiple Encoder layers, such as Figure 3 As shown, the Encoder layer includes a position encoding layer, a multi-head attention layer, an addition and normalization layer, and a feedforward network layer.

[0067] After the data is input into the input word embedding layer of the BERT model, each word in the sentence is converted into a dimensional vector to represent it, so Input: X = [X1, X2, ..., X n ], X becomes X after embedding embedding , the dimensions include the number of sentences in a batch, the number of words in each sentence, and the embedding dimension of each word;

[0068] Then enter the position encoding layer to mark the input position. After position encoding, we get a coding matrix X that is exactly the same as the input dimension. pos , when it is superimposed on the original word embedding, a new word embedding matrix is ​​obtained, as shown in formula (1):

[0069] X embedding =X embedding +X pos (1)

[0070] Next, the position-encoded word embedding matrix is ​​input into a multi-head attention layer, which consists of multiple scaling points and a self-attention mechanism.

[0071] The attention mechanism maps the query and a set of key-value pairs to a high-dimensional space, that is, e Perform a weighted sum calculation, where the weight value is calculated based on the query and the key.

[0072] For scaled dot product attention, we multiply the query vector (Q) by the key vector (K) and then perform a vector multiplication of size The scaling of k Represents the vector dimension size of the query vector (Q) and the key vector (K). After normalization, it is multiplied with the value vector (V) to obtain the final output, as shown in formula (2):

[0073]

[0074] Q, K, and V use the self-attention method to input their own X embedding To generate, that is, to generate by linear mapping, as shown in equations (3), (4), and (5):

[0075] Q=X embedding *WQ (3)

[0076] K=X embedding *W K (4)

[0077] V=X embedding *W V (5)

[0078] The three weight matrices are all square matrices with dimension Embedding;

[0079] Multi-head attention is to concatenate the output results of multiple scaled dot product attentions, which has multiple groups of W Q , W K , W V The weight matrix of each set of weight matrices is randomly initialized. embedding The vector is cut into multiple parts and multiplied by the weight to form the input vector W i X, as shown in equations (6), (7), and (8), produces Q i , K i , V i , i = 1, ..., h, where h represents the number of split heads, calculate the Attention weight matrix to get each head, and finally merge the heads and pass them through the fully connected layer (W O ) is transformed to obtain the final result, as shown in formula (9) and (10):

[0080] Q i =QW i Q , i = 1, ..., h (6)

[0081] K i =KW i K , i = 1, ..., h (7)

[0082] V i =VW i V , i = 1, ..., h (8)

[0083] head i =Attention(Q i , K i , V i ), i = 1, ..., h (9)

[0084] MultiHead(Q,K,V)=Concat(head1,...,head n )W O (10).

[0085] Next, the vector output by the multi-head attention layer is input into the addition and normalization layer, which includes the Add&Norm layer. Add adds the input and output of the self-attention layer (residual connection), and LayerNorm performs layer normalization to normalize the hidden layer in the neural network to a standard normal distribution to accelerate convergence.

[0086] Next, we enter the feedforward network layer (FFN), which consists of two fully connected layers plus a ReLU activation function, as shown in formula (11):

[0087] FFN(x)=max(0,xW1+b1)W2+b2 (11)

[0088] Where W1 and b1 represent the weight and bias of the first fully connected layer, respectively, and W2 and b2 represent the weight and bias of the second fully connected layer, respectively. Each fully connected layer is normalized using LayerNorm.

[0089] After passing through the FFN layer, an Add&Norm layer is performed, as described above, to obtain a new word output vector, which is then input into the next Encoder layer; after passing through multiple Encoder layers, the two tasks of BERT are achieved.

[0090] After passing through the FFN layer, an Add&Norm layer is performed, as described above, to obtain a new word output vector, which is then input into the next Encoder layer. After passing through multiple Encoder layers, the two tasks of BERT are achieved.

[0091] S3. Input the text word vector trained by BERT into the convolutional neural network (CNN) to learn the local semantics of the text;

[0092] The design of the convolutional neural network is as follows: Figure 4 As shown, it includes embedding layer, convolution layer, pooling layer and fully connected layer;

[0093] In the embedding layer, the text word vectors that have been trained by BERT are combined into an n×m matrix Z=[W1,…,W n ], where W i The word vector output by BERT;

[0094] The convolution layer extracts convolution features from the input text. Each convolution kernel is equivalent to a window. The window is continuously moved to perform semantic learning on the input text and extract different semantic features.

[0095] Convolve the convolution kernel with the input matrix. The convolution formula is shown in formula (12):

[0096] S=f(WZ+b)(12)

[0097] Where S represents the feature matrix after the convolution operation, W represents the weight matrix, and b represents the offset vector;

[0098] After convolution, feature mapping is required. Each convolution layer contains multiple feature maps. Each feature map is a "plane" composed of multiple neurons. A feature of the input is extracted through a convolution filter. The mapping operation is completed by the ReLU function, as shown in formula (13):

[0099] ReLU = max(0, x) (13)

[0100] Then enter the pooling layer, the function of the pooling layer is feature selection and filtering, simplifying the calculation complexity and extracting the main features;

[0101] Finally, we come to the output layer, which includes a fully connected layer and a softmax output. The function of the fully connected layer is to combine the extracted features, use the learned features to complete the learning objectives, and send the output value to the classifier.

[0102] S4. Obtain label results through softmax classifier classification.

[0103] The classifier uses the softmax function to calculate probability. In order to prevent data overflow, the maximum value among the output values ​​is subtracted from each output value, as shown in equations (14) and (15):

[0104] D=max(Z) (14)

[0105]

[0106] Where Z = [X [CLS] ,Z1,…,Z n ], X [CLS] After BERT training, the output is [Z1,…,Z n ] is the feature matrix extracted by CNN after BERT training, and D is [Z1,…,Z n ];

[0107] Finally, the cross entropy function is used to continuously update and optimize the parameters of the model, as shown in formula (16):

[0108]

[0109] Among them, C represents the size of the training set, y c Represents the label value of the comment, p c Model classification probability for the review.

[0110] The above is only a preferred embodiment of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A false review detection method based on BERT-CNN, characterized in that: The steps include: S1. Obtain a comment text dataset and preprocess the obtained dataset; S2. Input the processed data set into the BERT model. The BERT model consists of multiple Encoder layers, including a position encoding layer, a multi-head attention layer, an addition and normalization layer, and a feedforward network layer. After multiple Encoder layers, the BERT masked language task and the next sentence prediction task are completed to obtain the text word vector trained by BERT. S3. Input the text word vector trained by BERT into the convolutional neural network to learn the local semantics of the text; S4. Obtain label results through softmax classifier classification.

2. According to the method for detecting false comments based on BERT-CNN in claim 1, it is characterized in that: In step S1, the preprocessing includes deleting unnecessary symbols, processing the text format, and replacing numbers and characters with unified symbols.

3. According to the method for detecting false comments based on BERT-CNN in claim 1, it is characterized in that: In step S2, the masked language task refers to randomly masking a portion of words in a sentence, and then simultaneously predicting the masked words using contextual information, so as to better understand the meaning of the words based on the full text; The next sentence prediction task is to enable the model to better understand the relationship between sentences.

4. The method for detecting false comments based on BERT-CNN according to claim 1, characterized in that: In step S2, the position encoding layer uses different strategies to implement different position encodings. After position encoding, a coding matrix X with the same dimension as the input is obtained. pos , when it is superimposed on the original word embedding, a new word embedding matrix is ​​obtained, as shown in formula (1): X embedding =X embedding +X pos (1)。 5. The method for detecting false comments based on BERT-CNN according to claim 1, characterized in that: In step S2, the multi-head attention layer consists of multiple scaling points and a self-attention mechanism; The attention mechanism maps the query and a set of key-value pairs into a high-dimensional space, that is, it performs a weighted sum calculation on the value, where the weight value is calculated by the query and the key; For scaled dot product attention, we multiply the query vector (Q) by the key vector (K) and then perform a vector multiplication of size The scaling of k Represents the vector dimension size of the query vector (Q) and the key vector (K); after normalization, it is multiplied with the value vector (V) to obtain the final output, as shown in formula (2): Q, K, and V use the self-attention method to input their own X embedding To generate, that is, to generate by linear mapping, as shown in equations (3), (4), and (5): Q=X embedding *W Q (3) K=X embedding *W K (4) V=X embedding *W V (5) The three weight matrices are all square matrices with dimension Embedding; Multi-head attention is to concatenate the output results of multiple scaled dot product attentions, which has multiple groups of W Q , W K , W V The weight matrix of each set of weight matrices is randomly initialized. embedding The vector is cut into multiple parts and multiplied by the weight to form the input vector W i X, as shown in equations (6), (7), and (8), produces Q i , K i , V i , i = 1, ..., h, where h represents the number of split heads, calculate the Attention weight matrix to get each head, and finally merge the heads and pass them through the fully connected layer (W O ) is transformed to obtain the final result, as shown in formula (9) and (10): Q i =QW i Q ,i=1,…,h (6) K i =VW i K ,i=1,…,h (7) V i =VW i V ,i=1,…,h (8) head i =Attention(Q i ,K i ,V i ),i=1,…,h (9) MultiHead(Q,K,V)=Concat(head1,…,head n )W O (10)。 6. The method for detecting false comments based on BERT-CNN according to claim 1, characterized in that: In step S2, the addition and normalization layer includes an Add&Norm layer, Add adds the input and output of the self-attention layer (residual connection), and LayerNorm performs layer normalization to normalize the hidden layer in the neural network to a standard normal distribution to accelerate convergence.

7. The method for detecting fake comments based on BERT-CNN according to claim 1, characterized in that: In step S2, the feedforward network layer consists of two fully connected layers plus a ReLU activation function, as shown in formula (11): FFN(x)=max(0,xW1+b1)W2+b2 (11) Among them, W1 and b1 represent the weight and bias of the first fully connected layer, respectively, and W2 and b2 represent the weight and bias of the second fully connected layer, respectively.

8. The method for detecting fake comments based on BERT-CNN according to claim 1, characterized in that: In step S3, the convolutional neural network includes an embedding layer, a convolution layer, a pooling layer and a fully connected layer; In the embedding layer, the text word vectors that have been trained by BERT are combined into an n×m matrix Z=[W1,…,W n ], where W i The word vector output by BERT; The convolution layer extracts convolution features from the input text. Each convolution kernel is equivalent to a window. The window is continuously moved to perform semantic learning on the input text and extract different semantic features. Convolve the convolution kernel with the input matrix. The convolution formula is shown in formula (12): S=f(WZ+b) (12) Where S represents the feature matrix after the convolution operation, W represents the weight matrix, and b represents the offset vector; After convolution, feature mapping is required. Each convolution layer contains multiple feature maps. The mapping operation is completed by the ReLU function, as shown in formula (13): ReLU = max(0, x) (13) The function of the pooling layer is to select and filter features, simplify the computational complexity and extract the main features; The function of the fully connected layer is to combine the extracted features, use the learned features to complete the learning objectives, and send the output value to the classifier.

9. A false review detection method based on BERT-CNN according to any one of claims 1 to 8, characterized in that: In step S4, the softmax classifier uses the softmax function to perform probability calculation. In order to prevent data overflow, the maximum value among the output values ​​is subtracted from each output value, as shown in equations (14) and (15): D=max(Z)(14) Where Z = [X [CLS] ,Z1,…,Z n ], X [CLS] After BERT training, the output is [Z1,…,Z n ] is the feature matrix extracted by CNN after BERT training, and D is [Z1,…,Z n ]; Finally, the cross entropy function is used to continuously update and optimize the parameters of the model, as shown in formula (16): Among them, C represents the size of the training set, y c Represents the label value of the comment, p c Model classification probability for the review.

Images