A rumor detection method and system based on multi-layer coding network

Abstract

The invention discloses a rumor detection method and system based on a multi-layer coding network, which belongs to the field of natural language processing, and comprises the following steps: obtaining all texts to be detected and preprocessing the texts; embedding word fragments with a marked vocabulary into the preprocessed texts, converting words in the texts into token marked words, and then performing vector encoding to obtain word vectors corresponding to each text; processing all word vectors to obtain input vectors; inputting the input vectors into a pre-trained multi-layer coding network to generate output vectors; processing the output vectors to obtain hidden state vectors; sending the hidden state vectors into a hidden layer and a classifier to obtain the probability that the text to be detected is detected as each rumor category, and the category with the highest probability is the detection result of the text; the problems of slow training speed and poor effect are effectively alleviated, the learning of word and sentence level information is realized, the relationship representation between different texts is extracted, and the accuracy of rumor detection is improved.

Description

Technical Field

[0001] The present invention relates to a rumor detection method and system based on a multi-layer coding network, and belongs to the field of natural language processing. Background Art

[0002] The purpose of rumor verification is to determine the authenticity of a rumor. Existing research methods for rumor authenticity classification can be divided into two categories: methods based on manual feature extraction and methods based on deep learning. In recent years, in order to avoid heavy feature engineering, research has begun to apply deep learning to the task of rumor authenticity verification. In deep learning, the first model proposed used Word2Vec to obtain word embeddings for each word, which made it easier to find other words with similar semantics and was widely used in subsequent research. However, word embeddings cannot distinguish the different semantics of polysemous words, resulting in two different contextual information being encoded into the same word embedding space.

[0003] At the same time, more traditional rumor detection methods based on neural networks such as LSTM, CNNs, and RNN often ignore the relationship between different texts when encoding text vectors, and only input them as expressions of single vocabulary information. Even if the context information is paid attention to during feature extraction later, part of the original expression of the relationship between sentences will be lost. Summary of the invention

[0004] The purpose of the present invention is to provide a rumor detection method and system based on a multi-layer coding network, which effectively alleviates the problems of slow training speed and poor effect caused by the large number of words with different suffixes in English and the huge base of the vocabulary, and can help to better understand English semantics, facilitate the extraction of semantic features of sentences, and improve the accuracy of rumor detection; realize the learning of contextual information at the word level and semantic representation at the sentence level, which is conducive to extracting the relationship representation between different texts and improves the accuracy of rumor detection.

[0005] To achieve the above objectives, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a rumor detection method based on a multi-layer coding network, comprising:

[0007] Get all the texts to be detected and preprocess the texts;

[0008] Embed the word fragments with the tagged vocabulary into the preprocessed text, convert the words in the text into token tagged words, and then perform vector encoding to obtain the word vector corresponding to each text;

[0009] Process all word vectors to get the input vector;

[0010] Input the input vector into the pre-trained multi-layer encoding network to generate the output vector;

[0011] Process the output vector to obtain the hidden state vector;

[0012] The hidden state vector is sent to the hidden layer and the classifier to obtain the probability of the text to be detected as each rumor category. The category with the highest probability is the detection result of the text.

[0013] In combination with the first aspect, further, the number of texts to be detected is two.

[0014] In combination with the first aspect, further, the method for preprocessing text includes:

[0015] Punctuation marks and special characters are removed from the text to be detected, and all characters are converted to lowercase to obtain the preprocessed text.

[0016] In combination with the first aspect, further, the method of processing all word vectors to obtain input vectors includes:

[0017] A flag is added between the header of the first word vector and the two word vectors, multiple word vectors are separated and then merged, and then they are vertically spliced ​​with the position vector and the segment vectors corresponding to each word vector bit by bit to obtain the input vector.

[0018] In combination with the first aspect, further, inputting the input vector into a pre-trained multi-layer encoding network to generate an output vector includes the following steps:

[0019] The input vector is sent to the encoding network. Each layer of the encoding network contains an attention mechanism and a layer normalization network. Multiple self-attention network outputs are obtained and concatenated into a vector and output with reduced dimension. Then, the vector is residually connected with the input vector, and after layer normalization and two linear transformations, the first output vector with the same length as the input vector is obtained.

[0020] The first output vector is sent to the next layer of encoding network with the same structure to repeat the above steps until the last layer of encoding network to obtain the final output vector, which is composed of a flag representation vector representing the output of the corresponding position of the flag and each text representation vector representing the output of the corresponding position of each text.

[0021] In combination with the first aspect, further, a method of processing the output vector to obtain a hidden state vector includes:

[0022] The flag representation vector is activated and sent to the fully connected layer. Each text representation vector is added and averaged, and then activated and sent to the corresponding fully connected layer to obtain multiple primary hidden state vectors. The primary hidden state vectors are concatenated to obtain the hidden state vector.

[0023] In combination with the first aspect, further, the hidden state vector is sent to the hidden layer and the classifier to obtain the probability that the text to be detected is detected as each rumor category, including the following steps:

[0024] The hidden state vector is sent to the hidden layer, processed by the nonlinear activation function of the hidden layer, and the hidden output vector is output;

[0025] The hidden output vector is fed into the softmax regression classifier to obtain the probability that the text to be detected is detected as each rumor category:

[0026] O = softmax(ZW o +b o )

[0027] Where Z is the hidden output vector, W o is the connection weight between the hidden layer and the classifier, b o is the c-dimensional vector bias.

[0028] In combination with the first aspect, further, the rumor categories include true rumors, false rumors, and rumors whose authenticity has not been verified.

[0029] In combination with the first aspect, further, the tag vocabulary is established by a Subword algorithm.

[0030] In a second aspect, the present invention further provides a rumor detection system based on a multi-layer coding network, comprising:

[0031] Text preprocessing module: used to obtain all texts to be detected and preprocess the texts;

[0032] Vector encoding module: used to embed word fragments with a tagged vocabulary into the preprocessed text, convert the words in the text into token tagged words, and then perform vector encoding to obtain the word vector corresponding to each text;

[0033] Input vector acquisition module: used to process all word vectors to obtain input vectors;

[0034] Semantic learning module: used to input the input vector into the pre-trained multi-layer encoding network to generate the output vector;

[0035] Hidden state vector acquisition module: used to process the output vector to obtain the hidden state vector;

[0036] Detection module: used to send the hidden state vector into the hidden layer and classifier to obtain the probability of the text to be detected as each rumor category. The category with the highest probability is the detection result of the text.

[0037] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0038] The present invention provides a rumor detection method and system based on a multi-layer coding network. The method uses word fragments with a marked vocabulary to embed preprocessed text, converts words in the text into token marked words, that is, learns the relationship between affixes and constructs token-level embedding, effectively alleviating the problems of slow training speed and poor effect caused by the large number of words with different suffixes in English and the huge vocabulary base, and can help to better understand English semantics and extract semantic features of sentences, thereby improving the accuracy of rumor detection. The pre-trained multi-layer coding network is used to learn the feature vector of the text contained in the input vector and generate an output vector, which can not only learn the contextual information at the word level, but also learn the semantic representation of the higher level sentence level. For the rumor detection task, this sentence-level information has richer connotations, is conducive to extracting the relationship representation between different texts, and improves the accuracy of rumor detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a rumor detection method based on a multi-layer coding network provided by an embodiment of the present invention;

[0040] Figure 2 It is an overall framework diagram of a rumor detection method based on a multi-layer coding network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.

[0042] Example 1

[0043] With the emergence of the BERT model, encoding using the Subword algorithm has gradually become an important method for word representation; the WordPiece algorithm used in BERT encodes words as word pieces and has a good effect.

[0044] like Figure 1 As shown, a rumor detection method based on a multi-layer coding network provided by an embodiment of the present invention includes steps S1 to S6 as described below, and the method can be performed as follows Figure 2 The overall framework diagram shown is used to further illustrate the detection process.

[0045] S1. Obtain all texts to be detected and preprocess the texts.

[0046] In the text preprocessing stage, firstly, all the texts to be detected are obtained. In this embodiment, the number of the texts to be detected is two, namely Figure 2 postA and postB shown.

[0047] Two texts to be tested are extracted from the public rumor dataset for preprocessing: punctuation and special characters are removed, and all characters are converted to lowercase to obtain the preprocessed text for subsequent word embedding modeling.

[0048] S2. Embed the word fragments with the tagged vocabulary into the preprocessed text, convert the words in the text into token tagged words, and then perform vector encoding to obtain the word vector corresponding to each text.

[0049] Use the tokenizer to tokenize each sentence, and further segment the English vocabulary into finer-grained semantic units (word fragments) by querying the vocabulary table. For example, split playing into "play" and "##ing".

[0050] Perform vector encoding to obtain the word vector corresponding to each text. in is the i-th word vector in a tweet, n is the data length of the text, and the word vector is one-dimensional.

[0051] S3. Process all word vectors to obtain input vectors.

[0052] A [CLS] token-level flag is added to the head of the first word vector to indicate classification, which can extract the relationship information between sentences. Then, the two word vectors are separated by adding a [SEP] token-level flag to distinguish the two texts, and a [SEP] marker is added to the end of the last word vector.

[0053] Merge the two one-dimensional word vectors horizontally and then combine them with the segment vector (E A 、E B , where A and B represent two different text word vectors), position vector The input of the next multi-layer encoding network is obtained by vertically splicing bit by bit, that is, the input vector E = E CLS , E1, ..., E N , E SEP , E′1,...,E′ M , E′ SEP , where E i (i∈[1, N]) is the i-th word vector of the first text sequence input, E′ j (j∈[1,M]) is the jth word vector of the second text sequence input; i For example, E i By word vector Segment vector E A and the position vector Vertically spliced.

[0054] The values ​​of segment embeddings are automatically learned during the training process, which can represent the global semantic information of Twitter text and integrate the semantic information of context words.

[0055] Position embeddings represent the position information of words. Different vectors are attached to words in different positions to distinguish them and avoid differences in semantic information.

[0056] It should be noted that the sentence used as input to the subsequent multi-layer encoding network can be any span of continuous text, so a text sequence can also be regarded as a sentence.

[0057] S4. Input the input vector into the pre-trained multi-layer encoding network to generate an output vector.

[0058] Based on the NSP (Next Sentence Prediction) pre-training task, a multi-layer encoding network is trained to extract the relationship representation between texts. After fine-tuning, the following feature extraction task between different texts can be completed. Usually, a twelve-layer encoding network is used to have a better ability to understand the semantic connection between sentences.

[0059] The concatenated input vector E is sent to the pre-trained multi-layer encoding network, and the high-level network is used to learn the rich semantic information of the input sentence, and finally a set of high-dimensional vector representations H that integrate the semantic information of the context text is generated, that is, the output vector.

[0060] The input vector E is sent to the first layer encoding network, and the self-attention outputs of multiple different single heads are concatenated into a vector and output with reduced dimension. Then, a residual connection is made with the input, and after layer normalization and two linear transformations, the first output vector with the same length as the input vector is obtained.

[0061] The concatenated input vector E is sent to the first layer encoding network (i.e. Figure 2 Layer 1) shown in the figure extracts the word vector weight through the attention mechanism. The calculation process is as follows:

[0062] Q=EW Q

[0063] K=EW K

[0064] V=EW V

[0065] Among them, W Q, W K , W V They are the weight matrices of Q, K, and V generated by random initialization.

[0066] The word vector weights are weighted and summed through the softmax function to obtain the output head of the self-attention network:

[0067]

[0068] The outputs of n self-attention networks are concatenated using a matrix W O Perform a linear transformation on it and get the output E attention :

[0069] r i =concat(head1,...,head n )W O

[0070] E attention =r1, r2, r3, ..., r M+N+1

[0071] Then it is residually connected with the input vector, and transformed into data with a mean of 0 and a variance of 1 through layer normalization. Then, the output of the first layer encoding network is obtained through two linear transformations, that is, the first output vector is obtained. The calculation process is as follows:

[0072] E′ attention =E+E attention

[0073]

[0074] H1=ReLU(E Layer W1+b1)W2+b2

[0075] Among them, μ L and They represent the mean and variance of a layer of input, α and β represent scaling and translation variables, respectively. ∈ generally defaults to 1×10 -5 , represents a certain value added to the denominator to ensure numerical stability; the weight parameter of the first linear transformation is W1, the bias parameter is b1, the weight parameter of the second linear transformation is W2, the bias parameter is b2, and the ReLU activation function is used for nonlinear mapping; the weight size is related to the input and output dimensions.

[0076] The first output vector is sent to the next layer of encoding network with the same structure to repeat the above steps until the last layer of encoding network to obtain the final output vector, which is composed of a flag representation vector representing the output of the corresponding position of the flag and each text representation vector representing the output of the corresponding position of each text.

[0077] The first output vector is sent to the next multi-layer encoding network that is the same as the first layer (the subsequent encoding networks are respectively composed of Figure 2 The high-level encoding network learns the semantic level information of the context text sentences and obtains the high-dimensional vector representation H = H0, H1, ..., H N , ..., H N+2 , ..., H M+N+1 ..., where H0 represents the output of the position corresponding to the [CLS] tag, (H1, ..., H N ) and (H N+2 , ..., H M+N+1 ) represent the output of the corresponding positions of the first and second texts respectively.

[0078] Compared with other words in the text, the [CLS] representation output vector without obvious semantic information integrates the semantic information of each word in the text most fairly, so its corresponding final hidden state H0 is used as the aggregate sequence representation for the sentence pair relationship extraction task.

[0079] S5. Process the output vector to obtain a hidden state vector.

[0080] The flag representation vector is activated and sent to the fully connected layer. The flag representation vector is the [CLS] representation vector H0. Each text representation vector is added and averaged and then activated and sent to the corresponding fully connected layer. Each text representation vector is (H1, ..., H N ) and (H N+2 , ..., H M+N+1 ), and three primary hidden state vectors are obtained, namely, three vectors H′0, H′1 and H′2 containing text semantic information and expressions of the relationship between texts. The primary hidden state vectors are concatenated to obtain a hidden state vector, X = (H′0, H′1, H′2).

[0081] The calculation process of the three primary hidden state vectors is as follows:

[0082] H′0=W′0(tanh(H0))+b′0

[0083]

[0084]

[0085] Among them, N, N+2, and M+N+1 represent the last token position of the first text, the first token position of the second text, and the last token position of the second text respectively. The weight coefficient and bias term are shared parameters: W′0=W′1=W′2, b′0=b′1=b′2.

[0086] S6. Send the hidden state vector to the hidden layer and the classifier to obtain the probability of the text to be detected as each rumor category, and the category with the highest probability is the detection result of the text.

[0087] The hidden state vector X containing the semantic and relational information extracted from the rumor dataset is input into the hidden layer, processed by the nonlinear activation function of the hidden layer and outputs the hidden output vector Z:

[0088] Z=tanh(XW1+b1)

[0089] Among them, tanh is the nonlinear activation function of the hidden layer, W1 is the connection weight between the upper layer input and the hidden layer, the size is (d, h), d rows and h columns, d is the number of inputs, h is the number of hidden layer neural units, each column corresponds to the connection weight of each neuron in the hidden layer, b h is the h-dimensional vector bias.

[0090] Send the output of the hidden layer Z=z1,z2,z3,...,zd to the softmax regression classifier to get the classification result:

[0091] O = softmax(ZW o +b o )

[0092] Where Z is the hidden output vector, W o is the connection weight between the hidden layer and the classifier quality inspection, the size is (h, c), h rows and c columns, c is the number of output categories, b o is the c-dimensional vector bias; the output O represents the probability of the two input texts being detected as each rumor category, that is, the probability of each text message being divided into true rumor (True rumor, T), false rumor (False rumor, F) and unverified rumor (Unverified rumor, U); the category with the highest probability is the category to which the text belongs, which is the final detection result.

[0093] Example 2

[0094] The embodiment of the present invention provides a rumor detection system based on a multi-layer coding network, comprising:

[0095] Text preprocessing module: used to obtain all texts to be detected and preprocess the texts;

[0096] Vector encoding module: used to embed word fragments with a tagged vocabulary into the preprocessed text, convert the words in the text into token tagged words, and then perform vector encoding to obtain the word vector corresponding to each text;

[0097] Input vector acquisition module: used to process all word vectors to obtain input vectors;

[0098] Semantic learning module: used to input the input vector into the pre-trained multi-layer encoding network to generate the output vector;

[0099] Hidden state vector acquisition module: used to process the output vector to obtain the hidden state vector;

[0100] Detection module: used to send the hidden state vector into the hidden layer and classifier to obtain the probability of the text to be detected as each rumor category. The category with the highest probability is the detection result of the text.

[0101] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A rumor detection method based on a multi-layer coding network, characterized in that: include: Get all the texts to be detected and preprocess the texts; Embed the word fragments with the tagged vocabulary into the preprocessed text, convert the words in the text into token tagged words, and then perform vector encoding to obtain the word vector corresponding to each text; Process all word vectors to get the input vector; Input the input vector into the pre-trained multi-layer encoding network to generate the output vector; Process the output vector to obtain the hidden state vector; The hidden state vector is sent to the hidden layer and the classifier to obtain the probability of the text to be detected as each rumor category. The category with the highest probability is the detection result of the text. Input the input vector into the pre-trained multi-layer encoding network to generate the output vector, including the following steps: The input vector is sent to the encoding network. Each layer of the encoding network contains an attention mechanism and a layer normalization network. Multiple self-attention network outputs are obtained and concatenated into a vector and output with reduced dimension. Then, the vector is residually connected with the input vector, and after layer normalization and two linear transformations, the first output vector with the same length as the input vector is obtained. The first output vector is sent to the next layer of encoding network with the same structure to repeat the above steps until the last layer of encoding network, and the final output vector is obtained, which is composed of the flag bit representation vector representing the output of the corresponding position of the flag bit and the text representation vector representing the output of the corresponding position of each text; Methods for processing the output vector to obtain the hidden state vector include: The flag representation vector is activated and sent to the fully connected layer. Each text representation vector is added and averaged, and then activated and sent to the corresponding fully connected layer to obtain multiple primary hidden state vectors. The primary hidden state vectors are concatenated to obtain the hidden state vector.

2. A rumor detection method based on a multi-layer coding network according to claim 1, characterized in that: The number of the texts to be detected is two.

3. The rumor detection method based on a multi-layer coding network according to claim 1 is characterized in that: Methods for preprocessing text include: Punctuation marks and special characters are removed from the text to be detected, and all characters are converted to lowercase to obtain the preprocessed text.

4. The rumor detection method based on a multi-layer coding network according to claim 1 is characterized in that: The methods for processing all word vectors to obtain input vectors include: A flag is added between the header of the first word vector and the two word vectors, multiple word vectors are separated and then merged, and then they are vertically spliced ​​with the position vector and the segment vectors corresponding to each word vector bit by bit to obtain the input vector.

5. The rumor detection method based on a multi-layer coding network according to claim 1 is characterized in that: The hidden state vector is sent to the hidden layer and the classifier to obtain the probability of the text to be detected as each rumor category, including the following steps: The hidden state vector is sent to the hidden layer, processed by the nonlinear activation function of the hidden layer, and the hidden output vector is output; The hidden output vector is fed into the softmax regression classifier to obtain the probability that the text to be detected is detected as each rumor category: ； in, Z is the hidden output vector, is the connection weight between the hidden layer and the classifier quality inspection, yes c dimensional vector bias.

6. The rumor detection method based on a multi-layer coding network according to claim 1 is characterized in that: The rumor categories include true rumors, false rumors, and rumors whose authenticity has not been verified.

7. The rumor detection method based on a multi-layer coding network according to claim 1 is characterized in that: The tag vocabulary is established by the Subword algorithm.

8. A rumor detection system based on a multi-layer coding network, characterized in that: include: Text preprocessing module: used to obtain all texts to be detected and preprocess the texts; Vector encoding module: used to embed word fragments with a tagged vocabulary into the preprocessed text, convert the words in the text into token tagged words, and then perform vector encoding to obtain the word vector corresponding to each text; Input vector acquisition module: used to process all word vectors to obtain input vectors; Semantic learning module: used to input the input vector into the pre-trained multi-layer encoding network to generate the output vector; Hidden state vector acquisition module: used to process the output vector to obtain the hidden state vector; Detection module: used to send the hidden state vector into the hidden layer and classifier to obtain the probability of the text to be detected as each rumor category. The category with the highest probability is the detection result of the text; The input vector is input into a pre-trained multi-layer encoding network to generate an output vector, including the following steps: The input vector is sent to the encoding network. Each layer of the encoding network contains an attention mechanism and a layer normalization network. Multiple self-attention network outputs are obtained and concatenated into a vector and output with reduced dimension. Then, the vector is residually connected with the input vector, and after layer normalization and two linear transformations, the first output vector with the same length as the input vector is obtained. The first output vector is sent to the next layer of encoding network with the same structure to repeat the above steps until the last layer of encoding network, and the final output vector is obtained, which is composed of the flag bit representation vector representing the output of the corresponding position of the flag bit and the text representation vector representing the output of the corresponding position of each text; Methods for processing the output vector to obtain the hidden state vector include: The flag representation vector is activated and sent to the fully connected layer. Each text representation vector is added and averaged, and then activated and sent to the corresponding fully connected layer to obtain multiple primary hidden state vectors. The primary hidden state vectors are concatenated to obtain the hidden state vector.

Images