Credible rumor detection method based on uncertainty quantification from a Bayesian perspective

Abstract

The present application relates to a credible rumor detection method based on uncertainty quantification from a Bayesian perspective. The method first proposes to use the Bayesian neural network framework to realize the quantitative analysis of the uncertainty of the detection model and data, breaking through the limitations of traditional methods while improving the reliability of the prediction results; secondly, a Bayesian neural network based on the BERT model and Transformer architecture is constructed, which enhances the model's semantic parsing ability for text data, laying the foundation for improving the performance of downstream tasks such as text classification; finally, variational reasoning technology is used to optimize the calculation process, effectively alleviating the problem of low computational efficiency caused by the surge in neural network parameters after Bayesianization. The experimental results and analysis on two conventional data sets and two difficult data sets show that this method has achieved good performance, which can provide a technical reference for uncertainty quantification analysis in similar work such as rumor detection, and has a certain degree of versatility.

Description

Technical Field

[0001] The present application relates to the technical field of rumor detection in big data analysis, and in particular to a credible rumor detection method based on uncertainty quantification from a Bayesian perspective. Background Art

[0002] Information disseminated on social platforms often has the following elements: (1) News content: Information content is a multimodal information that includes text, pictures, videos, and other forms of content; (2) Publishers and commentators: These two groups constitute the dissemination network of social information, so this element can also be regarded as a dissemination network and dissemination rules. The user who first publishes the content on social media is called the publisher, and the user group that forwards or directly comments or likes the content is called the commentator. Generally, the historical content published by platform users can be traced back through social platforms, so the content or comments published by the publisher or commentator in the past can be seen, and the credibility of the user can be evaluated based on this; (3) External knowledge: External knowledge includes the time of publication of the content, other relevant information before and after the publication time, and a wide range of objective fact bases. Not only researchers, but most people also make basic judgments on news content based on external knowledge.

[0003] Based on the above description, it can be inferred that news on social platforms is usually a fusion of multi-dimensional data information. Therefore, most of the rumor detection methods today can be roughly classified into: (1) Rumor detection methods based on unimodality: This method only uses unimodal input information for feature extraction and true and false information identification. Since text-based news accounts for the vast majority, early researchers hoped to rely solely on the news content features of the text for true and false identification. Therefore, most unimodal detection work is based on text information; (2) Rumor detection methods based on multimodality: It is insufficient to use only unimodal information such as text content. With the deepening of research on text-based unimodal rumor detection, scholars have gradually realized that relying solely on text analysis has significant limitations in improving model performance. The composition of news content is far more than text. It usually contains rich multimedia elements, such as images and videos. These elements provide more intuitive and vivid support for the dissemination of information. In the process of rumor dissemination, the makers also usually use these multimedia elements to enhance the credibility and attractiveness of the information. In view of this, researchers began to explore and introduce multimodal rumor detection methods to enhance the credibility and attractiveness of information. Multimodal detection methods can be seen as an improvement on single-modal detection methods. Researchers use features such as text, images, and videos to perform detection and judgment. These methods aim to explore the hidden associations between the various modes and explore the unique patterns of rumors based on these associations. For some tasks, this method can improve the model, but generalization cannot be guaranteed. In addition, as the composition of news becomes more and more diverse, researchers will also choose to introduce other modal information for further research, such as the dissemination network of news and the credibility of news spokespersons. By leveraging the different focus points between different modalities, the rumor detection method based on multimodality can integrate the advantages of each modal feature well and can obtain better detection performance than the rumor detection method based on single modality.

[0004] When citing information from various modalities, they can be classified based on whether external information is cited: (1) Rumor detection method based on news content features: This method only relies on the news content itself and does not require external information to supplement it. Rumors are detected by extracting features from the input news content; (2) Evidence-based rumor detection method: In contrast, this method not only uses the news content itself, but also introduces relevant external information (such as evidence) for auxiliary detection. The authenticity of the news is usually judged by analyzing the correlation between external information and news content. This method searches for highly correlated terms in external information based on keywords extracted from the news, and uses these terms to assist in judging whether the content of the news text is true.

[0005] Although deep learning has been widely used in many fields, it still has limitations. When the training data is sparse, the model often overfits and becomes "overconfident" in its own output. This phenomenon usually manifests itself as the model not knowing when it will make a mistake and not considering the uncertainty issues related to the data and the model itself. In the field of high-risk decision-making, quantifying uncertainty is very important for optimizing deep learning models. In order to achieve this goal, Bayesian neural networks have become increasingly popular in recent years.

[0006] Bayesian neural networks use probability theory techniques to extract knowledge from the provided data set. They provide posterior probabilities by combining the prior information of "expert knowledge" and the likelihood value of the data, thereby inferring the unknown parameters of the model while processing the uncertainty parameters of the model. Compared with traditional neural networks, the core of Bayesian neural networks lies in the inference of the prior distribution and posterior distribution of model parameters. The prior distribution of parameters usually adopts Gaussian distribution or other continuous distributions. Through Bayesian reasoning, it can be updated to the posterior distribution, and through a continuous iterative process, the posterior distribution gradually fits the real distribution, thereby reflecting the constraints of the data on the parameter weights. This method of using probability distribution to represent model parameters naturally provides a way to calculate uncertainty. Generally speaking, Bayesian neural networks can handle two types of uncertainty: (1) Aleatoric uncertainty refers to the difference in results caused by the nature and noise of the data; (2) Epistemic uncertainty refers to the difference caused by the model structure and model parameters. Since aleatoric uncertainty comes from the data, this uncertainty cannot be reduced, while epistemic uncertainty can be effectively controlled by Bayesian methods, while also avoiding overfitting.

[0007] The posterior distribution of a Bayesian neural network is usually difficult to obtain through precise calculation. This is because the posterior distribution involved in the Bayesian neural network often has a complex high-dimensional structure, which is extremely difficult to calculate directly. Therefore, sampling or approximate methods are usually used to fit the posterior distribution.

[0008] In the field of rumor detection, judging the truth or falsity of information through associated evidence is a more credible means of causal reasoning. However, in actual operation, this method requires a huge associated knowledge base to list relevant evidence of the information being reviewed. Building this knowledge base requires high manpower and computing power, so it is difficult to promote further. Summary of the invention

[0009] In view of this, it is necessary to provide a credible rumor detection method based on uncertainty quantification from a Bayesian perspective to address the above technical issues.

[0010] A credible rumor detection method based on uncertainty quantification from a Bayesian perspective, the method comprising:

[0011] Acquire news text information, preprocess the news text information, and obtain target news text.

[0012] The target news text is processed using the BERT model to obtain the feature representation of the text.

[0013] The feature representation and position encoding of the text are input into the Bayesian Transformer layer to enhance the model's semantic analysis of the text data and obtain the first layer of decoding features; the Bayesian Transformer layer is obtained by Bayesianizing the Transformer decoder of the Transformer layer.

[0014] The first layer of decoded features is input into a decoding module composed of multiple stacked Transformer layers to obtain the final decoded features.

[0015] The final decoded features are processed through the output layer to obtain the rumor detection classification results.

[0016] In one embodiment, the target news text is processed using the BERT model to obtain feature representation of the text, including:

[0017] The target news text is input into the BERT model, and the word segmenter of the BERT model is used to convert the target news text into a numerical vector form that can be processed by a computer to obtain a vector representation in a high-dimensional space. The vector representation in the high-dimensional space is processed using the multi-layer bidirectional Transformer architecture of the BERT model, and the hidden state vector output by the last layer is used as the feature representation of the text.

[0018] In one embodiment, the Transformer layer is composed of 6 decoders in the Transformer architecture, the decoder includes a multi-head attention layer, a first regularization module, a feedforward neural network module and a second regularization module, wherein there is a residual connection module between the multi-head attention layer and the feedforward neural network module; the Bayesian processing method is to Bayesianize the feedforward neural network module in the first decoder.

[0019] In one embodiment, the Transformer layer is composed of 6 decoders in the Transformer architecture, the decoder includes a multi-head attention layer, a first regularization module, a feedforward neural network module and a second regularization module, wherein there is a residual connection module between the multi-head attention layer and the feedforward neural network module; the Bayesian processing method is to Bayesianize the multi-head attention layer in the first decoder.

[0020] In one embodiment, the first layer decoding features are input into a decoding module stacked by multiple Transformer layers to obtain the final decoding features, including:

[0021] Input the first layer of decoded features into the multi-head attention layer of the first decoder of the first Transformer layer, and the multi-head self-attention output is:

[0022] ;

[0023] ;

[0024] in, is the current decoder layer number, is the multi-head self-attention output after scaling, For the layer decodes feature maps, is the key vector The dimension of , , They are respectively mapping the decoded features to the Transformer decoder , , The mapping matrix, , , They are query vector, key vector and value vector respectively.

[0025] The multi-head self-attention output is applied with a residual connection operation and then input into the first regularization module of the first Transformer decoder to obtain the regularized output.

[0026] The regularized output is input to the feedforward neural network module of the first Transformer decoder. The output of the feedforward neural network module is processed by the second regularization module of the first Transformer decoder after applying the residual connection operation. The output of the first decoder of the first Transformer layer is obtained as follows:

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] in, For the The output of the decoder, The result of applying a residual connection operation to the output of the feed-forward network layer, is the output after regularization, To map the feature space of the GELU activation function to the weight matrix of the output space, To map the regularized output to the weight matrix of the intermediate feature space, represents the mapping matrix applied to the output of the residual connection operation, The result of applying residual connection operation to the multi-head self-attention output, is the regularization function.

[0032] The output of the first decoder of the first Transformer layer is used as the input of the second decoder of the first Transformer layer, and the output of the second decoder of the first Transformer layer is used as the input of the third decoder of the first Transformer layer, and so on, until the last decoder of the first Transformer layer, to obtain the decoded features of the first Transformer layer.

[0033] The decoded features of the first Transformer layer are used as the input of the second Transformer layer, and the decoded features output by the second Transformer layer are used as the input of the third Transformer layer, and so on, until the last Transformer layer, to obtain the final decoded features.

[0034] In one embodiment, the steps of Bayesian processing include: using probability distribution instead of point estimation to calculate the parameters of the prediction model, and using variational reasoning to obtain an approximate representation of the posterior distribution; by continuously optimizing the variational parameters, the variational distribution is made close to the true posterior distribution within a preset range, thereby providing a reasonable basis for subsequent probability inference.

[0035] The above-mentioned credible rumor detection method based on uncertainty quantification from a Bayesian perspective automatically evaluates the authenticity of information based on the input news text, aiming to improve the accuracy and efficiency of rumor detection. Firstly, it is proposed to use the Bayesian neural network framework to realize the uncertainty quantitative analysis of the detection model and data, breaking through the limitations of traditional methods and improving the reliability of the prediction results; secondly, a Bayesian neural network based on the BERT model and Transformer architecture is constructed, which enhances the model's semantic parsing ability of text data and lays the foundation for improving the performance of downstream tasks such as text classification; finally, the variational inference technology is used to optimize the calculation process, effectively alleviating the problem of low computational efficiency caused by the surge in neural network parameters after Bayesianization. The experimental results and analysis on two conventional data sets and two difficult data sets show that this method has achieved good performance, which can provide a technical reference for uncertainty quantitative analysis in similar work such as rumor detection, and has certain versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of a process of a credible rumor detection method based on uncertainty quantification from a Bayesian perspective in one embodiment;

[0037] Figure 2 A schematic diagram of the Transformer layer structure in another embodiment;

[0038] Figure 3 A diagram of a credible rumor detection framework based on uncertainty quantification from a Bayesian perspective in another embodiment;

[0039] Figure 4 Schematic diagram of the Bayesian Transformer layer structure in another embodiment, where (a) is a schematic diagram of the FFN structure, and (b) is a schematic diagram of the MHA structure;

[0040] Figure 5 Schematic diagrams of F1 Macro and F1 Micro indicators of various methods in an embodiment on the Snopes dataset, (a) is a schematic diagram of F1 Macro and F1 Micro indicators of method none on the Snopes dataset, (b) is a schematic diagram of F1 Macro and F1 Micro indicators of method FFN on the Snopes dataset, (c) is a schematic diagram of F1 Macro and F1 Micro indicators of method MHA on the Snopes dataset, and (d) is a schematic diagram of F1 Macro and F1 Micro indicators of method Only BERT on the Snopes dataset;

[0041] Figure 6Schematic diagrams of F1 Macro and F1 Micro indicators of various methods on the Politifact dataset in one embodiment, (a) is a schematic diagram of F1 Macro and F1 Micro indicators of method none on the Politifact dataset, (b) is a schematic diagram of F1 Macro and F1 Micro indicators of method FFN on the Politifact dataset, (c) is a schematic diagram of F1 Macro and F1 Micro indicators of method MHA on the Politifact dataset, and (d) is a schematic diagram of F1 Macro and F1 Micro indicators of method Only BERT on the Politifact dataset;

[0042] Figure 7 is the F1 Macro indicator of each method on the Snopes Hard dataset in one embodiment;

[0043] Figure 8 It is the F1 Macro indicator of each method on the Politifact Hard dataset in an embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The credible rumor detection method based on uncertainty quantification from a Bayesian perspective proposed in this application optimizes the output of the model by quantifying the uncertainty related to the model and the data to obtain a more convincing rumor detection method.

[0046] In one embodiment, Figure 1 A method for detecting credible rumors based on uncertainty quantification from a Bayesian perspective is provided, and the method comprises the following steps:

[0047] Step 100: Acquire news text information, pre-process the news text information, and obtain target news text.

[0048] Step 102: Process the target news text using the BERT model to obtain feature representation of the text.

[0049] Specifically, for rumor detection tasks, the input is often text-type data, so pre-trained language models can be used to better understand and represent the semantic information of the text. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language representation model based on Transformer. It captures deep semantic relationships in text through bidirectional training, thereby achieving remarkable results in various natural language processing tasks. The tokenizer of the BERT-base-uncased model is responsible for converting raw text into a numerical vector form that can be processed by computers. This conversion process involves mapping text sequences into vector representations in a high-dimensional space so that machine learning algorithms can analyze and process them. In order to ensure that the model can handle input sequences of different lengths, it is usually necessary to pad shorter sequences to achieve the same length as the longest sequence in the batch. Although the padding operation ensures the consistency of sequence length, this meaningless padding may have an adverse effect on the text classification performance of downstream tasks. Therefore, in order to optimize the performance of downstream classification tasks, this method chooses to use the hidden state vectors output by the last layer of the BERT model as the feature representation of the text. These vectors are calculated in the multi-layer bidirectional Transformer architecture of the model and can capture the comprehensive semantic information of the input sequence.

[0050] Step 104: Input the feature representation and position encoding of the text into the Bayesian Transformer layer to enhance the semantic parsing of the text data by the model and obtain the first layer of decoding features; the Bayesian Transformer layer is obtained by Bayesianizing the Transformer decoder. This method converts the model parameters into a posterior probability distribution through Bayesianization.

[0051] Specifically, the original Transformer model consists of an encoder and a decoder. This method follows the existing work and uses the decoder to complete the subsequent model construction work. The Transformer layer is composed of 6 Transformer decoders in the Transformer architecture. The Transformer layer structure is as follows: Figure 2 As shown, Figure 2 middle is the position code, is the word embedding layer weight, Represents the mapping matrix applied to the output of the residual connection operation.

[0052] Although the Transformer model performs well in many tasks, it fails to take into account the uncertainty associated with the model because it uses a fixed point estimation method. When there is insufficient training data, the model is prone to overfitting and poor generalization.

[0053] Bayesian neural networks use probability theory techniques to extract knowledge from the provided data set. They provide posterior probabilities by combining the prior information of "expert knowledge" and the likelihood value of the data, thereby inferring the unknown parameters of the model while processing the uncertainty parameters of the model. Compared with traditional neural networks, the core of Bayesian neural networks lies in the inference of the prior distribution and posterior distribution of model parameters. The prior distribution of parameters usually adopts Gaussian distribution or other continuous distributions. Through Bayesian reasoning, it can be updated to the posterior distribution, and through a continuous iterative process, the posterior distribution gradually fits the real distribution, thereby reflecting the constraints of the data on the parameter weights. This method of using probability distribution to represent model parameters naturally provides a way to calculate uncertainty. Generally speaking, Bayesian neural networks can handle two types of uncertainty: (1) Aleatoric uncertainty refers to the difference in results caused by the nature and noise of the data; (2) Epistemic uncertainty refers to the difference caused by the model structure and model parameters. Since aleatoric uncertainty comes from the data, this uncertainty cannot be reduced, while epistemic uncertainty can be effectively controlled by Bayesian methods, while also avoiding overfitting.

[0054] The posterior distribution of a Bayesian neural network is usually difficult to obtain through precise calculation. This is because the posterior distribution involved in the Bayesian neural network often has a complex high-dimensional structure, and direct calculation is extremely difficult. Therefore, sampling or approximate methods are usually used to fit the posterior distribution, where the sampling or approximate methods can be but are not limited to MCMC sampling, variational inference (VI), and Monte Carlo (MC) Dropout.

[0055] To model the uncertainty in the model parameters, the model parameters can be Processed into posterior probability distribution through Bayesian neural network ,in It represents the data set used for model training. It represents the posterior distribution of the model parameters after training:

[0056] (1)

[0057] Considering the complexity of the posterior probability distribution, achieving its exact solution is a very challenging task, so this paper uses variational inference to obtain an approximate representation of the posterior distribution. By continuously optimizing the variational parameters, the variational distribution can be made as close as possible to the true posterior distribution, thus providing a reasonable basis for subsequent probability inference. As shown below, Divergence measures the difference between two different distributions. In order to minimize the variational posterior With the true posteriori of Divergence to obtain the optimal variational parameter :

[0058] (2)

[0059] (3)

[0060] Above In the divergence derivation process, is the prior on the model parameters, is the likelihood value, is the model parameter So formula (2) is equivalent to minimizing the objective function shown in formula (4) :

[0061] (4)

[0062] This objective function is also called the evidence lower bound (ELBO). The first term is called complexity cost and is used for regularization, and the second term is called likelihood cost. To minimize the objective function , the distribution of model parameters can be learned by combining Monte Carlo sampling and back propagation, and the objective function can be obtained with less computational cost through sampling:

[0063] (5)

[0064] in, Represents the variational posterior The random sample samples, K is the total number of samples. In the setting of prior distribution, and All are set to diagonal Gaussian distribution, that is:

[0065] (6)

[0066] In addition, in order to avoid the problem of unstable training process, and Using the heavy parameter technique:

[0067] (7)

[0068] Under the above assumptions, the second term of formula (3) The divergence can be broken down into two parts: Divergence and biased Divergence , overall The divergence is the sum of the above two parts:

[0069] (8)

[0070] (9)

[0071] in, , are the mean square and standard deviation of the weights, , are the mean square and standard deviation of the bias, respectively.

[0072] Mean values ​​of model parameters and bias parameters and standard deviation The gradient calculation of can be done using the standard back propagation algorithm as follows, where :

[0073] (10)

[0074] Based on the above principles, the Transformer layer can be Bayesianized.

[0075] Step 106: Input the first layer decoding features into a decoding module stacked by multiple Transformer layers to obtain the final decoding features.

[0076] Step 108: Process the final decoded features through the output layer to obtain the rumor detection classification result.

[0077] Specifically, the overall framework of the Bayesian Transformer model for rumor detection based on textual input information is as follows: Figure 3 shown.

[0078] In the above-mentioned credible rumor detection method based on uncertainty quantification from a Bayesian perspective, the method automatically evaluates the authenticity of information based on the input news text, aiming to improve the accuracy and efficiency of rumor detection. Firstly, it is proposed to use the Bayesian neural network framework to realize the uncertainty quantitative analysis of the detection model and data, breaking through the limitations of traditional methods and improving the reliability of the prediction results; secondly, a Bayesian neural network based on the BERT model and Transformer architecture is constructed, which enhances the model's semantic parsing ability for text data and lays the foundation for improving the performance of downstream tasks such as text classification; finally, the variational inference technology is used to optimize the calculation process, effectively alleviating the problem of low computational efficiency caused by the surge in neural network parameters after Bayesianization. The experimental results and analysis on two conventional data sets and two difficult data sets show that this method has achieved good performance, which can provide a technical reference for uncertainty quantitative analysis in similar work such as rumor detection, and has certain versatility.

[0079] In one embodiment, step 102 includes: inputting the target news text into the BERT model, using the word segmenter of the BERT model to convert the target news text into a numerical vector form that can be processed by a computer, and obtaining a vector representation in a high-dimensional space; processing the vector representation in the high-dimensional space using the multi-layer bidirectional Transformer architecture of the BERT model, and using the hidden state vector output by the last layer as the feature representation of the text.

[0080] In one embodiment, the Transformer layer is composed of 6 decoders in the Transformer architecture, the decoder includes a multi-head attention layer, a first regularization module, a feedforward neural network module and a second regularization module, wherein there is a residual connection module between the multi-head attention layer and the feedforward neural network module; the Bayesian processing method is to Bayesianize the feedforward neural network module in the first decoder.

[0081] In one embodiment, the Transformer layer is composed of 6 decoders in the Transformer architecture, the decoder includes a multi-head attention layer, a first regularization module, a feedforward neural network module and a second regularization module, wherein there is a residual connection module between the multi-head attention layer and the feedforward neural network module; the Bayesian processing method is to Bayesianize the multi-head attention layer in the first decoder.

[0082] Specifically, since the Transformer layer consists of 6 Transformer decoders, and Xue's experiments have shown that Bayesianization in shallow Transformer layers will have better results, this method only Bayesianizes the decoder of the first layer. There are two methods for processing: partial Bayesianization of the feedforward processing network (FFN) and partial Bayesianization of the multi-head attention mechanism (MHA). The specific structure is as follows Figure 4 As shown, (a) is a schematic diagram of the FFN structure, (b) is a schematic diagram of the MHA structure, Figure 4 Parameters in (a) In the case of data set D and The probability distribution of Figure 4 Parameters in (b) In the case of data set D The probability distribution of Indicates that the parameter is Multi-head attention layer.

[0083] In one embodiment, step 106 includes: inputting the first layer decoded features into the multi-head attention layer of the first decoder of the first Transformer layer, and obtaining the multi-head self-attention output as:

[0084] (11)

[0085] (12)

[0086] in, is the current decoder layer number, is the multi-head self-attention output after scaling, For the Layer decoded feature map, as preferred , is the key vector The dimension of , , They are respectively mapping the decoded features to the Transformer decoder , , The mapping matrix, , , They are query vector, key vector and value vector respectively.

[0087] The multi-head self-attention output is applied with a residual connection operation and then input into the first regularization module of the first Transformer decoder to obtain the regularized output; the regularized output is input into the feedforward neural network module of the first Transformer decoder, and the output of the feedforward neural network module is applied with a residual connection operation and then processed by the second regularization module of the first Transformer decoder to obtain the output of the first decoder of the first Transformer layer:

[0088] (13)

[0089] (14)

[0090] (15)

[0091] (16)

[0092] in, For the The output of the decoder, The result of applying a residual connection operation to the output of the feed-forward network layer, is the output after regularization, To map the feature space of the GELU activation function to the weight matrix of the output space, To map the regularized output to the weight matrix of the intermediate feature space, represents the mapping matrix applied to the output of the residual connection operation, The result of applying residual connection operation to the multi-head self-attention output, is the regularization function;

[0093] The output of the first decoder of the first Transformer layer is used as the input of the second decoder of the first Transformer layer, and the output of the second decoder of the first Transformer layer is used as the input of the third decoder of the first Transformer layer, and so on, until the last decoder of the first Transformer layer, to obtain the decoding features of the first Transformer layer; similarly, the decoding features of the first Transformer layer are used as the input of the second Transformer layer, and the decoding features output by the second Transformer layer are used as the input of the third Transformer layer, and so on, until the last Transformer layer, to obtain the final decoding features.

[0094] Specifically, Figure 4As shown in the figure, there are encoding layers and position encoding layers before the input Transformer layer. This step allows the Transformer model to have a better understanding of the semantics and position of the text input. Each Transformer decoder has a multi-head attention module and a feedforward module, and there are layer regularization and residual connection modules between these two modules. is the input of the previous layer Transformer decoder, , , The input Mapped to this layer of Transformer decoder , , The mapping matrix, It represents the multi-head self-attention output after scaling. is the key vector The dimension of represents the mapping matrix applied to the output of the residual connection operation, represents the layer regularization operation, then the multi-head self-attention output after scaling is as shown in formula (11), the multi-head self-attention output The result after applying the residual connection operation is shown in formula (16), and the output after regularization is shown in formula (15).

[0095] Output after regularization Then it is input into the feedforward module. This method uses Gaussian error linear unit (GELU) as the activation function, and finally obtains the The output of the layer Transformer decoder is shown in formula (13).

[0096] In one embodiment, the steps of Bayesian processing include: using probability distribution instead of point estimation to calculate the parameters of the prediction model, and using variational reasoning to obtain an approximate representation of the posterior distribution; by continuously optimizing the variational parameters, the variational distribution is made close to the true posterior distribution within a preset range, thereby providing a reasonable basis for subsequent probability inference.

[0097] Specifically, several types of fitting posterior distribution methods are briefly introduced as follows:

[0098] (1) MCMC sampling

[0099] Bayesian neural networks differ from traditional neural networks in that they use probability distributions rather than fixed point estimates to represent the unknown parameters of the model, and therefore require a different optimization method for calculation, namely sampling.

[0100] Markov Chain Monte Carlo (MCMC) is a popular sampling method. It is an accurate sampling method that generates a sample chain that mimics the target distribution based on previously sampled samples. One of its major features is that it can transition from a given state to any state in the distribution through a finite step size, while avoiding the possibility of staying in the chain cycle for a long time. Common MCMC sampling methods include Metropolis Hasting algorithm, Gibbs sampling, Hamiltonian Monte Carlo, etc.

[0101] (2) Variational Inference (VI)

[0102] Compared with MCMC, variational inference is an efficient method for approximating the posterior distribution. For complex deep learning models and large datasets, MCMC sampling methods are computationally expensive and difficult to scale. Although VI does not have the same exact distribution, it performs well in estimating the target distribution and greatly reduces the number of sampling times, making it suitable for large-scale complex models and datasets. In addition, the KullbackLeibler (KL) divergence can be used to weigh the difference or similarity between the true posterior distribution and the variational posterior. The divergence formula is as follows:

[0103] (17)

[0104] In general, the goal of the variational model is to obtain a variational posterior distribution that is sufficiently similar to the true posterior distribution, which means that the distance between the variational posterior distribution and the true posterior distribution should be minimized as much as possible.

[0105] (3) Monte Carlo (MC) Dropout

[0106] In deep learning models, the Dropout method is used for regularization and to prevent the model from overfitting during training. During training, some neurons in a given layer are closed or discarded with a certain probability. MC dropout also discards neurons with a certain probability during the test phase to estimate the probability distribution of the target. It is similar to variational inference for estimating the posterior distribution. Compared with VI, MC dropout has fewer parameters and takes less time for the model to converge.

[0107] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0108] In a verification example, this example uses two data sets specifically for the news field. The news content of these data sets and their corresponding label data are mainly collected from two well-known fact-checking platforms, Politifact and Snopes. Snopes has only "true" and "false" labels, which provide directly available labels for the experiment. However, Politifact has six labels. In order to unify the label system, it is necessary to set the three labels "pants on fire", "mostly false", and "false" to "false", and the remaining labels to "true". Politifact has 13,581 data, the ratio of training set, verification set and test set is 6:2:2, and the ratio of "true" label to "false" label is 3:7. Snopes has 5,069 data, the ratio of training set, verification set and test set is 7:1:2, and the ratio of "true" label to "false" label is 3:7. In order to further enhance the challenge of the experiment, this paper also screened out samples that are difficult for the BERT model to correctly classify from the original dataset, forming two more challenging sub-datasets, Politifact Hard and Snopes Hard. Compared with the original dataset, the Hard dataset is more challenging. Politifact Hard has 1971 data, and the ratio of "true" labels to "false" labels is 4.5:5.5. Snopes Hard has 357 data, and the ratio of "true" labels to "false" labels is 3:7. The adjustable parameters of this experiment are shown in Table 1:

[0109] Table 1 Summary of adjustable parameters

[0110]

[0111] The above hyperparameter table requires additional supplementation and explanation: In order to prevent gradient explosion during gradient backpropagation and thus affect the training effect, the gradient clipping technique is introduced, and clip is the control parameter for controlling the clipping range; during the training and verification process, in order to speed up the training efficiency, the total number of samples K=1, and in the final test stage, K=100; the learning rate is initially set to 0.05. If the F1_Macro indicator of this round of training is not the historical optimal, the learning rate is set to half of the current one.

[0112] This experiment was conducted in Python 3.9, using PyTorch 2.0.0, on an NVIDIA RTX 4060 GPU:

[0113] In binary classification problems, the F1 score is a common performance evaluation indicator. This article uses two indicators, F1-Macro and F1-Micro, to evaluate the experimental results:

[0114] Precision: How many of the samples predicted to be positive are positive samples.

[0115] (18)

[0116] Recall: How many positive samples are correctly predicted.

[0117] (19)

[0118] The F1 score reflects the harmonic mean of the model's Precision and Recall, and can more reasonably reflect the model's performance when positive and negative samples are unevenly distributed:

[0119] (20)

[0120] In a binary classification scenario, F1-Micro is usually consistent with the results of Accuracy. Its calculation process is divided into the following two steps: (1) Calculate the total Precision and Recall of all categories;

[0121] (twenty one)

[0122] (twenty two)

[0123] (2) Calculate F1-Micro using the F1 formula:

[0124] (twenty three)

[0125] The calculation process of F1-Macro is slightly different from that of F1-Micro, but it is more suitable for scenarios with uneven data distribution: (1) Calculate the average of Precision and Recall for each category:

[0126] (twenty four)

[0127] (25)

[0128] in, is the precision of the i-th category, is the recall rate of the ith category.

[0129] (2) Calculate F1-Macro using the F1 formula:

[0130] (26)

[0131] In this experiment, considering the uneven distribution of positive and negative samples in the data set, the F1 Macro score is used as the most important indicator to evaluate the performance of a specific method on a specified data set. Figures 5 to 8 Tables 2 to 5 show the experimental results of different methods on four datasets. They include the dataset name (dataset), the dataset retention ratio (Mark), the method used (Method), and the F1 Micro and F1 Macro scores on the standard and difficult test sets. The Dataset options are Snopes, Snopes Hard, Politifact, and Politifact Hard; the Mark options are 1, 0.5, 0.25, 0.1, and 0.05; the Method options are none, FFN, MHA, and only BERT, which respectively represent the Transformer layer without Bayesianization, the Bayesianization of the feedforward network of the first layer of the Transformer, the Bayesianization of the multi-head attention of the first layer of the Transformer, and the output is directly obtained through the output layer after the input passes through the BERT model.

[0132] Table 2 Comparative experimental results of different methods on Snopes dataset

[0133]

[0134] Table 3 Comparative experimental results of different methods on the Politifact dataset

[0135]

[0136] Figure 5 The diagrams of F1 Macro and F1 Micro indicators of four methods on the Snopes dataset are presented, (a) is the diagram of F1 Macro and F1 Micro indicators of method none on the Snopes dataset, (b) is the diagram of F1 Macro and F1 Micro indicators of method FFN on the Snopes dataset, (c) is the diagram of F1Macro and F1 Micro indicators of method MHA on the Snopes dataset, and (d) is the diagram of F1 Macro and F1Micro indicators of method Only BERT on the Snopes dataset; Figure 6 The F1 Macro and F1 Micro indicators of four methods on the Politifact dataset are presented. (a) is the F1 Macro and F1 Micro indicator diagram of the method none on the Politifact dataset, (b) is the F1 Macro and F1 Micro indicator diagram of the method FFN on the Politifact dataset, (c) is the F1 Macro and F1 Micro indicator diagram of the method MHA on the Politifact dataset, and (d) is the F1 Macro and F1 Micro indicator diagram of the method Only BERT on the Politifact dataset. Figure 6 In (b), except for Mark=0.1, all F1 Macro curves decrease as the retention ratio of the data set decreases. In addition, the F1 Micro curves also generally follow this rule.

[0137] Figure 7 and Figure 8 It reflects the F1 Macro index presented on the two Hard data sets. In most cases, the Bayesian methods (FFN and MHA) can achieve higher indicators.

[0138] In addition to the F1 indicator, the introduction of Bayesian methods can also quantify the uncertainty associated with the model and data. Kwon et al. proposed a method to measure uncertainty by calculating the variance of the predicted distribution. Accidental uncertainty describes the uncertainty associated with the data set and is usually unrelated to the model itself. Epistemic uncertainty is related to model parameter uncertainty and reflects the uncertainty caused by imperfect models or insufficient data. Combining the Monte Carlo method with the Bayesian method, the variance of the model output is calculated. Accidental uncertainty and epistemic uncertainty The calculation formula is as follows:

[0139] (27)

[0140] (28)

[0141] In formula (27), Indicates that given parameters Under the condition of The variance calculated above is represents the expected value; in formula (28), The model is trained with given parameters and training set The epistemic uncertainty of FFN and MHA on the Snopes and Politifact datasets is summarized in Tables 4 and 5:

[0142] Table 4 Epistemic uncertainty of FFN and MHA on Snope

[0143]

[0144] Table 5 Epistemic uncertainty of FFN and MHA on Politifact

[0145]

[0146] In Table 4, the epistemic uncertainty of the MHA method is greater than that of FFN in all cases, indicating that the model uncertainty of the MHA model on the Snopes dataset is greater than that of FFN. Similarly, Table 5 proves that the model uncertainty of the FFN model on the Politifact dataset is greater than that of MHA. Based on the above conclusions, we can make the following judgments: the performance of the FFN model on the Snopes dataset will be better, and the performance of the MHA model on the Politifact dataset will be better, which is consistent with the rules presented in Tables 2 and 3.

[0147] The following conclusions can be drawn from the above experimental results:

[0148] (1) The introduction of the Transformer layer can improve the classification performance of the rumor detection task. Compared with the output results of the BERT model alone, the F1 Macro index after introducing the Transformer layer or the Bayesian Transformer layer is generally higher.

[0149] (2) On the original Snopes and Poliltifact datasets, it can be found that the F1Macro scores of the Bayesianization (FFN and MHA) are generally higher, and to a certain extent this reflects that as the training data continues to decrease, the Bayesianization Transformer layer performs better in maintaining the original performance.

[0150] (3) Based on epistemic uncertainty, we can compare the performance of different models on the same data set. The experiment introduces the Bayesian method to quantify uncertainty, and the experimental results show that the performance of the model is consistent with the epistemic uncertainty in terms of rules.

[0151] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A credible rumor detection method based on uncertainty quantification from a Bayesian perspective, characterized in that: The method comprises: Acquire news text information, and preprocess the news text information to obtain target news text; Processing the target news text using the BERT model to obtain feature representation of the text; The feature representation and position encoding of the text are input into the Bayesian Transformer layer to enhance the semantic analysis of the text data by the model, and obtain the first layer of decoding features; the Bayesian Transformer layer is obtained by Bayesianizing the Transformer decoder of the Transformer layer; the Transformer layer is composed of 6 decoders in the Transformer architecture, and the decoder includes a multi-head attention layer, a first regularization module, a feedforward neural network module and a second regularization module, wherein a residual connection module is provided between the multi-head attention layer and the feedforward neural network module; the Bayesian processing method is to Bayesianize the feedforward neural network module or the multi-head attention layer in the first decoder; Input the first layer of decoding features into a decoding module composed of multiple stacked Transformer layers to obtain the final decoding features; Processing the final decoded features through the output layer to obtain rumor detection classification results; The first layer of decoding features is input into a decoding module stacked by multiple Transformer layers to obtain the final decoding features, including: The first layer of decoded features is input into the multi-head attention layer of the first decoder of the first Transformer layer, and the multi-head self-attention output is obtained as: in, l is the current decoder layer number, is the multi-head self-attention output after scaling, For the l layer decodes feature maps, is the key vector The dimension of , , They are respectively mapping the decoded features to the Transformer decoder , , The mapping matrix, , , They are query vector, key vector and value vector respectively; Applying a residual connection operation to the multi-head self-attention output and inputting it into the first regularization module of the first Transformer decoder to obtain a regularized output; The regularized output is input to the feedforward neural network module of the first Transformer decoder, and the output of the feedforward neural network module is processed by the second regularization module of the first Transformer decoder after applying the residual connection operation, so that the output of the first decoder of the first Transformer layer is obtained as follows: in, For the l The output of the decoder, The result of applying a residual connection operation to the output of the feed-forward network layer, is the output after regularization, To map the feature space of the GELU activation function to the weight matrix of the output space, To map the regularized output to the weight matrix of the intermediate feature space, represents the mapping matrix applied to the output of the residual connection operation, The result of applying residual connection operation to the multi-head self-attention output, is the regularization function; The output of the first decoder of the first Transformer layer is used as the input of the second decoder of the first Transformer layer, and the output of the second decoder of the first Transformer layer is used as the input of the third decoder of the first Transformer layer, and so on, until the last decoder of the first Transformer layer, to obtain the decoding features of the first Transformer layer; The decoded features of the first Transformer layer are used as the input of the second Transformer layer, and the decoded features output by the second Transformer layer are used as the input of the third Transformer layer, and so on, until the last Transformer layer, to obtain the final decoded features.

2. The method according to claim 1, characterized in that The target news text is processed using the BERT model to obtain feature representation of the text, including: The target news text is input into the BERT model, and the word segmenter of the BERT model is used to convert the target news text into a numerical vector form that can be processed by a computer to obtain a vector representation in a high-dimensional space; the vector representation in the high-dimensional space is processed using the multi-layer bidirectional Transformer architecture of the BERT model, and the hidden state vector output by the last layer is used as the feature representation of the text.

3. The method according to claim 1, characterized in that The steps of Bayesian processing include: using probability distribution instead of point estimation to calculate the parameters of the prediction model, using variational inference to obtain an approximate representation of the posterior distribution; by continuously optimizing the variational parameters, the variational distribution is made close to the true posterior distribution within a preset range, thereby providing a reasonable basis for subsequent probability inference.

Images