A Multimodal Attention-Based Rumor Detection Method Based on Causal Graph

Abstract

A multimodal attention-based rumor detection method based on causal graphs is disclosed, relating to the field of multimodal rumor detection based on causal graphs. The method includes the following steps: (1) extracting causal relationships contained in rumor data at the word level with fine granularity, and constructing a causal graph for the rumor based on this; (2) extracting node features and structural features of the causal graph through a two-layer graph convolutional network, and obtaining its embedding representation through downsampling; (3) extracting features from text and images respectively, and fusing these features with the embedding representation of the causal graph; (4) inputting the causally enhanced multimodal features into a rumor classification network with a self-attention mechanism to obtain the rumor detection result. This method effectively utilizes the causal semantics contained in rumor data and enhances the causal nature of multimodal features, achieving excellent results in rumor detection tasks.

Description

Technical Field

[0001] This invention relates to the field of multimodal rumor detection based on causal graphs, and in particular to a multimodal attention-based rumor detection method based on causal graphs. Background Technology

[0002] In the information-saturated new media era, the rapid dissemination and sharing of various messages through media has become a prevailing trend. Due to the low barriers to entry and openness of new media platforms, these platforms have also gradually become breeding grounds for rumors. Rumors spread rapidly and widely within these platforms, misleading users, causing social panic, and other negative impacts, posing a significant threat to society. Therefore, the detection and control of rumors has become a crucial issue.

[0003] Chinese patent application CN202211340799.X discloses a rumor detection method based on dynamic heterogeneous graphs and multi-level attention, including the following steps: Step A: Collect posts from social media platforms, extract the text content, comments, and related user information from the posts, and label the posts with authenticity tags to construct a training dataset DE; Step B: Use the training dataset DE to train a deep learning network model based on dynamic heterogeneous graphs and multi-level attention, and use this model to analyze the authenticity of posts and predict their authenticity tags; Step C: Input the text content, comments, and related user information of the posts to be detected into the trained deep learning network model to obtain the posts' authenticity tags. CN202211322158.1 discloses a rumor detection method based on topic visualization, comprising: cleaning the acquired data; visualizing the comment topology network in two dimensions, mapping each comment node to each pixel in the image; establishing a sentiment interaction function based on user internal and external factors, constructing a sentiment interaction model based on evolutionary game theory, and quantifying comment sentiment; embedding the quantified sentiment into the corresponding pixels in the topic image, convolving the two-dimensional topic image using a convolutional neural network, and using a function to convert the network output into probability values ​​for two categories, with the category corresponding to the highest probability value being used as the prediction result. Many current rumor detection methods typically only extract and utilize the semantic information of the text, such as semantic features and sentiment features, while ignoring the linguistic logic and causal relationships inherent in the rumor text. Methods based on causal relationships have been shown to have a positive impact on improving the robustness and verifiability of models in various natural language processing tasks. Summary of the Invention

[0004] The purpose of this invention is to fully utilize the causal relationships, visual information, and textual information of rumor data to provide a multimodal attention-based rumor detection method based on causal graphs, thereby achieving multimodal rumor detection with stronger robustness, better stability, and higher accuracy in rumor detection tasks.

[0005] This invention includes the following steps:

[0006] 1) Extract the causal relationships contained in the rumor data at the word level with fine granularity, and use this as a basis to construct a causal graph for the rumors;

[0007] 2) Extract node and structural features of the causal graph through a two-layer graph convolutional network, and obtain the embedding representation of the causal graph through downsampling;

[0008] 3) Extract the features from the text and images respectively, and fuse them with the embedding representation of the causal graph;

[0009] 4) Input the causal-enhanced multimodal features into a rumor classification network with a self-attention mechanism to obtain rumor detection results.

[0010] In step 1), the specific method for extracting the causal relationships contained in the rumor data at the word level with fine granularity and constructing a causal graph for the rumors based on this is as follows: First, extract all the rumor data; then, using a causal discovery algorithm, extract the causal relationships between entity nodes and entities in the rumor data; and construct a causal graph based on this. Where E represents the set of all entity nodes in the graph, and R is the set of causal relationships between nodes.

[0011] In step 2), the specific method for extracting node features and structural features of the causal graph through a two-layer graph convolutional network and obtaining the embedding representation of the causal graph through downsampling is as follows:

[0012] For causal graphs Each entity node E in i The GloVe algorithm is used to obtain its word vector representation. Form a set of node feature vectors Based on the set R of causal relationships between nodes, the adjacency matrix of the nodes can be constructed. This results in an effective representation of the graph's structure; a two-layer graph convolutional network is used to learn the node features and structural features of the graph respectively; finally, a downsampling module is used to obtain embedded graph features.

[0013] In step 3), the specific method for extracting the features of text and images respectively and fusing them with the embedded representation of the causal graph is as follows: For each piece of rumor data ε i Text feature extractor Obtaining the feature representation of text Regarding the visual aspects of rumors The VGG-16 model is used to extract visual features; by fusing multimodal information with causal graphs, causal-enhanced multimodal semantics are formed.

[0014] In step 4), the specific method for inputting the causally enhanced multimodal features into a rumor classification network with a self-attention mechanism to obtain the rumor detection result is as follows: The causally enhanced multimodal semantic information MF... c The model obtains attention weights v through a self-attention layer, and then uses these attention weights to make the model focus more on valuable features. Finally, a fully connected layer is used to classify the data, thereby obtaining the rumor detection results.

[0015] Compared with the prior art, the advantages of the present invention are as follows:

[0016] This invention achieves excellent results in multimodal rumor detection tasks, with improved robustness and stability, and has profound practical significance for rumor detection. This invention fully utilizes the causal relationships, visual information, and textual information in rumor data to achieve multimodal rumor detection, resulting in stronger robustness, better stability, and higher accuracy in rumor detection tasks. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the multimodal attention-based rumor detection method based on causal graphs according to an embodiment of the present invention.

[0018] Figure 2 for Figure 1 The diagram contains a causal graph. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 and 2 As shown, the embodiments of the present invention include the following steps:

[0021] 1) Extract the causal relationships contained in the rumor dataset based on the causal discovery algorithm, and construct a causal graph for the rumors based on this;

[0022] (1.1) Use the GES causal discovery algorithm to extract the causal relationships between entity nodes and entities in the Pheme rumor dataset;

[0023] (1.2) Construct a causal graph of the dataset based on the set of entity nodes E and the set of causal relationships R.

[0024] 2) Extract node and structural features of the graph through a two-layer graph convolutional network, and obtain its embedding representation through downsampling;

[0025] (2.1) Use the GloVe algorithm to obtain the feature vector set of entity nodes Obtain the adjacency matrix between nodes based on the set of causal relationships.

[0026] (2.2) The set of node feature vectors and the adjacency matrix of the graph are used as input, and the embedding representation of the graph is obtained through a two-layer graph convolutional network.

[0027] 3) Text features are extracted using the pre-trained language model BERT, and image features are extracted using the VGG16 network. The information from the two modalities is then fused with the embedding representation of the causal graph.

[0028] (3.1) Using the pre-trained language model BERT to extract rumors ε i Text features The VGG16 network was used to extract its image features.

[0029] (3.2) By fusing the information from the two modalities with the embedding form of the causal graph, causal-enhanced multimodal semantics are obtained.

[0030] 4) Input the causal-enhanced multimodal features into a rumor classification network with a self-attention mechanism to obtain rumor detection results.

[0031] (4.1) Input the causal-enhanced multimodal semantics into the attention layer to obtain the attention weight v, and use this weight to make the model focus on more valuable feature information;

[0032] (4.2) The classification result of the model is obtained through a fully connected layer. This enables the classification and detection of rumors.

[0033] The accuracy, precision, recall, and F1 score of this invention are used to evaluate its practical performance in multimodal rumor detection. The implementation results are shown in Table 1 on the English dataset Phae and the Chinese dataset Weibo.

[0034] Table 1 Implementation Results

[0035]

[0036] As shown in Table 1, our method achieves accuracies of 92.81% and 91.32% on the two datasets, respectively. Compared to existing multimodal models, our method improves model performance by enhancing the causal semantics of multimodal features. This demonstrates the effectiveness of causal graphs in rumor detection tasks.

Claims

1. A multimodal attention-based rumor detection method based on causal graphs, characterized in that... Includes the following steps: 1) Extract causal relationships from rumor data at the word level with fine granularity, and construct a causal graph for the rumors based on this. The specific method is as follows: First, for each piece of rumor data, use a causal discovery algorithm to extract the causal relationships between entity nodes and entities in the rumor data, and construct a causal graph based on this. Where E represents the set of all entity nodes in the graph, and R is the set of causal relationships between nodes; 2) Extract node and structural features of the causal graph using a two-layer graph convolutional network, and obtain the embedding representation of the causal graph through downsampling. The specific method is as follows: For causal graphs Each entity node E in i The GloVe algorithm is used to obtain its word vector representation. Form a set of node feature vectors Construct the adjacency matrix of the nodes based on the set R of causal relationships between them. This is used to characterize the graph structure; a two-layer graph convolutional network is used to learn the node features and structural features of the graph respectively, and finally a downsampling module is used to obtain embedded graph features. 3) Extract the textual and image features of the rumors separately, and fuse them with the embedding representation of the causal graph. The specific method is as follows: for each rumor data ε i The text feature extractor f is used. τ Obtaining the feature representation of text Regarding the visual aspects of rumors The VGG-16 model is used to extract visual features; by fusing multimodal features with causal maps, causal-enhanced multimodal semantics are formed. 4) Input the causally enhanced multimodal features into a rumor classification network with a self-attention mechanism to obtain rumor detection results. The specific method is as follows: input the causally enhanced multimodal semantic information... The model obtains attention weights v through a self-attention layer, and then uses these attention weights to make the model focus more on valuable features. Finally, a fully connected layer is used to classify the data, thereby obtaining the rumor detection results.

Images