Multi-modal Rumor Detection Method and System with Enhanced Dynamic Propagation Features

Abstract

The present invention relates to a multimodal rumor detection method and system with enhanced dynamic propagation features. The method comprises the following steps: Step A: Collecting multimedia posts on social media, extracting the text, comments, images, and user profiles from the posts, and annotating the posts with authenticity labels to construct a training dataset DS; Step B: Using the training dataset DS to train a multimodal rumor detection model M with enhanced dynamic propagation features, using model M to analyze the authenticity of the posts and derive predicted labels for the posts; Step C: Inputting the text, comments, images, and user profiles of undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts. This method and system are beneficial for improving the accuracy and reliability of rumor detection on social media.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to a multimodal rumor detection method and system with enhanced dynamic propagation features. Background Art

[0002] Rumor detection, also known as fake news detection, is a key research task in the field of natural language processing (NLP). With the rise of social media, platforms like Weibo and Twitter have become indispensable communication tools. However, due to the rapid speed and wide reach of social media, rumors have gradually become a significant concern. On social media, rumors are unverified representations of information, often using false or fabricated images and inflammatory text to mislead users. The emergence of rumors often causes significant economic losses and serious harm to individuals and society. Although various social media organizations have launched rumor-refuting platforms to strengthen rumor control, these platforms still require significant manpower and resources to collect information to determine the authenticity of rumors. Therefore, there is an urgent need for an automated method that can effectively detect rumors.

[0003] Because deep learning methods offer significant performance advantages over traditional machine learning methods, a growing number of researchers are applying them to rumor detection. Deep neural networks, such as convolutional neural networks and recurrent neural networks, can automatically learn higher-quality, more representative data features. Convolutional neural networks can capture local semantic information from text, leading some researchers to apply them to text-based rumor detection. However, convolutional neural networks often fail to fully learn the contextual information within sentences, which is crucial for understanding the overall semantics of a text. Consequently, rumor detection models based on convolutional neural networks are limited in performance. To address this issue, some researchers have proposed using recurrent neural networks, such as long short-term memory (LSTM) and gated recurrent units (GRU), to extract contextual semantic information from text. Unlike convolutional neural networks, recurrent neural networks view sentences as a sequence of words, selecting each word in chronological order and using the output of one hidden layer as the input to the next hidden layer. They continuously learn contextual information from the sequence data, enabling the model to better understand the overall semantics of a sentence and improving rumor detection performance. Although rumor detection models based on recurrent neural networks have achieved certain performance breakthroughs, they still have some drawbacks. This method only uses text information to detect rumors and ignores the topological relationship of the rumor propagation process.

[0004] Therefore, some researchers have studied rumor detection methods based on text information and propagation graphs. By modeling the propagation process of a post and constructing a propagation graph, they use graph neural networks to aggregate information from neighboring nodes to capture the structural characteristics of rumor propagation. For example, Wu et al. constructed a propagation graph for each sample post and used a gated graph neural network to exchange information between adjacent nodes along the relationship path, updating the node representations and thus enhancing the node feature representation of the propagation graph. Ma et al. used top-down and bottom-up tree-structured recurrent neural networks to capture the semantic and propagation characteristics of rumors. Furthermore, some researchers have begun to explore constructing heterogeneous propagation graphs to provide a new research direction for simulating the rumor propagation process, achieving excellent performance. Huang et al. constructed a heterogeneous post-word-user graph and used a heterogeneous graph attention network based on relationship paths to capture the global semantic relationships of text and the global structural information of post propagation. Khoo et al. used a multi-head attention mechanism to model long-range interactions between different posts and proposed a post-level attention model. Ran et al. proposed a multi-channel graph attention network that models the relationship between source posts and comments, source posts and users, and source tweets and their words, constructing three heterogeneous subgraphs. They also designed an event sharing module to learn common features across different rumor events. Bian et al. used a bidirectional graph convolutional network to learn rumor propagation patterns and capture the structural characteristics of rumor diffusion. Yuan et al. modeled the global relationship between posts, reposts, and users into a heterogeneous graph, then captured local semantic information and global structural information from the propagation relationship between posts and users.

[0005] With the development of multimedia technology, information on social media has gradually become multimodal, containing multiple types of information such as text and images. Images have also gradually become an important component of rumors. They are often more likely to attract and mislead users than text, thus spreading rapidly. Therefore, rumor detection based on multimodality has become a new hot research direction. Khattar et al. proposed a multimodal variational autoencoder for rumor detection to learn shared representations of text and images. Zhang et al. designed a multimodal multi-task learning framework by introducing stance tasks. Zhou et al. considered the correlation between textual and visual information and identified rumors based on the mismatch between text and images.

[0006] While current research on rumor detection has achieved some success in terms of accuracy, challenges remain in learning the rumor propagation process and in multimodal fusion. Existing multimodal rumor detection models typically only consider the text and image information of a post, ignoring the rumor propagation process and patterns. Furthermore, when performing multimodal fusion, they simply concatenate features from different modalities without considering cross-modal relationships. Furthermore, existing rumor detection models based on propagation graphs, when using graph neural networks to aggregate node information in the propagation graph, are unable to distinguish the influence of adjacent nodes on the target node. These issues can lead to reduced rumor detection performance. Summary of the Invention

[0007] The purpose of the present invention is to provide a multimodal rumor detection method and system with enhanced dynamic propagation characteristics, which are conducive to improving the accuracy and reliability of rumor detection in social media.

[0008] To achieve the above objectives, the present invention adopts a technical solution: a multimodal rumor detection method with enhanced dynamic propagation features, comprising the following steps:

[0009] Step A: Collect multimedia posts from social media, extract text, comments, images, and user profiles from the posts, and annotate the posts with authenticity labels to construct a training dataset DS.

[0010] Step B: Use the training dataset DS to train a multimodal rumor detection model M with dynamic propagation feature enhancement to analyze the authenticity of the post and derive the predicted label of the post.

[0011] Step C: Input the text, comments, images and user profiles of the undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts.

[0012] Furthermore, the step B specifically includes the following steps:

[0013] Step B1: Initially encode all training samples in the training dataset DS to obtain the initial text representation vector E t , initial comment representation vector E r and the user's initial representation vector E u ;

[0014] Step B2: Initial text representation vector E t The images in the post are input into Text-CNN and ResNet50 respectively to obtain the post text feature representation vector F t and visual feature representation vector F v ;

[0015] Step B3: Model the users, source posts, comments and their corresponding interactive relationships in the post propagation process to construct a heterogeneous propagation graph G; input the heterogeneous propagation graph G into the multi-layer dynamic heterogeneous graph neural network DyHGN. DyHGN mainly includes a symbolic attention mechanism and a root node enhancement module. First, the symbolic attention mechanism is used to obtain the node n i The attention coefficient vector A of all adjacent nodes is then used as a weight to aggregate the information of all adjacent nodes, and the node n is obtained after being enhanced by the root node. i Information representation vector H (l) [n i ], and finally get node n after L layers of DyHGN i The dynamic propagation feature representation vector F g ;

[0016] Step B4: Represent the text feature vector F of the post t and visual feature representation vector F v Input into the joint attention network to obtain the joint representation vector F of the post text and vision tv ; Then the text feature representation vector F of the post t and dynamic propagation feature representation vector F g Input into the joint attention network to obtain the joint representation vector F of the post text and the spread graph tg ;

[0017] Step B5: Combine the post text and visual representation vector F obtained in step B4 tv and the joint representation vector F of the post text and the diffusion graph tg Perform modality alignment under self-supervisory loss to obtain the aligned text-visual joint representation vector and the joint representation vector of the text diffusion graph Then the text-visual joint representation vector after modality alignment is and text propagation graph jointly represent the vector Input into the joint attention network to obtain the multimodal representation vector F of the post tvg ;

[0018] Step B6: Transform the multimodal representation vector F of the post tvg The data is input into the fully connected layer to obtain the predicted result of the post. Then, based on the target loss function, the gradient of each parameter in the multimodal rumor detection model is calculated by backpropagation, and all parameters of the model are iteratively updated using backpropagation.

[0019] Step B7: When the loss value change generated by each iteration of the multimodal rumor detection model is less than a given threshold or reaches the maximum number of iterations, the model training process is terminated.

[0020] Furthermore, the step B1 specifically includes the following steps:

[0021] Step B11: Traverse the sample data of the training dataset DS, perform word segmentation on the text and comment content of the post, remove stop words and special symbols, and delete posts containing invalid images; each sample post s in DS i Indicated as s i =(t i , r i ,u i , v i , a); where t i , v i and u i Represent the text, image and publisher of the post, r i Indicates posts i The corresponding comment set, Indicates posts i The jth comment of All by the corresponding users Release; a is the authenticity label corresponding to the post, a∈{non-rumor, rumor};

[0022] The text of the post i Expressed as:

[0023]

[0024] in, For the post text content i The kth word in the post, k = 1, 2, ..., n, n is the post text t i The number of words;

[0025] Comments on the post Expressed as:

[0026]

[0027] in, For comment content The kth word in, i = 1, 2, ..., m, where m is a comment The number of words;

[0028] Step B12: Get the text from step B11 Encode through word2vec to get the text t i The initial characterization vector E t ;E t Expressed as:

[0029]

[0030] in, Represents the word vector matrix from pre-training The word embedding representation vector obtained by looking up in Represents the kth word The corresponding word embedding representation vector, d represents the dimension of the word embedding representation vector, |V| represents the number of words in the dictionary V;

[0031] Step B13: Comments on Step B11 Encode through word2vec to get comments The initial characterization vector E r ;E r Expressed as:

[0032]

[0033] in, Represents the word vector matrix from pre-training The word embedding representation vector obtained by looking up in Represents the kth word The corresponding word embedding representation vector, d represents the dimension of the word embedding representation vector, |V| is the number of words in the dictionary V;

[0034] Step B14: Use word2vec to encode the historical posts in the user profile, and then take the average of all encoded vectors to get the user u i The initial representation vector Where d is the dimension of the representation vector.

[0035] Furthermore, the step B2 specifically includes the following steps:

[0036] Step B21: Translate the post text into i The initial characterization vector E t Input into Text-CNN, use the convolution kernel of size h to perform convolution operation on the input matrix, and obtain the sequence feature representation c of h consecutive words starting from the jth word j , the calculation process is as follows:

[0037]

[0038] Among them, the convolution kernel size h∈{3, 4, 5}, W t represents the weight matrix of the filter, σ(·) represents the ReLU activation function;

[0039] Step B22: Then perform the maximum pooling operation on the result c after the convolution operation of all words in the text to obtain the representation vector after convolution. Finally, the pooling results of convolution kernels of different sizes are spliced to obtain the text feature representation vector F. t :

[0040] c=(c1…c j …c n-h+1 )

[0041]

[0042]

[0043] Among them, maxpool(·) represents the maximum pooling function;

[0044] Step B23: Convert the image in the post to i Input into ResNet50, extract the output of the second layer of ResNet50 and represent it as V ResNet , and then pass it through a fully connected layer to obtain the visual feature representation vector F v :

[0045] F v =σ(W v ·V ResNet )

[0046] Among them, W v are the parameters of the fully connected layer, and σ(·) represents the nonlinear activation function.

[0047] Furthermore, the step B3 specifically includes the following steps:

[0048] Step B31: Model the users, source posts, comments, and their corresponding interactive relationships during the post propagation process, and construct a heterogeneous propagation graph G = (V, E); where the node set V contains all post nodes, comment nodes, user nodes that post posts, and user nodes that post comments in the dataset DS, and the edge set E contains the post and comment relationships; and the text initial representation vector E obtained in step B1 is converted into t , initial comment representation vector E r and the user's initial representation vector E u As the initial representation vector of each corresponding node;

[0049] Step B32: Input the heterogeneous propagation graph G = (V, E) into the multi-layer dynamic heterogeneous graph neural network DyHGN. DyHGN first uses the relative time encoding mechanism to model the dynamic dependency relationship in the heterogeneous propagation graph, taking into account the timestamp information of the nodes. For the post node ni in the graph and all its adjacent node sets The calculation process is as follows:

[0050]

[0051]

[0052] RTE(ΔT(n i , n′ j ))=Linear(PE(ΔT(n i , n′ j )))

[0053] Among them, Linear(·) is the linear mapping function, ΔT(n i , n′ j ) represents node n i and node n′ j relative time intervals;

[0054] Step B33: Then relative to node n i The time code is added to the comment node n′ j In the representation of , we get node n′ j The initial representation vector The calculation process is as follows:

[0055]

[0056] Among them, n′ j Represents node n i The jth adjacent node of ;

[0057] Step B34: After obtaining the initial representation vectors of all adjacent nodes through step B33, DyHGN sets different linear mapping functions and parameter matrices for different types of nodes and edges, and retains the characteristics specific to the node and edge type as much as possible through parameter sharing. The node n is calculated separately. i and the attention weights of adjacent nodes to represent the attention of different adjacent nodes to node n i The different importance of node n i Attention coefficient vector of all adjacent nodes The calculation process is as follows:

[0058]

[0059]

[0060]

[0061]

[0062] Among them, τ(n i ) represents node s i type, φ(e) represents the type of edge e; They are expressed as τ(n i ) type and τ(n′ j ) type node linear mapping function, The parameter matrix representing the φ(e) type edge;

[0063] Step B35: DyHGN uses the symbolic attention mechanism to capture the positive and negative correlations between adjacent nodes to distinguish the influence of adjacent nodes on the target node; specifically: the attention coefficient vector A obtained in step B34 is inverted to obtain the attention vector And obtain the normalized weight vector A′ and Then, the information of all neighbor nodes is aggregated using the standardized weights to obtain node n i New feature representation vector The calculation process is as follows:

[0064]

[0065] A′=softmax(A)

[0066]

[0067]

[0068]

[0069] Among them, W g represents the parameter matrix of the fully connected layer, || represents the concatenation symbol, and σ(·) represents the activation function;

[0070] Step B36: Set node n i New feature representation vector By enhancing the root node, we can better utilize the information from the source node to obtain a more accurate node representation H (l) [n i ], and its calculation process is as follows:

[0071]

[0072] in, The linear mapping function represents the node n i Map back to the feature space of the corresponding category; finally, the output result of the L-th layer DyHGN is used as the final output, that is, after repeating steps B32 to B36 L times, node n is obtained. iThe dynamic propagation feature representation F g .

[0073] Furthermore, the step B4 specifically includes the following steps:

[0074] Step B41: The text feature representation vector F obtained in step B2 is t and visual feature representation vector F v Input into the joint attention network, and obtain the image-based text feature representation vector Z through two cross-attention sub-layers t and text-based image feature representation vector Z v , the calculation process is as follows:

[0075] Z t =CrossAtt(F t , F v )

[0076] Z v =CrossAtt(F v , F t )

[0077] Among them, CrossAtt(·) represents the cross attention function;

[0078] Step B42: Represent the image-based text feature vector Z t and text-based image feature representation vector Z v The two self-attention sub-layers are input respectively to guide the model to pay attention to the key features specific to each single modality. Then the outputs of the two self-attention sub-layers are concatenated to obtain the joint representation of text and vision F. tv , the calculation process is as follows:

[0079]

[0080]

[0081]

[0082] Among them, SelfAtt(·) represents the self-attention function, and Concat(·) represents the concatenation function;

[0083] Step B43: The text feature representation vector F obtained in step B2 is t and the dynamic propagation feature characterization vector F obtained in step B3 g Input them together into the joint attention network and perform the same operations as steps B41 and B42 to obtain the joint representation vector F of the text and the propagation graph. tg .

[0084] Furthermore, the step B5 specifically includes the following steps:

[0085] Step B51: Combine the post text and visual representation vector F obtained in step B4 tv and the joint representation vector F of the post text and the spread graph tg The modality alignment operation is performed under the condition of self-supervisory loss, and the mean square loss is used as the self-supervised loss function to map the two modalities into the same feature space to obtain the aligned text-visual joint representation vector. and the joint representation vector of the text diffusion graph And the alignment loss function is expressed as L align , the calculation process is as follows:

[0086]

[0087]

[0088]

[0089] Where W tv and W tg represents the learnable parameter matrix;

[0090] Step B52: Represent the aligned joint features and Input into the common attention network for the next step of multimodal fusion, perform the same operations as steps B41 and B42, and obtain the multimodal feature representation F of the multimedia post tvg , which is calculated as follows:

[0091]

[0092]

[0093]

[0094]

[0095]

[0096] Furthermore, the step B6 specifically includes the following steps:

[0097] Step B61: The multimodal representation vector F obtained in step B52 is tvg The input is fed into the fully connected layer and normalized using softmax to get the predicted probability of the post belonging to each category. The calculation formula is as follows:

[0098] y=W o F tvg+b

[0099] p a (y) = softmax(y)

[0100] Among them, y represents the output vector of the fully connected layer, is the fully connected layer parameter matrix, b is the fully connected layer bias term, p a (y) represents the probability of predicting that the post label category is a, 0≤p a (y)≤1, a∈{non-rumor, rumor};

[0101] Step B62: Use the cross entropy loss function as the classification loss function to calculate the loss value. The calculation formula is as follows:

[0102]

[0103] in, is the L2 regularization term, λ is the learning rate, and θ is all the parameters contained in the model;

[0104] Step B63: Alignment loss L align and classification loss L classify Together they serve as the final loss function of the model. The gradient optimization algorithm Adam is used as the optimizer to iteratively update all parameters of the model through backpropagation to minimize the loss function to train the model. The calculation formula is as follows:

[0105] loss = β a L align +β c L classify

[0106] Among them, β a and β c is the coefficient used to balance the two losses.

[0107] The present invention also provides a multimodal rumor detection system with enhanced dynamic propagation characteristics using the above method, comprising:

[0108] The data collection module is used to extract the text, comments, images and related user information of posts in social media, annotate the authenticity label categories of the posts, and build a training dataset;

[0109] A data processing module is used to preprocess the training sample data in the training data set;

[0110] The text encoding module is used to find the word vectors of the pre-processed post text content and comment content from the pre-trained word vector dictionary to obtain the initial representation vectors of the post text content and the initial representation vectors of the comment content;

[0111] A user encoding module is used to encode the user according to the user profile information to obtain the user's initial representation vector;

[0112] The model training module is used to input the initial representation vector of the post text, the initial representation vector of the comment, and the initial representation vector of the user into the multimodal rumor detection model to obtain the multimodal representation vector of the post. This representation vector is used to predict the probability of the post belonging to a certain label category. The loss is calculated by comparing it with the true category label of the post. The entire multimodal rumor detection model is trained with the goal of minimizing the loss, resulting in a multimodal rumor detection model with enhanced dynamic propagation features.

[0113] The multimodal rumor detection module is used to detect the authenticity of multimedia posts. It uses a trained multimodal rumor detection model with enhanced dynamic propagation features to analyze and process the input posts to be detected, extract the post's textual semantic information, visual information, and dynamic propagation information in the heterogeneous propagation graph, effectively integrate the three modal information, and finally output the predicted label of the post.

[0114] Compared with the existing technology, the present invention has the following beneficial effects: it provides a multimodal rumor detection method and system with enhanced dynamic propagation features. The method and system construct a multimodal rumor detection model with enhanced dynamic propagation features, perform multimodal characterization of posts based on the enhanced dynamic propagation features, and fuse multimodal features to judge the authenticity of the posts, thereby improving the accuracy and reliability of rumor detection in posts on social media, and has strong practicality and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 is a flowchart of a method implementation of an embodiment of the present invention;

[0116] Figure 2 is an architectural diagram of a multimodal rumor detection model in an embodiment of the present invention;

[0117] Figure 3 Schematic diagram of the system structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0118] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0119] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0120] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0121] like Figure 1 As shown, this embodiment provides a multimodal rumor detection method with enhanced dynamic propagation features, including the following steps:

[0122] Step A: Collect multimedia posts in social media, extract text, comments, images and user information from the posts, and annotate the authenticity labels of the posts to construct a training dataset DS.

[0123] Step B: Use the training dataset DS to train a multimodal rumor detection model M with dynamic propagation feature enhancement. The model architecture is as follows: Figure 2 As shown, the model M is used to analyze the authenticity of the post and obtain the predicted label of the post.

[0124] Step C: Input the text, comments, images and user profiles of the undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts.

[0125] In this embodiment, step B specifically includes the following steps:

[0126] Step B1: Initially encode all training samples in the training dataset DS to obtain the initial text representation vector E t , initial comment representation vector E r and the user's initial representation vector E u .

[0127] In this embodiment, step B1 specifically includes the following steps:

[0128] Step B11: Traverse the sample data of the training dataset DS, perform word segmentation on the text and comment content of the post, remove stop words and special symbols, and delete posts containing invalid images; each sample post s in DS i Indicated as s i =(t i , r i ,u i , v i , a); where t i , v i and u iRepresent the text, image and publisher of the post, r i Indicates posts i The corresponding comment set, Indicates posts i The jth comment of All by the corresponding users Release; a is the authenticity label corresponding to the post, a∈{non-rumor, rumor}.

[0129] The text of the post i Expressed as:

[0130]

[0131] in, For the post text content i The kth word in the post, k = 1, 2, ..., n, n is the post text t i The number of words.

[0132] Comments on the post Expressed as:

[0133]

[0134] in, For comment content The kth word in, i = 1, 2, ..., m, where m is a comment The number of words.

[0135] Step B12: Get the text from step B11 Encode through word2vec to get the text t i The initial characterization vector E t ;E t Expressed as:

[0136]

[0137] in, Represents the word vector matrix from pre-training The word embedding representation vector obtained by looking up in Represents the kth word The corresponding word embedding representation vector, d represents the dimension of the word embedding representation vector, and |V| represents the number of words in the dictionary V.

[0138] Step B13: Comments on Step B11 Encode through word2vec to get comments The initial characterization vector E r ;E r Expressed as:

[0139]

[0140] in, Represents the word vector matrix from pre-training The word embedding representation vector obtained by looking up in Represents the kth word The corresponding word embedding representation vector, d represents the dimension of the word embedding representation vector, and |V| is the number of words in the dictionary V.

[0141] Step B14: Use word2vec to encode the historical posts in the user profile, and then take the average of all encoded vectors to get the user u i The initial representation vector Where d is the dimension of the representation vector.

[0142] Step B2: Initial text representation vector E t The images in the post are input into Text-CNN and ResNet50 respectively to obtain the post text feature representation vector F t and visual feature representation vector F v .

[0143] In this embodiment, step B2 specifically includes the following steps:

[0144] Step B21: Translate the post text into i The initial characterization vector E t Input into Text-CNN, use the convolution kernel of size h to perform convolution operation on the input matrix, and obtain the sequence feature representation c of h consecutive words starting from the jth word j , the calculation process is as follows:

[0145]

[0146] Among them, the convolution kernel size h∈{3, 4, 5}, W t represents the weight matrix of the filter, and σ(·) represents the ReLU activation function.

[0147] Step B22: Then perform the maximum pooling operation on the result c after the convolution operation of all words in the text to obtain the representation vector after convolution. Finally, the pooling results of convolution kernels of different sizes are spliced to obtain the text feature representation vector F. t :

[0148] c=(c1…c j …c n-h+1 )

[0149]

[0150]

[0151] Among them, maxpool(·) represents the maximum pooling function.

[0152] Step B23: Convert the image in the post to i Input into ResNet50, extract the output of the second layer of ResNet50 and represent it as V ResNet , and then pass it through a fully connected layer to obtain the visual feature representation vector F v :

[0153] F v v =σ(W v ·V ResNet )

[0154] Among them, W v are the parameters of the fully connected layer, and σ(·) represents the nonlinear activation function.

[0155] Step B3: Model the users, source posts, comments and their corresponding interactive relationships in the post propagation process to construct a heterogeneous propagation graph G; input the heterogeneous propagation graph G into the multi-layer dynamic heterogeneous graph neural network DyHGN. The DyHGN network mainly includes a symbolic attention mechanism and a root node enhancement module. First, the symbolic attention mechanism is used to obtain the node n i The attention coefficient vector A of all adjacent nodes is then used as a weight to aggregate the information of all adjacent nodes, and the node n is obtained after being enhanced by the root node. i Information representation vector H (l) [n i ], and finally get node n after L layers of DyHGN i The dynamic propagation feature representation vector F g .

[0156] In this embodiment, step B3 specifically includes the following steps:

[0157] Step B31: Model the users, source posts, comments, and their corresponding interactive relationships during the post propagation process, and construct a heterogeneous propagation graph G = (V, E); where the node set V contains all post nodes, comment nodes, user nodes that post posts, and user nodes that post comments in the dataset DS, and the edge set E contains the post and comment relationships; and the text initial representation vector E obtained in step B1 is converted into t , initial comment representation vector E r and the user's initial characterization vector Eu are respectively used as the initial characterization vectors of each corresponding node.

[0158] Step B32: Input the heterogeneous propagation graph G = (V, E) into the multi-layer dynamic heterogeneous graph neural network DyHGN. DyHGN first uses the relative time encoding mechanism to model the dynamic dependency relationship in the heterogeneous propagation graph, taking into account the timestamp information of the nodes. For the post node n in the graph, i and the set of all its adjacent nodes The calculation process is as follows:

[0159]

[0160]

[0161] RTE(ΔT(n i , n′ j ))=Linear(PE(ΔT(n i , n′ j )))

[0162] Among them, Linear(·) is the linear mapping function, ΔT(n i , n′ j ) represents node n i and node n′ j relative time interval.

[0163] Step B33: Then relative to node n i The time code is added to the comment node n′ j In the representation of , we get node n′ j The initial representation vector The calculation process is as follows:

[0164]

[0165] Among them, n′ j Represents node n i The jth adjacent node of .

[0166] Step B34: After obtaining the initial representation vectors of all adjacent nodes through step B33, DyHGN sets different linear mapping functions and parameter matrices for different types of nodes and edges, and retains the characteristics specific to the node and edge type as much as possible through parameter sharing. The node n is calculated separately. i and the attention weights of adjacent nodes to represent the attention of different adjacent nodes to node n i The different importance of node n i Attention coefficient vector of all adjacent nodes The calculation process is as follows:

[0167]

[0168]

[0169]

[0170]

[0171] Among them, τ(n i ) represents node s i type, φ(e) represents the type of edge e; They are expressed as τ(n i ) type and τ(n′ j ) type node linear mapping function, Parameter matrix representing the φ(e) type edge.

[0172] Step B35: DyHGN uses the symbolic attention mechanism to capture the positive and negative correlations between adjacent nodes to distinguish the influence of adjacent nodes on the target node; specifically: the attention coefficient vector A obtained in step B34 is inverted to obtain the attention vector And obtain the normalized weight vector A′ and Then, the information of all neighbor nodes is aggregated using the standardized weights to obtain node n i New feature representation vector The calculation process is as follows:

[0173]

[0174] A′=softmax(A)

[0175]

[0176]

[0177]

[0178] Among them, W g represents the parameter matrix of the fully connected layer, || represents the concatenation symbol, and σ(·) represents the activation function.

[0179] Step B36: Set node n i New feature representation vector By enhancing the root node, we can better utilize the information from the source node to obtain a more accurate node representation H (l) [n i ], and its calculation process is as follows:

[0180]

[0181] in, The linear mapping function represents the node n i Map back to the feature space of the corresponding category; finally, the output result of the L-th layer DyHGN is used as the final output, that is, after repeating steps B32 to B36 L times, node n is obtained. i The dynamic propagation feature representation F g .

[0182] Step B4: Represent the text feature vector F of the post t and visual feature representation vector F v Input into the joint attention network to obtain the joint representation vector F of the post text and vision tv ; Then the text feature representation vector F of the post t and dynamic propagation feature representation vector F g Input into the joint attention network to obtain the joint representation vector F of the post text and the spread graph tg .

[0183] In this embodiment, step B4 specifically includes the following steps:

[0184] Step B41: The text feature representation vector F obtained in step B2 is t and visual feature representation vector F v Input into the joint attention network, and obtain the image-based text feature representation vector Z through two cross-attention sub-layers t and text-based image feature representation vector Z v , the calculation process is as follows:

[0185] Z t =CrossAtt(F t , F v )

[0186] Z v =CrossAtt(F v , F t )

[0187] Among them, CrossAtt(·) represents the cross attention function.

[0188] Step B42: Represent the image-based text feature vector Z t and text-based image feature representation vector Z v The two self-attention sub-layers are input respectively to guide the model to pay attention to the key features specific to each single modality. Then the outputs of the two self-attention sub-layers are concatenated to obtain the joint representation of text and vision F. tv , the calculation process is as follows:

[0189]

[0190]

[0191]

[0192] Among them, SelfAtt(·) represents the self-attention function, and Concat(·) represents the concatenation function.

[0193] Step B43: The text feature representation vector F obtained in step B2 is t and the dynamic propagation feature characterization vector F obtained in step B3 g Input them together into the joint attention network and perform the same operations as steps B41 and B42 to obtain the joint representation vector F of the text and the propagation graph. tg .

[0194] Step B5: Combine the post text and visual representation vector F obtained in step B4 tv and the joint representation vector F of the post text and the spread graph tg Perform modality alignment under self-supervisory loss to obtain the aligned text-visual joint representation vector and the joint representation vector of the text diffusion graph Then the text-visual joint representation vector after modality alignment is and text propagation graph jointly represent the vector Input into the joint attention network to obtain the multimodal representation vector F of the post tvg .

[0195] In this embodiment, step B5 specifically includes the following steps:

[0196] Step B51: Combine the post text and visual representation vector F obtained in step B4 tv and the joint representation vector F of the post text and the spread graph tg The modality alignment operation is performed under the condition of self-supervisory loss, and the mean square loss is used as the self-supervised loss function to map the two modalities into the same feature space to obtain the aligned text-visual joint representation vector. and the joint representation vector of the text diffusion graph And the alignment loss function is expressed as L align , the calculation process is as follows:

[0197]

[0198]

[0199]

[0200] Where W tv and W tg represents the learnable parameter matrix.

[0201] Step B52: Represent the aligned joint features and Input into the common attention network for the next step of multimodal fusion, perform the same operations as steps B41 and B42, and obtain the multimodal feature representation F of the multimedia post tvg , which is calculated as follows:

[0202]

[0203]

[0204]

[0205]

[0206]

[0207] Step B6: Transform the multimodal representation vector F of the post tvg The data is input into the fully connected layer to obtain the prediction results of the post; then, according to the target loss function, the gradient of each parameter in the multimodal rumor detection model is calculated by using the back propagation method, and all parameters of the model are iteratively updated using back propagation.

[0208] In this embodiment, step B6 specifically includes the following steps:

[0209] Step B61: The multimodal representation vector F obtained in step B52 is tvg The input is fed into the fully connected layer and normalized using softmax to get the predicted probability of the post belonging to each category. The calculation formula is as follows:

[0210] y=W o F tvg +b

[0211] p a (y) = softmax(y)

[0212] Among them, y represents the output vector of the fully connected layer, is the fully connected layer parameter matrix, b is the fully connected layer bias term, p a (y) represents the probability of predicting that the post label category is a, 0≤p a (y)≤1, a∈{non-rumor, rumor}.

[0213] Step B62: Use the cross entropy loss function as the classification loss function to calculate the loss value. The calculation formula is as follows:

[0214]

[0215] in, is the L2 regularization term, λ is the learning rate, and θ is all the parameters contained in the model.

[0216] Step B63: Alignment loss L align and classification loss L classify Together they serve as the final loss function of the model. The gradient optimization algorithm Adam is used as the optimizer to iteratively update all parameters of the model through backpropagation to minimize the loss function to train the model. The calculation formula is as follows:

[0217] loss = β a L align +β c L classify

[0218] Among them, β a and β c is the coefficient used to balance the two losses.

[0219] Step B7: When the loss value change generated by each iteration of the multimodal rumor detection model is less than a given threshold or reaches the maximum number of iterations, the model training process is terminated.

[0220] like Figure 3 As shown, this embodiment also provides a multimodal rumor detection system with enhanced dynamic propagation features using the above method, including: a data collection module, a data processing module, a text encoding module, a user encoding module, a model training module and a multimodal rumor detection module.

[0221] The data collection module is used to extract text, comments, images and related user information from social media posts, and to annotate the authenticity label categories of the posts to construct a training data set.

[0222] The data processing module is used to preprocess the training sample data in the training data set.

[0223] The text encoding module is used to search the pre-processed word vectors of the words in the post text content and the comment content from the pre-trained word vector dictionary to obtain the initial representation vector of the post text content and the initial representation vector of the comment content.

[0224] The user encoding module is used to encode the user according to the user profile information to obtain the user's initial representation vector.

[0225] The model training module is used to input the initial representation vector of the post text, the initial representation vector of the comment and the initial representation vector of the user into the multimodal rumor detection model to obtain the multimodal representation vector of the post, use the representation vector to predict the probability that it belongs to a certain label category, and calculate the loss by comparing it with the true category label of the post. The entire multimodal rumor detection model is trained with the goal of minimizing the loss to obtain a multimodal rumor detection model with enhanced dynamic propagation features.

[0226] The multimodal rumor detection module is used to detect the authenticity of multimedia posts. It uses a trained multimodal rumor detection model with enhanced dynamic propagation features to analyze and process the input posts to be detected, extract the textual semantic information, visual information and dynamic propagation information in the heterogeneous propagation graph of the posts, effectively integrate the three modal information, and finally output the predicted label of the post.

[0227] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0228] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0229] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0230] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0231] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A multimodal rumor detection method with enhanced dynamic propagation features, characterized in that: The following steps are involved: Step A: Collect multimedia posts from social media, extract text, comments, images, and user profiles from the posts, and annotate the posts with authenticity labels to construct a training dataset DS. Step B: Use the training dataset DS to train a multimodal rumor detection model M with dynamic propagation feature enhancement to analyze the authenticity of the post and derive the predicted label of the post. Step C: Input the text, comments, images, and user profiles of the undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts; The step B specifically comprises the following steps: Step B1: Initially encode all training samples in the training dataset DS to obtain the initial text representation vector E t , initial comment representation vector E r and the user's initial representation vector E u ; Step B2: Initial text representation vector E t The images in the post are input into Text-CNN and ResNet50 respectively to obtain the post text feature representation vector F t and visual feature representation vector F v ; Step B3: Model the users, source posts, comments and their corresponding interactive relationships in the post propagation process to construct a heterogeneous propagation graph G; input the heterogeneous propagation graph G into the multi-layer dynamic heterogeneous graph neural network DyHGN. DyHGN mainly includes a symbolic attention mechanism and a root node enhancement module. First, the symbolic attention mechanism is used to obtain the node n i The attention coefficient vector A of all adjacent nodes is then used as a weight to aggregate the information of all adjacent nodes, and the node n is obtained after being enhanced by the root node. i Information representation vector H (l) [n i ], and finally get node n after L layers of DyHGN i The dynamic propagation feature representation vector F g ; Step B4: Represent the text feature vector F of the post t and visual feature representation vector F v Input into the joint attention network to obtain the joint representation vector F of the post text and vision tv ; Then the text feature representation vector F of the post t and dynamic propagation feature representation vector F g Input into the joint attention network to obtain the joint representation vector F of the post text and the spread graph tg ; Step B5: Combine the post text and visual representation vector F obtained in step B4 tv and the joint representation vector F of the post text and the diffusion graph tg Perform modality alignment under self-supervisory loss to obtain the aligned text-visual joint representation vector and the joint representation vector of the text diffusion graph Then the text-visual joint representation vector after modality alignment is and text propagation graph jointly represent the vector Input into the joint attention network to obtain the multimodal representation vector F of the post tvg ; Step B6: Transform the multimodal representation vector F of the post tvg The data is input into the fully connected layer to obtain the predicted result of the post. Then, based on the target loss function, the gradient of each parameter in the multimodal rumor detection model is calculated by backpropagation, and all parameters of the model are iteratively updated using backpropagation. Step B7: When the loss value change generated by each iteration of the multimodal rumor detection model is less than a given threshold or reaches the maximum number of iterations, the model training process is terminated; The step B3 specifically includes the following steps: Step B31: Model the users, source posts, comments, and their corresponding interactive relationships during the post propagation process, and construct a heterogeneous propagation graph G = (V, E); where the node set V contains all post nodes, comment nodes, user nodes that post posts, and user nodes that post comments in the dataset DS, and the edge set E contains the post and comment relationships; and the initial text representation vector E obtained in step B1 is converted to r , initial comment representation vector E r and the user's initial representation vector E u As the initial representation vector of each corresponding node; Step B32: Input the heterogeneous propagation graph G = (V, E) into the multi-layer dynamic heterogeneous graph neural network DyHGN. DyHGN first uses the relative time encoding mechanism to model the dynamic dependency relationship in the heterogeneous propagation graph, taking into account the timestamp information of the nodes. For the post node n in the graph, i and the set of all its adjacent nodes The calculation process is as follows: RTE(ΔT(n i ,n′ j ))=Linear(PE(ΔT(n i ,n′ j ))) Among them, Linear(·) is the linear mapping function, ΔT(n i ,n′ j ) represents node n i and node n′ j relative time intervals; Step B33: Then relative to node n i The time code is added to the adjacent node n′ j In the representation of , we get node n′ j The initial representation vector The calculation process is as follows: Among them, n′ j Represents node n i The jth adjacent node of ; Step B34: After obtaining the initial representation vectors of all adjacent nodes through step B33, DyHGN sets different linear mapping functions and parameter matrices for different types of nodes and edges, and retains the characteristics specific to the node and edge type through parameter sharing. The node n is calculated separately. i and the attention weights of adjacent nodes to represent the attention of different adjacent nodes to node n i The different importance of node n i Attention coefficient vector of all adjacent nodes The calculation process is as follows: Among them, τ(n i ) represents node s i type, φ(e) represents the type of edge e; They are expressed as τ(n i ) type and τ(n ′ j ) type node linear mapping function, The parameter matrix representing the φ(e) type edge; Step B35: DyHGN uses the symbolic attention mechanism to capture the positive and negative correlations between adjacent nodes to distinguish the influence of adjacent nodes on the target node; specifically: the attention coefficient vector A obtained in step B34 is inverted to obtain the attention vector And get the standardized weight vector A through softmax ′ and Then, the information of all neighbor nodes is aggregated using the standardized weights to obtain node n i New feature representation vector The calculation process is as follows: A ′ =softmax(A) Among them, W g represents the parameter matrix of the fully connected layer, || represents the concatenation symbol, and σ(·) represents the activation function; Step B36: Set node n i New feature representation vector By enhancing the root node, we can better utilize the information from the source node to obtain a more accurate node representation H (l) [n i ], and its calculation process is as follows: in, The linear mapping function represents the node n i Map back to the feature space of the corresponding category; finally, the output result of the L-th layer DyHGN is used as the final output, that is, after repeating steps B32 to B36 L times, node n is obtained. i The dynamic propagation feature representation F g ; The step B5 specifically includes the following steps: Step B51: Combine the post text and visual representation vector F obtained in step B4 tv and the joint representation vector F of the post text and the spread graph tg The modality alignment operation is performed under the condition of self-supervisory loss, and the mean square loss is used as the self-supervised loss function to map the two modalities into the same feature space to obtain the aligned text-visual joint representation vector. and the joint representation vector of the text diffusion graph And the alignment loss function is expressed as L align , the calculation process is as follows: Where W tv and W tg represents the learnable parameter matrix; Step B52: Represent the aligned joint features and Input into the common attention network for the next step of multimodal fusion, perform the same operations as steps B41 and B42, and obtain the multimodal feature representation F of the multimedia post tvg , which is calculated as follows: Among them, CrossAtt(·) represents the cross attention function, SelfAtt(·) represents the self-attention function, and Concat(·) represents the concatenation function.

2. The multimodal rumor detection method with enhanced dynamic propagation features according to claim 1 is characterized in that: The step B1 specifically includes the following steps: Step B11: Traverse the sample data of the training dataset DS, perform word segmentation on the text and comment content of the post, remove stop words and set symbols, and delete posts containing invalid images; each sample post s in DS i Indicated as s i =(t i ,r i ,u i ,v i ,a); where t i , v i and u i Represent the text, image and publisher of the post, r i Indicates posts i The corresponding comment set, Indicates posts i The jth comment of All by the corresponding users Release; a is the authenticity label corresponding to the post, a∈{non-rumor, rumor}; The text of the post i Expressed as: in, For the post text content i The kth word in the post, k = 1, 2, ..., n, n is the post text t i The number of words; Comments on the post Expressed as: in, For comment content The kth word in, i = 1, 2, ..., m, m is the comment The number of words; Step B12: Get the text from step B11 Encode through word2vec to get the text t i The initial characterization vector E t ;E t Expressed as: in, Represents the word vector matrix from pre-training The word embedding representation vector obtained by looking up in Represents the kth word The corresponding word embedding representation vector, d represents the dimension of the word embedding representation vector, |V| represents the number of words in the dictionary V; Step B13: Comments on Step B11 Encode through word2vec to get comments The initial characterization vector E r ;E r Expressed as: in, Represents the word vector matrix from pre-training The word embedding representation vector obtained by looking up in Represents the kth word The corresponding word embedding representation vector, d represents the dimension of the word embedding representation vector, |V| is the number of words in the dictionary V; Step B14: Use word2vec to encode the historical posts in the user profile, and then take the average of all encoded vectors to get the user u i The initial representation vector Where d is the dimension of the representation vector.

3. The multimodal rumor detection method with enhanced dynamic propagation features according to claim 2 is characterized in that: The step B2 specifically includes the following steps: Step B21: Translate the post text into i The initial characterization vector E t Input into Text-CNN, use the convolution kernel of size h to perform convolution operation on the input matrix, and obtain the sequence feature representation c of h consecutive words starting from the jth word j , the calculation process is as follows: Among them, the convolution kernel size h∈{3,4,5}, W t represents the weight matrix of the filter, σ(·) represents the ReLU activation function; Step B22: Then perform the maximum pooling operation on the result c after the convolution operation of all words in the text to obtain the representation vector after convolution. Finally, the pooling results of convolution kernels of different sizes are spliced to obtain the text feature representation vector F. t : c=(c1…c j …c n-h+1 ) Among them, maxpool(·) represents the maximum pooling function; Step B23: Convert the image in the post to i Input into ResNet50, extract the output of the second layer of ResNet50 and represent it as V ResNet , and then pass it through a fully connected layer to obtain the visual feature representation vector F v : F v =σ(W v ·V ResNet ) Among them, W v are the parameters of the fully connected layer, and σ(·) represents the nonlinear activation function.

4. The multimodal rumor detection method with enhanced dynamic propagation features according to claim 3 is characterized in that: The step B4 specifically includes the following steps: Step B41: The text feature representation vector F obtained in step B2 is t and visual feature representation vector F v Input into the joint attention network, and obtain the image-based text feature representation vector Z through two cross-attention sub-layers t and text-based image feature representation vector Z v , the calculation process is as follows: Z t =CrossAtt(F t ,F v ) Z v =CrossAtt(F v ,F t ) Step B42: Represent the image-based text feature vector Z t and text-based image feature representation vector Z v The two self-attention sub-layers are input respectively to guide the model to pay attention to the key features specific to each single modality. Then the outputs of the two self-attention sub-layers are concatenated to obtain the joint representation of text and vision F. tv , the calculation process is as follows: Step B43: The text feature representation vector F obtained in step B2 is t and the dynamic propagation feature characterization vector F obtained in step B3 g Input them together into the joint attention network and perform the same operations as steps B41 and B42 to obtain the joint representation vector F of the text and the propagation graph. tg .

5. The multimodal rumor detection method with enhanced dynamic propagation features according to claim 4 is characterized in that: The step B6 specifically includes the following steps: Step B61: The multimodal representation vector F obtained in step B52 is tvg The input is fed into the fully connected layer and normalized using softmax to get the predicted probability of the post belonging to each category. The calculation formula is as follows: y=W o F tvg +b p a (y)=softmax(y) Among them, y represents the output vector of the fully connected layer, is the fully connected layer parameter matrix, b is the fully connected layer bias term, p a (y) represents the probability of predicting that the post label category is a, 0≤p a (y)≤1, a∈{non-rumor, rumor}; Step B62: Use the cross entropy loss function as the classification loss function to calculate the loss value. The calculation formula is as follows: in, is the L2 regularization term, λ is the learning rate, and θ is all the parameters contained in the model; Step B63: Alignment loss L align and classification loss L classify Together they serve as the final loss function of the model. The gradient optimization algorithm Adam is used as the optimizer to iteratively update all parameters of the model through backpropagation to minimize the loss function to train the model. The calculation formula is as follows: loss=β a L align +b c L classify Among them, β a and β c is the coefficient used to balance the two losses.

6. A multimodal rumor detection system using the method according to any one of claims 1 to 5 with enhanced dynamic propagation features, characterized in that: include: The data collection module is used to extract the text, comments, images and related user information of posts in social media, annotate the authenticity label categories of the posts, and build a training dataset; A data processing module is used to preprocess the training sample data in the training data set; The text encoding module is used to find the word vectors of the pre-processed post text content and comment content from the pre-trained word vector dictionary to obtain the initial representation vectors of the post text content and the initial representation vectors of the comment content; A user encoding module is used to encode the user according to the user profile information to obtain the user's initial representation vector; The model training module is used to input the initial representation vector of the post text, the initial representation vector of the comment, and the initial representation vector of the user into the multimodal rumor detection model to obtain the multimodal representation vector of the post. This representation vector is used to predict the probability of the post belonging to a certain label category. The loss is calculated by comparing it with the true category label of the post. The entire multimodal rumor detection model is trained with the goal of minimizing the loss, resulting in a multimodal rumor detection model with enhanced dynamic propagation features. The multimodal rumor detection module is used to detect the authenticity of multimedia posts. It uses a trained multimodal rumor detection model with enhanced dynamic propagation features to analyze and process the input posts to be detected, extract the post's textual semantic information, visual information, and dynamic propagation information in the heterogeneous propagation graph, effectively integrate the three modal information, and finally output the predicted label of the post.

Images