Multi-modal rumor detection method and system fusing multi-granularity features

Abstract

The invention relates to a multi-modal rumor detection method and system fusing multi-granularity features, and the method comprises the steps: collecting multimedia posts in social media, extracting texts, images and comments in the posts, and labeling authenticity labels of the posts, thereby constructing a training data set; a multi-modal rumor detection model fused with multi-granularity features is constructed, the multi-modal rumor detection model extracts the multi-granularity features in posts, image-text features are deeply fused, resolution of image-text ambiguity is achieved through image-text similarity weights, comment information is fully mined, comments are utilized from different angles, and the image-text ambiguity resolution is achieved. Finally, similar information of the same category of social media posts is captured through comparative learning; using the training data set to train a multi-modal rumor detection model; and inputting texts, images and comments of undetected multimedia posts into the trained multi-modal rumor detection model to obtain authenticity labels of the multimedia posts. The method and the system are beneficial to improving the rumor detection accuracy and reliability.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to a multimodal rumor detection method and system integrating multi-granularity features. Background Art

[0002] Rumor detection, also known as fake news detection, is an important research task in the field of natural language processing (NLP). With the rise of social media, social media such as Weibo and Twitter have become an indispensable communication tool for people. However, due to the fast speed and wide range of social media, rumors have gradually become an important issue of concern. In social media, rumors are a kind of unverified information expression, which usually uses false or forged images and inflammatory texts to mislead users. The emergence of rumors often causes huge economic losses and serious harm to individuals or society. Although various social media organizations have successively launched rumor-refuting platforms to strengthen the control of rumors, these platforms still need to spend a lot of manpower and material resources to collect information to judge the authenticity of rumors. Therefore, there is an urgent need for an automated method that can effectively detect rumors.

[0003] Since deep learning methods have great performance advantages over traditional machine learning methods, more and more researchers have begun to apply deep learning methods to the problem of rumor detection. Liu Zheng et al. proposed a rumor detection model based on convolutional neural networks. The text content of the rumor is vectorized, and the deep features of the text are mined by convolutional neural networks. These features include some features that are not easy to be discovered by humans, thereby improving performance. Ma et al. first applied recurrent neural networks to rumor detection. The algorithm can learn the hidden representation of Weibo posts and use this representation to show the contextual information of related posts that changes over time. The experimental results show the obvious advantages of deep learning models in this field. Bi et al. modeled the information transmission network of Weibo as a heterogeneous graph with various semantic information, and then constructed a graph-based rumor detection model, using the attention layer to capture and aggregate its semantic information. Dou et al. considered the impact of user preferences on the rumor detection task. The article shows that the user's history and social activities (such as posts) provide rich information about the user's news preferences and have great potential to promote rumor detection.

[0004] Nowadays, posts published on social media usually take the form of a combination of text and images. These two different modalities can provide their own unique information and complement each other. Studies have shown that images can help users better understand and remember text content by providing rich visual information. Therefore, in the rumor detection task, it is necessary to use both text and image information to identify rumors. Khattar et al. proposed a multimodal variational autoencoder for rumor detection to learn shared representations of the two modalities of text and images. Zhang et al. designed a multimodal multi-task learning framework by introducing stance tasks. Zhou et al. considered the correlation between text information and visual information and identified rumors based on the mismatched information of text and images. Wang et al. proposed KMGCN, which integrates text, vision and knowledge information into a unified framework to model semantic representation and further improve accuracy.

[0005] Although existing rumor detection methods have achieved some results, they still have certain limitations, especially in the research of multimodal rumor detection. There is still a lack of full utilization of text and image data and effective fusion between the two modalities. There is no effective measurement of the ambiguity of images and texts, which leads to the introduction of noise. Secondly, for comments, all comments are simply introduced without studying the internal relationship of the comments. These problems may lead to a decline in rumor detection performance. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal rumor detection method and system integrating multi-granularity features, which are conducive to improving the accuracy and reliability of rumor detection.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a multimodal rumor detection method integrating multi-granularity features, comprising the following steps:

[0008] Step A: Collect multimedia posts in social media, extract text, images and comments from the posts, and annotate the authenticity labels of the posts to construct a training dataset DS;

[0009] Step B: construct a multimodal rumor detection model M integrating multi-granular features, wherein the multimodal rumor detection model M extracts multi-granular features from posts, deeply integrates image and text features, and resolves image and text ambiguity through image and text similarity weights, and fully mines the information of comments, utilizes comments from different angles, and finally captures similar information of social media posts of the same category through comparative learning; the multimodal rumor detection model M is trained using the training dataset DS;

[0010] Step C: Input the text, images, and comments of the undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts.

[0011] Furthermore, the step B specifically includes the following steps:

[0012] Step B1: Process all training samples in the training dataset DS to obtain supplementary evidence, mainstream opinions, and text representations of images in the comments, and use the AMR graph tool to generate a graph representation G of the text. t ;

[0013] Step B2: Use BERT to encode the text representation of text, supplementary evidence, mainstream opinion and image respectively to obtain the text fine-grained representation vector T fine , supplementary evidence representation vector C sup , mainstream opinion representation vector C main , image coarse-grained representation vector V coarse , use Swin-T to encode the image to obtain the image fine-grained representation vector V fine , use BERT to encode the text graph representation and input GCN to obtain the coarse-grained text representation vector T coarse ;

[0014] Step B3: Cascade image coarse-grained representation vector V coarse and the text coarse-grained representation vector T coarse Get the coarse-grained representation vector TV of the image and text coarse , through the collaborative attention mechanism to fuse the text fine-grained representation vector T fine and the image fine-grained representation vector V fine , and obtain the fine-grained representation vector TV of the image and text fine , cascaded image and text coarse-grained representation vector TV coarse and fine-grained representation vector TV fine , and multiply it by the image-text similarity weight to get the cross-modal representation vector F m ;

[0015] Step B4: Cascaded aggregated text fine-grained representation vector T fine and the aggregated text coarse-grained representation vector T coarse , the text unimodal representation vector F is obtained through projection head mapping t , the cascaded aggregated image fine-grained representation vector V fine and the aggregated image coarse-grained representation vector V coarse , the image unimodal representation vector F is obtained through projection head mapping v ;

[0016] Step B5: Use the collaborative attention mechanism to realize the supplementary evidence representation vector Csup and the text unimodal representation vector F t , image unimodal representation vector F v , cross-modal representation vector F m The interaction of text comments is obtained by tc , image comment interaction representation vector F vc , cross-modal comment interaction representation vector F mc ;

[0017] Step B6: Concatenate text-comment interaction representation vector F tc , image comment interaction representation vector F vc , cross-modal comment interaction representation vector F mc Get the multimodal representation vector F out , aligning the multimodal representation vector F through supervised contrastive learning out ;

[0018] Step B7: Transform the multimodal representation vector F out and the mainstream opinion representation vector C main The combined input is fed into the fully connected layer to obtain the prediction result of the post; then, according to the target loss function, the gradient of each parameter in the multimodal rumor detection model is calculated by using the back propagation method, and all the parameters of the model are iteratively updated using back propagation;

[0019] Step B8: When the loss value change generated by each iteration of the multimodal rumor detection model is less than the given threshold or reaches the maximum number of iterations, the training process of the model is terminated.

[0020] Furthermore, the step B1 specifically includes the following steps:

[0021] Step B11: Traverse the sample data of the training data set DS, perform word segmentation on the text and comment content of the post, remove stop words and special symbols, and delete the posts containing invalid images; each sample post in DS

[0022] s i Represented as s i =(t i ,r i ,v i ,a); where t i and u i Represent the text and image of the post, r i Indicates posts i The corresponding set of comments, Indicates posts i The jth comment of the post; a is the authenticity label corresponding to the post, a∈{non-rumor, rumor};

[0023] The text of the post 表 Shown as:

[0024]

[0025] in, The text content of the post i The kth word in the post, k = 1, 2, ..., n, n is the post text t i The number of words;

[0026] Comments on the post It is expressed as:

[0027]

[0028] in, For comment content The kth word in, i = 1, 2, ..., m, m is the comment r j The number of words;

[0029] Step B12: Retrieve comments from social media posts and obtain a set of supplementary evidence from the comments Indicates posts i Article k of the supplementary evidence;

[0030] The mainstream opinions are extracted through a similarity-based selection mechanism; first, BERT is used to classify the comments Encode and get the comment representation vector It is expressed as:

[0031]

[0032] in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector;

[0033] By calculating the similarity of each comment with other comments, we get an inter-sequence attention matrix Where N is the number of comments under a social media post; U m,n Represents the similarity between comment m and comment n, formalized as:

[0034]

[0035] After obtaining the inter-sequence attention matrix, calculate the sum of similarities between each comment and all other comments, and select the top-K comments with the highest sum of similarities as the mainstream opinions. Indicates posts i The kth mainstream opinion;

[0036] Step B13: Use the multimodal large model VisualGLM to translate the image and obtain the text description vt of the image i , i represents the image of the i-th social media post;

[0037] Step B14: Generate an abstract semantic graph representation G of the text of the post using the AMR graph interpreter t , using BERT for G t Encode to get the text abstract semantic representation H t .

[0038] Furthermore, the step B2 specifically includes the following steps:

[0039] Step B21: The text obtained in step B11 Encoded by BERT, we get the text t i The fine-grained representation vector T fine ; T fine It is expressed as:

[0040]

[0041] in, Represents the kth word The output of the last hidden state in BERT encoding, b represents the dimension of the word embedding representation vector;

[0042] v i Input into Swin-T for encoding to obtain the image fine-grained representation vector V fine :

[0043]

[0044] in, represents the hidden state corresponding to the kth window in the input at the output of the last layer of the model, n p is the number of batches in Swin-T, s is the hidden size of the visual embedding;

[0045] Step B22: Supplementary evidence set obtained in step B12 Encoding through BERT to obtain additional evidence The initial characterization vector

[0046]

[0047] in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector;

[0048] The average vector of the initial representation vectors of all supplementary evidence is represented as the final supplementary evidence representation vector C sup :

[0049]

[0050] The mainstream opinion set obtained in step B12 Encoding through BERT to obtain mainstream views The initial characterization vector

[0051]

[0052] in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector;

[0053] The average vector of the initial representation vectors of all mainstream opinions is represented as the final mainstream opinion representation vector C main :

[0054]

[0055] Step B23: Text description vt of the image obtained in step B13 i Use BERT to encode and get the coarse-grained representation vector V of the image coarse ; V coarse It is expressed as:

[0056]

[0057] in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector;

[0058] Step B24: The text abstract semantic representation H obtained in step B14 is t Input graph convolutional neural network GCN to obtain the coarse-grained text representation vector T coarse ; T coarse It is expressed as:

[0059] T coarse =GCN(H t ).

[0060] Furthermore, the step B3 specifically includes the following steps:

[0061] Step B31: The text fine-grained representation vector T obtained in step B2 fine and the image fine-grained representation vector V fine , using the Co-Attention Transformer CT for fusion;

[0062] CT consists of a multi-head attention network and a feed-forward neural network;

[0063] Denote different modality inputs as I1 and I2; in CT, U1 is used as query Q, and U2 is used as key K and value V. CT treats each head h i The collaborative attention matrix is ​​calculated as:

[0064]

[0065] Among them, Softmax represents the activation function, and the projection matrix Corresponding to Q, K, V, d respectively h =d m / m,d m is the dimension of CT, m is the number of heads;

[0066] The multi-head attention H is the projection of all collaborative attention matrices after cascading:

[0067] H=(h1;h2;...;h m )W o

[0068] Among them, the symbol; represents the cascade operation, the projection matrix

[0069] Then H and I1 pass through a two-layer normalized feed-forward neural network FFN to obtain an attention-based multimodal representation H':

[0070] H'=Norm(I1+FFN(Norm(I1+HI1)))

[0071] The multimodal representation H' is average-pooled into a feature vector F as the output of CT;

[0072] Feature T fine and feature V fine Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output features are obtained, including a visual attention weighted text feature F vt and a text-attention-weighted visual feature F tv :

[0073]

[0074] in,

[0075] Step B32: Cascade the image coarse-grained representation vector V obtained in step B2 coarse and the text coarse-grained representation vector T coarse Send it to the feedforward neural network FFN1 to get the coarse-grained representation vector TV of the image and text coarse , the visual attention weighted text feature F in the cascade step B31 vt and the text-attention-weighted visual features F tv Send it to the feedforward neural network FFN2 to get the fine-grained representation vector TV of the image and text fine :

[0076]

[0077] Step B33: Calculate the image coarse-grained representation vector V coarse and the text coarse-grained representation vector T coarse The cosine similarity is regarded as the image-text similarity weight, and is used as the weighted coefficient of the cross-modal feature to guide the learning process of the classifier; the cosine similarity is calculated as follows:

[0078]

[0079] Cascaded image and text coarse-grained representation vector TV coarse and fine-grained representation vector TV fine , and input the projection head Φ M Generate a cross-modal representation in ; each element in the output vector is multiplied by the image-text similarity weight, and the final cross-modal representation vector F m as follows:

[0080] F m =similarity·Φ M (TV coarse ;TV fine ).

[0081] Furthermore, the specific implementation method of step B4 is: respectively transforming the text fine-grained representation vector T fine and the text coarse-grained representation vector T coarse Aggregate into feature vectors and concatenate them, then input into the projection head consisting of two fully connected networks with ReLU activation function to obtain the text unimodal representation vector F t ; Represent the aggregated image into a fine-grained vector V fine and the aggregated image coarse-grained representation vector Vcoarse Splicing, input the projection head with the same structure as the text branch but different parameters to obtain the image unimodal representation vector F v :

[0082]

[0083] Furthermore, the specific implementation method of step B5 is:

[0084] Text unimodal representation vector F t and the supplementary evidence representation vector C sup Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output feature is obtained, a review-attention weighted text feature F ts and a text-attention-weighted review feature F st , and then fed into the feedforward neural network FFN3 to obtain the text comment interaction representation vector F tc :

[0085]

[0086] Where W t ,

[0087] Image unimodal representation vector F v and the supplementary evidence representation vector C sup Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output feature is obtained, a review-attention weighted image feature F vs and an image attention weighted review feature F sv , and then fed into the feedforward neural network FFN4 to obtain the image comment interaction representation vector F vc :

[0088]

[0089] Where W t ,

[0090] Cross-modal representation vector F m and the supplementary evidence representation vector C sup Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output feature is obtained, a review-attention weighted image feature F ms and an image attention weighted review feature F sm, and then fed into the feedforward neural network FFN5 to obtain the cross-modal comment interaction representation vector F mc :

[0091]

[0092] Where W t ,

[0093] Furthermore, the specific implementation method of step B6 is:

[0094] Connect text comment interaction representation vector F tc , image comment interaction representation vector F vc , cross-modal comment interaction representation vector F mc Send it to the feedforward neural network FFN6 to get the multimodal representation vector F out , aligning the multimodal representation vector F through supervised contrastive learning out , randomly select sample g as the anchor point, regard p with the same label as the anchor point as the positive multimodal sample, regard a with different label from the anchor point as the negative multimodal sample, and the supervised contrast loss is expressed as L align :

[0095] F out =FFN6(F tc ; F vc ; F mc )

[0096]

[0097] Among them, G is the small training set extracted during the training process, A(g) is the negative sample set in the small training set, and P(g) is the positive sample set in the small training set; sim(·) represents the cosine similarity function, l g Represents the multimodal representation vector of sample g, l p Represents the multimodal representation vector of sample p, l a represents the multimodal characterization vector of sample a, and τ is the temperature coefficient.

[0098] Furthermore, the step B7 specifically includes the following steps:

[0099] Step B71: The multimodal representation vector F obtained in step B6 is out and the mainstream opinion representation vector C main Splice the input into the classifier to get the prediction result of the post:

[0100]

[0101] Among them, FNC(·) is a rumor classifier, which consists of a two-layer fully connected network with ReLu activation function;

[0102] The cross entropy loss function is used as the classification loss function to calculate the loss value. The calculation formula is as follows:

[0103]

[0104] Among them, y is the true label;

[0105] Step B72: Alignment loss L align and classification loss L classify Together they are used as the final loss function of the model. The gradient optimization algorithm Adam is used as the optimizer to iteratively update all parameters of the model through back propagation to minimize the loss function to train the model. The calculation formula is as follows:

[0106] loss = β a L align +β c L classify

[0107] Among them, β a and β c Used to balance the two losses.

[0108] The present invention also provides a multimodal rumor detection system integrating multi-granularity features, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0109] Compared with the prior art, the present invention has the following beneficial effects:

[0110] 1. Combining the multimodal large model VisualGLM, BERT, and Swin-T, we realized the multimodal and multi-granular feature extraction of posts, fully mined the information of pictures and texts in posts, and enabled the model to understand and analyze the content of posts more comprehensively;

[0111] 2. Through the retrieval and selection mechanism, we can obtain supplementary evidence and mainstream opinions from comments, fully explore the comment information in posts from different angles, enrich the information source, improve the model's ability to identify rumors, and enable it to more accurately judge the authenticity of post content;

[0112] 3. By calculating the similarity between images and texts, we can guide the use of cross-modal fusion features, effectively alleviating the impact of image-text ambiguity, making the model more robust when the images and texts are weakly correlated, enhancing the performance of the model when processing posts with weak correlation between images and texts, and improving the reliability of overall detection;

[0113] 4. Through comparative learning, similar information of rumors of the same category is captured, which helps to correctly classify marginal samples, improves the model's ability to distinguish different types of rumors, significantly improves the accuracy and robustness of the model, and performs particularly well for complex rumor detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 is a flow chart of a method implementation of an embodiment of the present invention;

[0115] Figure 2 4 is an architecture diagram of a multimodal rumor detection model M integrating multi-granularity features in an embodiment of the present invention. DETAILED DESCRIPTION

[0116] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0117] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0118] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0119] like Figure 1 As shown, this embodiment provides a multimodal rumor detection method integrating multi-granularity features, comprising the following steps:

[0120] Step A: Collect multimedia posts in social media, extract text, images and comments from the posts, and annotate the authenticity labels of the posts to construct a training dataset DS;

[0121] Step B: Construct a multimodal rumor detection model M integrating multi-granular features. The multimodal rumor detection model M extracts multi-granular features from posts. The model can understand the content of posts more comprehensively and meticulously, deeply integrates image and text features, and enhances the comprehensive understanding of image and text content. It also resolves the ambiguity of images and texts through image and text similarity weights, making the model more robust to posts with weak correlation between images and texts. In addition, it fully mines the information of comments and uses comments from different angles to further enrich and strengthen the understanding of the content. Finally, it captures similar information of social media posts of the same category through comparative learning, further improving the accuracy of the model. Use the training dataset DS to train the multimodal rumor detection model M.

[0122] Step C: Input the text, images, and comments of the undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts.

[0123] Figure 2 : is an architecture diagram of a multimodal rumor detection model M integrating multi-granularity features in this embodiment. In this embodiment, the specific implementation steps of step B are as follows.

[0124] Step B1: Process all training samples in the training dataset DS to obtain supplementary evidence, mainstream opinions, and text representations of images in the comments, and use the AMR graph tool to generate a graph representation G of the text. t .

[0125] In this embodiment, step B1 specifically includes the following steps:

[0126] Step B11: Traverse the sample data of the training data set DS, perform word segmentation on the text and comment content of the post, remove stop words and special symbols, and delete the posts containing invalid images; each sample post in DS

[0127] s i Represented as s i =(t i ,r i ,v i ,a); where t i and u i Represent the text and image of the post, r i Indicates posts i The corresponding set of comments, Indicates posts i The jth comment of the post; a is the authenticity label corresponding to the post, a∈{non-rumor, rumor}.

[0128] The text of the post i It is expressed as:

[0129]

[0130] in, The text content of the post i The kth word in the post, k = 1, 2, ..., n, n is the post text t i The number of words.

[0131] Comments on the post It is expressed as:

[0132]

[0133] in, For comment content The kth word in, i = 1, 2, ..., m, m is the comment r j The number of words.

[0134] Step B12: Retrieve comments from social media posts and obtain a set of supplementary evidence from the comments Indicates posts i Article k of supplementary evidence.

[0135] The mainstream opinions are extracted through a similarity-based selection mechanism; first, BERT is used to classify the comments Encode and get the comment representation vector It is expressed as:

[0136]

[0137] in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector.

[0138] By calculating the similarity of each comment with other comments, we get an inter-sequence attention matrix Where N is the number of comments under a social media post; U m,n Represents the similarity between comment m and comment n, formalized as:

[0139]

[0140] After obtaining the inter-sequence attention matrix, calculate the sum of similarities between each comment and all other comments, and select the top-K comments with the highest sum of similarities as the mainstream opinions. Indicates posts i The kth mainstream opinion.

[0141] Step B13: Use the multimodal large model VisualGLM to translate the image and obtain the text description vt of the image i , i represents the image of the i-th social media post.

[0142] Step B14: Generate an abstract semantic graph representation G of the text of the post using the AMR graph interpreter t , using BERT for G t Encode to get the text abstract semantic representation H t .

[0143] Step B2: Use BERT to encode the text representation of text, supplementary evidence, mainstream opinion and image respectively to obtain the text fine-grained representation vector T fine , supplementary evidence representation vector C sup , mainstream opinion representation vector C main , image coarse-grained representation vector V coarse , use Swin-T to encode the image to obtain the image fine-grained representation vector V fine , use BERT to encode the text graph representation and input GCN to obtain the coarse-grained text representation vector T coarse .

[0144] In this embodiment, step B2 specifically includes the following steps:

[0145] Step B21: The text obtained in step B11 Encoded by BERT, we get the text t i The fine-grained representation vector T fine ; T fine It is expressed as:

[0146]

[0147] in, Represents the kth word The output of the last hidden state in BERT encoding, b represents the dimension of the word embedding representation vector.

[0148] v i Input into Swin-T for encoding to obtain the image fine-grained representation vector V fine :

[0149]

[0150] in, represents the hidden state corresponding to the kth window in the input at the output of the last layer of the model, n p is the number of batches in Swin-T and s is the hidden size of the visual embedding.

[0151] Step B22: Supplementary evidence set obtained in step B12 Encoding through BERT to obtain additional evidence The initial characterization vector

[0152]

[0153] in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector.

[0154] The average vector of the initial representation vectors of all supplementary evidence is represented as the final supplementary evidence representation vector C sup :

[0155]

[0156] The mainstream opinion set obtained in step B12 Encoding through BERT to obtain mainstream views The initial characterization vector

[0157]

[0158] in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector.

[0159] The average vector of the initial representation vectors of all mainstream opinions is represented as the final mainstream opinion representation vector C main :

[0160]

[0161] Step B23: Text description vt of the image obtained in step B13 i Use BERT to encode and get the coarse-grained representation vector V of the image coarse ; V coarse It is expressed as:

[0162]

[0163] in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector.

[0164] Step B24: The text abstract semantic representation H obtained in step B14 is t Input graph convolutional neural network GCN to obtain the coarse-grained text representation vector T coarse ; T coarse It is expressed as:

[0165] T coarse =GCN(H t ).

[0166] Step B3: Cascade image coarse-grained representation vector V coarse and the text coarse-grained representation vector T coarse Get the coarse-grained representation vector TV of the image and text coarse , through the collaborative attention mechanism to fuse the text fine-grained representation vector T fine and the image fine-grained representation vector V fine , and obtain the fine-grained representation vector TV of the image and text fine , cascaded image and text coarse-grained representation vector TV coarse and fine-grained representation vector TV fine , and multiply it by the image-text similarity weight to get the cross-modal representation vector F m .

[0167] In this embodiment, step B3 specifically includes the following steps:

[0168] Step B31: The text fine-grained representation vector T obtained in step B2 fine and the image fine-grained representation vector V fine , using the Co-Attention Transformer CT for fusion.

[0169] CT consists of a multi-head attention network and a feed-forward neural network.

[0170] Denote different modality inputs as I1 and I2; in CT, I1 is used as query Q, and I2 is used as key K and value V. CT treats each head h i The collaborative attention matrix is ​​calculated as:

[0171]

[0172] Among them, Softmax represents the activation function, and the projection matrix Corresponding to Q, K, V, d respectively h =d m / m,d m is the dimension of CT, and m is the number of heads.

[0173] The multi-head attention H is the projection of all collaborative attention matrices after cascading:

[0174] H=(h1;h2;...;h m )W o

[0175] Among them, the symbol; represents the cascade operation, the projection matrix

[0176] Then H and I1 pass through a two-layer normalized feed-forward neural network FFN to obtain an attention-based multimodal representation H':

[0177] H'=Norm(I1+FFN(Norm(I1+HI1)))

[0178] The multimodal representation H' is average-pooled into a feature vector F as the output of CT.

[0179] Feature T fine and feature V fine Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output features are obtained, including a visual attention weighted text feature F vt and a text-attention-weighted visual feature F tv :

[0180]

[0181] in,

[0182] Step B32: Cascade the image coarse-grained representation vector V obtained in step B2 coarse and the text coarse-grained representation vector T coarse Send it to the feedforward neural network FFN1 to get the coarse-grained representation vector TV of the image and text coarse , the visual attention weighted text feature F in the cascade step B31 vt and the visual features F weighted by textual attention tv Send it to the feedforward neural network FFN2 to get the fine-grained representation vector TV of the image and text fine :

[0183]

[0184] Step B33: Calculate the image coarse-grained representation vector V coarse and the text coarse-grained representation vector T coarse The cosine similarity is regarded as the image-text similarity weight, and is used as the weighted coefficient of the cross-modal feature to guide the learning process of the classifier; the cosine similarity is calculated as follows:

[0185]

[0186] Cascaded image and text coarse-grained representation vector TV coarse and fine-grained representation vector TV fine , and input the projection head Φ M Generate a cross-modal representation in ; each element in the output vector is multiplied by the image-text similarity weight, and the final cross-modal representation vector F m as follows:

[0187] F m =similarity·Φ M (TV coarse ;TV fine ).

[0188] Step B4: Cascaded aggregated text fine-grained representation vector T fine and the aggregated text coarse-grained representation vector T coarse , the text unimodal representation vector F is obtained through projection head mapping t , the cascaded aggregated image fine-grained representation vector V fine and the aggregated image coarse-grained representation vector V coarse , the image unimodal representation vector F is obtained through projection head mapping v .

[0189] In this embodiment, the specific implementation method of step B4 is: respectively convert the text fine-grained representation vector T fine and the text coarse-grained representation vector T coarse Aggregate into feature vectors and concatenate them, then input into the projection head consisting of two fully connected networks with ReLU activation function to obtain the text unimodal representation vector F t ; Represent the aggregated image into a fine-grained vector V fine and the aggregated image coarse-grained representation vector V coarse Splicing, input the projection head with the same structure as the text branch but different parameters to obtain the image unimodal representation vector F v :

[0190]

[0191] Step B5: Use the collaborative attention mechanism to realize the supplementary evidence representation vector C sup and the text unimodal representation vector F t , image unimodal representation vector F v , cross-modal representation vector F m The interaction of text comments is obtained by tc , image comment interaction representation vector F vc , cross-modal comment interaction representation vector F mc .

[0192] In this embodiment, the specific implementation method of step B5 is:

[0193] Text unimodal representation vector F t and the supplementary evidence representation vector C sup Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output feature is obtained, a review-attention weighted text feature F ts and a text-attention-weighted review feature F st , and then fed into the feedforward neural network FFN3 to obtain the text comment interaction representation vector F tc :

[0194]

[0195] Where W t ,

[0196] Image unimodal representation vector F v and the supplementary evidence representation vector C sup Mapped to the same dimension by a linear layer As U1 and I2 are input into a shared weighted CT in different orders, the output feature is obtained, a review-attention weighted image feature F vs and an image attention weighted review feature F sv , and then fed into the feedforward neural network FFN4 to obtain the image comment interaction representation vector F vc :

[0197]

[0198] Where W t ,

[0199] Cross-modal representation vector F m and the supplementary evidence representation vector C sup Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output feature is obtained, a review-attention weighted image feature F ms and an image attention weighted review feature F sm , and then fed into the feedforward neural network FFN5 to obtain the cross-modal comment interaction representation vector F mc :

[0200]

[0201] Where W t,

[0202] Step B6: Concatenate text-comment interaction representation vector F tc , image comment interaction representation vector F vc , cross-modal comment interaction representation vector F mc Get the multimodal representation vector F out , aligning the multimodal representation vector F through supervised contrastive learning out .

[0203] In this embodiment, the specific implementation method of step B6 is:

[0204] Connect text comment interaction representation vector F tc , image comment interaction representation vector F vc , cross-modal comment interaction representation vector F mc Send it to the feedforward neural network FFN6 to get the multimodal representation vector F out , aligning the multimodal representation vector F through supervised contrastive learning out , randomly select sample g as the anchor point, regard p with the same label as the anchor point as the positive multimodal sample, regard a with different label from the anchor point as the negative multimodal sample, and the supervised contrast loss is expressed as L align :

[0205] F out =FFN6(F tc ; F vc ; F mc )

[0206]

[0207] Among them, G is the small training set extracted during the training process, A(g) is the negative sample set in the small training set, and P(g) is the positive sample set in the small training set; sim(·) represents the cosine similarity function, l g Represents the multimodal representation vector of sample g, l p Represents the multimodal representation vector of sample p, l a represents the multimodal characterization vector of sample a, and τ is the temperature coefficient.

[0208] Step B7: Transform the multimodal representation vector F out and the mainstream opinion representation vector C main The joint input is sent to the fully connected layer to obtain the prediction result of the post; then according to the target loss function, the gradient of each parameter in the multimodal rumor detection model is calculated by using the back propagation method, and all parameters of the model are iteratively updated using back propagation.

[0209] In this embodiment, step B7 specifically includes the following steps:

[0210] Step B71: The multimodal representation vector F obtained in step B6 is out and the mainstream opinion representation vector C main Splice the input into the classifier to get the prediction result of the post:

[0211]

[0212] Among them, FNC(·) is a rumor classifier, which consists of a two-layer fully connected network with ReLu activation function.

[0213] The cross entropy loss function is used as the classification loss function to calculate the loss value. The calculation formula is as follows:

[0214]

[0215] Among them, y is the true label.

[0216] Step B72: Alignment loss L align and classification loss L classify Together they are used as the final loss function of the model. The gradient optimization algorithm Adam is used as the optimizer to iteratively update all parameters of the model through back propagation to minimize the loss function to train the model. The calculation formula is as follows:

[0217] loss = β a L align +β c L classify

[0218] Among them, β a and β c Used to balance the two losses.

[0219] Step B8: When the loss value change generated by each iteration of the multimodal rumor detection model is less than the given threshold or reaches the maximum number of iterations, the training process of the model is terminated.

[0220] This embodiment also provides a multimodal rumor detection system that integrates multi-granularity features, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0221] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0222] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0223] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0224] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0225] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A multimodal rumor detection method integrating multi-granularity features, characterized in that: The following steps are involved: Step A: Collect multimedia posts in social media, extract text, images and comments from the posts, and annotate the authenticity labels of the posts to construct a training dataset DS; Step B: Construct a multimodal rumor detection model M integrating multi-granular features. The multimodal rumor detection model M extracts multi-granular features from posts, deeply integrates image and text features, and resolves image and text ambiguity through image and text similarity weights. It also fully mines the information of comments, utilizes comments from different angles, and finally captures similar information of posts of the same category through comparative learning; Use the training dataset DS to train the multimodal rumor detection model M; Step C: Input the text, images, and comments of the undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts.

2. The multimodal rumor detection method integrating multi-granularity features according to claim 1 is characterized in that: The step B specifically comprises the following steps: Step B1: Process all training samples in the training dataset DS to obtain supplementary evidence, mainstream opinions, and text representations of images in the comments, and use the AMR graph tool to generate a graph representation G of the text. t ; Step B2: Use BERT to encode the text representation of text, supplementary evidence, mainstream opinion and image respectively to obtain the text fine-grained representation vector T fine , supplement evidence representation vector C sup , mainstream opinion representation vector C main , image coarse-grained representation vector V coarse , use Swin-T to encode the image to obtain the image fine-grained representation vector V fine , use BERT to encode the text graph representation and input GCN to obtain the coarse-grained text representation vector T coarse ; Step B3: Cascade image coarse-grained representation vector V coarse and the text coarse-grained representation vector T coarse Get the coarse-grained representation vector TV of the image and text coarse , through the collaborative attention mechanism to fuse the text fine-grained representation vector T fine and the image fine-grained representation vector V fine , and obtain the fine-grained representation vector TV of the image and text fine , cascaded image and text coarse-grained representation vector TV coarse and fine-grained representation vector TV fine , and multiply it by the image-text similarity weight to get the cross-modal representation vector F m ; Step B4: Cascaded aggregated text fine-grained representation vector T fine and the aggregated text coarse-grained representation vector T coarse , the text unimodal representation vector F is obtained through projection head mapping t , the cascaded aggregated image fine-grained representation vector V fine and the aggregated image coarse-grained representation vector V coarse , the image unimodal representation vector F is obtained through projection head mapping v ; Step B5: Use the collaborative attention mechanism to realize the supplementary evidence representation vector C sup and the text unimodal representation vector F t , image unimodal representation vector F v , cross-modal representation vector F m The interaction of text comments is obtained by tc , image comment interaction representation vector F vc , cross-modal comment interaction representation vector F mc ; Step B6: Concatenate text-comment interaction representation vector F tc , image comment interaction representation vector F vc , cross-modal comment interaction representation vector F mc Get the multimodal representation vector F out , aligning the multimodal representation vector F through supervised contrastive learning out ; Step B7: Transform the multimodal representation vector F out and mainstream opinion representation vector C main The combined input is fed into the fully connected layer to obtain the prediction result of the post; then, according to the target loss function, the gradient of each parameter in the multimodal rumor detection model is calculated by using the back propagation method, and all the parameters of the model are iteratively updated using back propagation; Step B8: When the loss value change generated by each iteration of the multimodal rumor detection model is less than the given threshold or reaches the maximum number of iterations, the training process of the model is terminated.

3. The multimodal rumor detection method integrating multi-granularity features according to claim 2 is characterized in that: The step B1 specifically comprises the following steps: Step B11: Traverse the sample data of the training data set DS, perform word segmentation on the text and comment content of the post, remove stop words and special symbols, and delete the posts containing invalid images; each sample post in DS s i Represented as s i =(t i ,r i ,v i ,a); where t i and u i Represent the text and image of the post, r i Indicates posts i The corresponding set of comments, Indicates posts i The jth comment of the post; a is the authenticity label corresponding to the post, a∈{non-rumor, rumor}; The text of the post i It is expressed as: in, The text content of the post i The kth word in the post, k = 1, 2, ..., n, n is the post text t i The number of words; Comments on the post It is expressed as: in, For comment content The kth word in, i = 1, 2, ..., m, m is the comment r j The number of words; Step B12: Retrieve comments from social media posts and obtain a set of supplementary evidence from the comments Indicates posts i Article k of the supplementary evidence; The mainstream opinions are extracted through a similarity-based selection mechanism; first, BERT is used to classify the comments Encode and get the comment representation vector It is expressed as: in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector; By calculating the similarity of each comment with other comments, we get an inter-sequence attention matrix Where N is the number of comments under a social media post; U m,n Represents the similarity between comment m and comment n, formalized as: After obtaining the inter-sequence attention matrix, calculate the sum of similarities between each comment and all other comments, and select the top-K comments with the highest sum of similarities as the mainstream opinions. Indicates posts i The kth mainstream opinion; Step B13: Use the multimodal large model VisualGLM to translate the image and obtain the text description vt of the image i , i represents the image of the i-th social media post; Step B14: Generate an abstract semantic graph representation G of the text of the post using the AMR graph interpreter t , using BERT for G t Encode to get the text abstract semantic representation H t .

4. The multimodal rumor detection method integrating multi-granularity features according to claim 3 is characterized in that: The step B2 specifically comprises the following steps: Step B21: The text obtained in step B11 Encoded by BERT, we get the text t i The fine-grained representation vector T fine ; T fine It is expressed as: in, Represents the kth word The output of the last hidden state in BERT encoding, b represents the dimension of the word embedding representation vector; v i Input into Swin-T for encoding to obtain the image fine-grained representation vector V fine : in, represents the hidden state corresponding to the kth window in the input at the output of the last layer of the model, n p is the number of batches in Swin-T, s is the hidden size of the visual embedding; Step B22: Supplementary evidence set obtained in step B12 Encoding through BERT to obtain additional evidence The initial characterization vector in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector; The average vector of the initial representation vectors of all supplementary evidence is represented as the final supplementary evidence representation vector C sup : The mainstream opinion set obtained in step B12 Encoding through BERT to obtain mainstream views The initial characterization vector in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector; The average vector of the initial representation vectors of all mainstream opinions is represented as the final mainstream opinion representation vector C main : Step B23: Text description vt of the image obtained in step B13 i Use BERT to encode and get the coarse-grained representation vector V of the image coarse ; V coarse It is expressed as: in, Represents the kth word The output of the last hidden state in BERT encoding, d represents the dimension of the word embedding representation vector; Step B24: The text abstract semantic representation H obtained in step B14 is t Input graph convolutional neural network GCN to obtain the coarse-grained text representation vector T coarse ; T coarse It is expressed as: T coarse =GCN(H t )。 5. The multimodal rumor detection method integrating multi-granularity features according to claim 4 is characterized in that: The step B3 specifically comprises the following steps: Step B31: The text fine-grained representation vector T obtained in step B2 fine and the image fine-grained representation vector V fine , using the Co-Attention Transformer CT for fusion; CT consists of a multi-head attention network and a feed-forward neural network; Denote different modality inputs as I1 and I2; in CT, I1 is used as query Q, and I2 is used as key K and value V. CT treats each head h i The collaborative attention matrix is ​​calculated as: Among them, Softmax represents the activation function, and the projection matrix Corresponding to Q, K, V, d respectively h =d m / m,d m is the dimension of CT, m is the number of heads; The multi-head attention H is the projection of all collaborative attention matrices after cascading: H=(h1;h2;...;h m )W o Among them, the symbol; represents the cascade operation, the projection matrix Then H and I1 pass through a two-layer normalized feed-forward neural network FFN to obtain an attention-based multimodal representation H': H′=Norm(I1+FFN(Norm(I1+HI1))) The multimodal representation H' is average-pooled into a feature vector F as the output of CT; Feature T fine and feature V fine Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output features are obtained, including a visual attention weighted text feature F vt and a text-attention-weighted visual feature F tv : in, Step B32: Cascade the image coarse-grained representation vector V obtained in step B2 coarse and the text coarse-grained representation vector T coarse Send it to the feedforward neural network FFN1 to get the coarse-grained representation vector TV of the image and text coarse , the visual attention weighted text feature F in the cascade step B31 vt and the text-attention-weighted visual features F tv Send it to the feedforward neural network FFN2 to get the fine-grained representation vector TV of the image and text fine : Step B33: Calculate the image coarse-grained representation vector V coarse and the text coarse-grained representation vector T coarse The cosine similarity is regarded as the image-text similarity weight, and is used as the weighted coefficient of the cross-modal feature to guide the learning process of the classifier; the cosine similarity is calculated as follows: Cascaded image and text coarse-grained representation vector TV coarse and fine-grained representation vector TV fine , and input the projection head Φ M Generate a cross-modal representation in ; each element in the output vector is multiplied by the image-text similarity weight, and the final cross-modal representation vector F m as follows: F m =similarity·Φ M (TV coarse ;TV fine )。 6. The multimodal rumor detection method integrating multi-granularity features according to claim 5 is characterized in that: The specific implementation method of step B4 is: respectively transform the text fine-grained representation vector T fine and the text coarse-grained representation vector T coarse Aggregate into feature vectors and concatenate them, then input into the projection head consisting of two fully connected networks with ReLU activation function to obtain the text unimodal representation vector F t ; The aggregated image fine-grained representation vector V fine and the aggregated image coarse-grained representation vector V coarse Splicing, input the projection head with the same structure as the text branch but different parameters to obtain the image unimodal representation vector F v :

7. The multimodal rumor detection method integrating multi-granularity features according to claim 6 is characterized in that: The specific implementation method of step B5 is: Text unimodal representation vector F t and the supplementary evidence representation vector C sup Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output feature is obtained, a review-attention weighted text feature F ts and a text-attention-weighted review feature F st , and then fed into the feedforward neural network FFN3 to obtain the text comment interaction representation vector F tc : Where W t , Image unimodal representation vector F v and the supplementary evidence representation vector C sup Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output feature is obtained, a review-attention weighted image feature F vs and an image attention weighted review feature F sv , and then fed into the feedforward neural network FFN4 to obtain the image comment interaction representation vector F vc : Where W t , Cross-modal representation vector F m and the supplementary evidence representation vector C sup Mapped to the same dimension by a linear layer As I1 and I2 are input into a shared weighted CT in different orders, the output feature is obtained, a review-attention weighted image feature F ms and an image attention weighted review feature F sm , and then fed into the feedforward neural network FFN5 to obtain the cross-modal comment interaction representation vector F mc : Where W t , 8. The multimodal rumor detection method integrating multi-granularity features according to claim 7 is characterized in that: The specific implementation method of step B6 is: Connect text comment interaction representation vector F tc , image comment interaction representation vector F vc , cross-modal comment interaction representation vector F mc Send it to the feedforward neural network FFN6 to get the multimodal representation vector F out , aligning the multimodal representation vector F through supervised contrastive learning out , randomly select sample g as the anchor point, regard p with the same label as the anchor point as the positive multimodal sample, regard a with different label from the anchor point as the negative multimodal sample, and the supervised contrast loss is expressed as L align : F out =FFN6(F tc ;F vc ;F mc ) Among them, G is the small training set extracted during the training process, A(g) is the negative sample set in the small training set, and P(g) is the positive sample set in the small training set; sim(·) represents the cosine similarity function, l g Represents the multimodal representation vector of sample g, l p Represents the multimodal representation vector of sample p, l a represents the multimodal characterization vector of sample a, and τ is the temperature coefficient.

9. The multimodal rumor detection method integrating multi-granularity features according to claim 8 is characterized in that: The step B7 specifically includes the following steps: Step B71: The multimodal representation vector F obtained in step B6 is out and mainstream opinion representation vector C main Splice the input into the classifier to get the prediction result of the post: Among them, FNC(·) is a rumor classifier, which consists of a two-layer fully connected network with ReLu activation function; The cross entropy loss function is used as the classification loss function to calculate the loss value. The calculation formula is as follows: Among them, y is the true label; Step B72: Alignment loss L align and classification loss L classify Together they are used as the final loss function of the model. The gradient optimization algorithm Adam is used as the optimizer to iteratively update all parameters of the model through back propagation to minimize the loss function to train the model. The calculation formula is as follows: loss=β a L align +b c L classify Among them, β a and β c Used to balance the two losses.

10. A multimodal rumor detection system integrating multi-granularity features, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method steps as described in any one of claims 1 to 9 can be implemented.

Images