Multimodal rumor detection method and system fusing multi-granularity features

Abstract

The present application relates to a kind of multi-modal rumor detection method and system of fusion multi-granularity feature, the method includes: in social media, collect multimedia post, extract text, image and comment in post, and the authenticity label of post is labeled, to construct training data set in this way;Multi-modal rumor detection model of fusion multi-granularity feature is constructed, multi-modal rumor detection model extracts the multi-granularity feature in post, depth fusion graphic text feature, and through graphic text similarity weight realizes the resolution of graphic text ambiguity, also fully excavate the information of comment, utilize comment from different angles, finally through contrast learning, capture the similar information of social media post of same class;Multi-modal rumor detection model is trained using training data set;The text, image and comment of undetected multimedia post are input into trained multi-modal rumor detection model, obtain the authenticity label of multimedia post.The method and system are conducive to improving the accuracy and reliability of rumor detection.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, specifically relating to a multimodal rumor detection method and system that integrates multi-granularity features. Background Technology

[0002] Rumor detection, also known as fake news detection, is an important research task in the field of Natural Language Processing (NLP). With the rise of social media, platforms like Weibo and Twitter have become indispensable communication tools. However, due to the rapid and widespread dissemination of information on social media, rumors have gradually become a significant concern. In social media, a rumor is an unverified statement of information, often using false or fabricated images and inflammatory text to mislead users. Rumors frequently cause substantial economic losses and serious harm to individuals and society. Although various social media organizations have launched rumor-debunking platforms to strengthen the control of rumors, these platforms still require significant human and material resources to collect information and determine the veracity of rumors. Therefore, there is an urgent need for an automated method that can effectively detect rumors.

[0003] Due to the significant performance advantages of deep learning methods compared to traditional machine learning methods, an increasing number of researchers are applying deep learning to the problem of rumor detection. Liu et al. proposed a rumor detection model based on convolutional neural networks. They vectorized the text content of rumors and used convolutional neural networks to mine deep text features, which include some features that are not easily discovered by humans, thus improving performance. Ma et al. were the first to apply recurrent neural networks to rumor detection. This algorithm can learn the hidden representations of Weibo posts and use these representations to show the contextual information of related posts changing over time. Experimental results show the significant advantages of deep learning models in this field. Bi et al. modeled the information transmission network of Weibo as a heterogeneous graph with various semantic information, and then constructed a graph-based rumor detection model, using attention layers to capture and aggregate its semantic information. Dou et al. considered the impact of user preferences on the rumor detection task, showing that a user's history and social activities (e.g., posts) provide rich information about the user's news preferences and have great potential to promote rumor detection.

[0004] Currently, posts on social media often combine text and images, with these two different modalities providing unique and complementary information. Research has shown that images can help users better understand and remember textual content by providing rich visual information. Therefore, in rumor detection tasks, it is necessary to utilize both textual and image information to identify rumors. Khattar et al. proposed a multimodal variational autoencoder for rumor detection to learn shared representations for both text and image modalities. Zhang et al. designed a multimodal, multi-task learning framework by introducing a position task. Zhou et al. considered the correlation between textual and visual information, identifying rumors based on mismatches between text and images. Wang et al. proposed KMGCN, integrating textual, visual, and knowledge information into a unified framework to model semantic representations, further improving accuracy.

[0005] While existing rumor detection methods have achieved some success, they still have certain limitations, especially in multimodal rumor detection research. They lack sufficient utilization of text and image data, as well as effective fusion between the two modalities. Furthermore, they fail to effectively measure the ambiguity of images and text, leading to the introduction of noise. Secondly, for comments, they simply include all comments without studying the inherent relationships between them. These problems can all contribute to a decline in rumor detection performance. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal rumor detection method and system that integrates multi-granularity features, which is beneficial to improving the accuracy and reliability of rumor detection.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a multimodal rumor detection method that integrates multi-granularity features, comprising the following steps:

[0008] Step A: Collect multimedia posts from social media, extract the text, images, and comments from the posts, and label the posts with authenticity tags to construct the training dataset DS;

[0009] Step B: Construct a multimodal rumor detection model M that integrates multi-granular features. The multimodal rumor detection model M extracts multi-granular features from posts, deeply integrates image and text features, and resolves image and text ambiguity through image-text similarity weights. It also fully mines the information in comments, utilizes comments from different angles, and finally captures similar information of social media posts of the same category through comparative learning. The multimodal rumor detection model M is trained using the training dataset DS.

[0010] Step C: Input the text, images, and comments of the undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts.

[0011] Furthermore, step B specifically includes the following steps:

[0012] Step B1: Process all training samples in the training dataset DS to obtain supplementary evidence, mainstream viewpoints, and textual representations of images in the comments. Use the AMR graph tool to generate a graphical representation G of the text. t ;

[0013] Step B2: Use BERT to encode the text representations of the text, supplementary evidence, mainstream viewpoints, and images respectively, to obtain the fine-grained text representation vector T. fine Supplementary evidence representation vector C sup Mainstream viewpoint representation vector C main Image coarse-grained representation vector V coarse The image is encoded using Swin-T to obtain the fine-grained image representation vector V. fine BERT is used to encode the text graph representation, and the input to GCN yields a coarse-grained text representation vector T. coarse ;

[0014] Step B3: Concatenate image coarse-grained representation vector V coarse and text coarse-grained representation vector T coarse Obtain the coarse-grained representation vector TV of the image and text coarse By fusing the fine-grained text representation vector T through a collaborative attention mechanism fine and image fine-grained representation vector V fine The fine-grained representation vector TV of the image and text is obtained. fine Cascaded coarse-grained representation vector TV coarse And fine-grained representation vector TV fine The cross-modal representation vector F is obtained by multiplying it by the image-text similarity weight. m ;

[0015] Step B4: Cascaded aggregation of fine-grained text representation vectors T fine and the aggregated text coarse-grained representation vector T coarse The text single-modal representation vector F is obtained through projection head mapping. t Cascaded aggregation of fine-grained image representation vectors V fine and the aggregated image coarse-grained representation vector V coarse The image single-modal representation vector F is obtained through projection head mapping. v ;

[0016] Step B5: Implement the supplementary evidence representation vector C using a collaborative attention mechanism.sup With the text single-modal representation vector F t Image single-modal representation vector F v Cross-modal representation vector F m The interaction yields the text comment interaction representation vector F. tc Image comment interaction representation vector F vc Cross-modal comment interaction representation vector F mc ;

[0017] Step B6: Cascaded text comment interaction representation vector F tc Image comment interaction representation vector F vc Cross-modal comment interaction representation vector F mc Obtain the multimodal representation vector F out Aligning multimodal representation vectors F through supervised contrastive learning out ;

[0018] Step B7: Convert the multimodal representation vector F out and the representation vector C of mainstream viewpoints main The combined input is fed into a fully connected layer to obtain the prediction results of the posts; then, based on the target loss function, the gradients of each parameter in the multimodal rumor detection model are calculated using the backpropagation method, and all parameters of the model are iteratively updated using backpropagation.

[0019] Step B8: When the change in loss value generated by the multimodal rumor detection model in each iteration is less than the given threshold or the maximum number of iterations is reached, the training process of the model is terminated.

[0020] Furthermore, step B1 specifically includes the following steps:

[0021] Step B11: Iterate through the sample data in the training dataset DS, perform word segmentation on both the text of the posts and the comment content, remove stop words and special characters, and delete posts containing invalid images; for each sample post in DS...

[0022] s i Represented as s i =(t i ,r i ,v i ,a); where t i and u i r represents the text and image of the post, respectively. i Indicates post s i The corresponding collection of comments, Indicates post s i The j-th comment; a is the authenticity tag corresponding to the post, a∈{non-rumor, rumor};

[0023] The text of the post 表 As shown:

[0024]

[0025] in, For the post text content t i The k-th word in the text, k = 1, 2, ..., n, where n is the post text t. i The number of words;

[0026] Comments on the post Represented as:

[0027]

[0028] in, For comment content The k-th word in the text, i = 1, 2, ..., m, where m is the comment r. j The number of words;

[0029] Step B12: Utilize social media posts to retrieve comments and obtain a set of supplementary evidence from the comments. Indicates post s i The kth piece of supplementary evidence;

[0030] Mainstream viewpoints are extracted using a similarity-based selection mechanism; BERT is first used to analyze comments. Encode the comments to obtain a comment representation vector. Represented as:

[0031]

[0032] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector;

[0033] An inter-sequence attention matrix is ​​obtained by calculating the similarity between each comment and other comments. Where N is the number of comments under a social media post; U m,n The similarity between comment m and comment n can be formalized as:

[0034]

[0035] After obtaining the inter-sequence attention matrix, the sum of similarity between each comment and all other comments is calculated. The top-K comments with the highest sum of similarity are selected as the mainstream viewpoints. Indicates post s i The kth mainstream viewpoint;

[0036] Step B13: Use the multimodal large model VisualGLM to translate the image and obtain the text description vt of the image. i , where i represents the image of the i-th social media post;

[0037] Step B14: Use the AMR graph interpreter to generate an abstract semantic graph representation G of the post's text. t Using BERT for G t Encoding is performed to obtain the abstract semantic representation H of the text. t .

[0038] Furthermore, step B2 specifically includes the following steps:

[0039] Step B21: Process the text obtained in step B11 The text t is obtained by encoding with BERT. i fine-grained representation vector T fine ;T fine Represented as:

[0040]

[0041] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, b, represents the dimension of the word embedding representation vector;

[0042] Image v in the post i The input is encoded in Swin-T to obtain the fine-grained image representation vector V. fine :

[0043]

[0044] in, This represents the hidden state of the k-th window in the input at the output of the last layer of the model, where n is the hidden state. p is the batch size in Swin-T, and s is the hidden size of the visual embedding;

[0045] Step B22: Supplementary evidence set obtained in step B12 Supplementary evidence is obtained by encoding with BERT. initial representation vector

[0046]

[0047] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector;

[0048] The average vector of the initial representation vectors of all supplementary evidence is represented as the final supplementary evidence representation vector C. sup :

[0049]

[0050] The set of mainstream viewpoints obtained in step B12 Encoding with BERT yields mainstream viewpoints. initial representation vector

[0051]

[0052] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector;

[0053] The average vector of the initial representation vectors of all mainstream viewpoints is represented as the final mainstream viewpoint representation vector C. main :

[0054]

[0055] Step B23: Text description of the image obtained in step B13 (vt) i Encoding using BERT yields the image coarse-grained representation vector V. coarse V coarse Represented as:

[0056]

[0057] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector;

[0058] Step B24: Abstract the semantic representation H of the text obtained in step B14. t The input graph convolutional neural network (GCN) produces a coarse-grained text representation vector T. coarse ;T coarse Represented as:

[0059] T coarse =GCN(H t ).

[0060] Furthermore, step B3 specifically includes the following steps:

[0061] Step B31: Process the fine-grained text representation vector T obtained in step B2. fine and image fine-grained representation vector V fine Using collaborative attention transformer CT for fusion;

[0062] CT consists of a multi-head attention network and a feedforward neural network;

[0063] Different modal inputs are represented as I1 and I2; in CT, U1 is used as query Q, and U2 is used as key K and value V. CT will assign each head h i The collaborative attention matrix is ​​calculated as follows:

[0064]

[0065] Where Softmax represents the activation function and the projection matrix is... Corresponding to Q, K, V, and d respectively h =d m / m,d m 'm' represents the dimension of the CT scan, and 'm' represents the number of heads.

[0066] Multi-head attention H is the projection of the concatenation of all collaborative attention matrices:

[0067] H = (h1; h2; ...; h m W o

[0068] Where, the symbol ; denotes cascade operation and projection matrix.

[0069] Then, H and I1 are passed through a two-layer normalized feedforward neural network FFN to obtain an attention-based multimodal representation H':

[0070] H'=Norm(I1+FFN(Norm(I1+HI1)))

[0071] The multimodal representation H' is average-pooled into a feature vector F as the output of CT;

[0072] Feature T fine and feature V fine They are mapped to the same dimension through a linear layer. I1 and I2 are input into a shared weighted CT in different orders to obtain output features, including a visual attention-weighted text feature F. vt A visual feature F weighted by text attention tv :

[0073]

[0074] in,

[0075] Step B32: Cascade the image coarse-grained representation vector V obtained in step B2 coarse and text coarse-grained representation vector T coarse The image is fed into a feedforward neural network FFN1 to obtain the coarse-grained representation vector TV. coarse The visual attention-weighted text features F in the cascade step B31 vt Visual features F weighted by text attention tv The vector TV is fed into a feedforward neural network FFN2 to obtain a fine-grained representation of the image and text. fine :

[0076]

[0077] Step B33: Calculate the image coarse-grained representation vector V coarse and text coarse-grained representation vector T coarse The cosine similarity is considered as a weight in the image-text similarity ratio and serves as a weighting coefficient for cross-modal features to guide the classifier's learning process. The cosine similarity is calculated as follows:

[0078]

[0079] Cascaded coarse-grained representation vector TV coarse And fine-grained representation vector TV fine And input the projection head Φ M A cross-modal representation is generated; each element in the output vector is multiplied by a text-image similarity weight, resulting in the final cross-modal representation vector F. m as follows:

[0080] F m =similarity·Φ M (TV coarse TV fine ).

[0081] Furthermore, the specific implementation method of step B4 is as follows: The text fine-grained representation vector T is processed at the word level. fine and text coarse-grained representation vector T coarse The features are aggregated and concatenated, then input into a projection head consisting of two fully connected networks with ReLU activation functions to obtain the text unimodal representation vector F. t The aggregated image fine-grained representation vector V fine and the aggregated image coarse-grained representation vector Vcoarse By concatenating the images, the single-modal representation vector F is obtained from the input projection head, which has the same branching structure as the text but different parameters. v :

[0082]

[0083] Furthermore, the specific implementation method of step B5 is as follows:

[0084] Text unimodal representation vector F t and supplementary evidence representation vector C sup They are mapped to the same dimension through a linear layer. I1 and I2 are input into a shared weighted CT in different orders to obtain the output features, a comment attention-weighted text feature F. ts A text attention-weighted comment feature F st The text comment interaction representation vector F is obtained by feeding it into the feedforward neural network FFN3. tc :

[0085]

[0086] Among them W t ,

[0087] Image single-modal representation vector F v and supplementary evidence representation vector C sup They are mapped to the same dimension through a linear layer. I1 and I2 are input into a shared weighted CT in different orders to obtain output features, a comment attention-weighted image feature F. vs A comment feature F weighted by image attention sv The image comment interaction representation vector F is obtained by feeding it into the feedforward neural network FFN4. vc :

[0088]

[0089] Among them W t ,

[0090] Cross-modal representation vector F m and supplementary evidence representation vector C sup They are mapped to the same dimension through a linear layer. I1 and I2 are input into a shared weighted CT in different orders to obtain output features, a comment attention-weighted image feature F. ms A comment feature F weighted by image attention smThe cross-modal comment interaction representation vector F is obtained by feeding it into the feedforward neural network FFN5. mc :

[0091]

[0092] Among them W t ,

[0093] Furthermore, the specific implementation method of step B6 is as follows:

[0094] Connect the text comment interaction representation vector F tc Image comment interaction representation vector F vc Cross-modal comment interaction representation vector F mc The multimodal representation vector F is obtained by feeding it into the feedforward neural network FFN6. out Aligning multimodal representation vectors F through supervised contrastive learning out A random sample g is selected as the anchor point. p, which shares the same label as the anchor point, is considered a positive multimodal sample, and a, which has a different label, is considered a negative multimodal sample. The supervised contrastive loss is denoted as L. align :

[0095] F out =FFN6(F tc ;F vc ;F mc )

[0096]

[0097] Where G is the small training set extracted during training, A(g) is the negative sample set in the small training set, and P(g) is the positive sample set in the small training set; sim(·) represents the cosine similarity function, l g Let l represent the multimodal representation vector of sample g. p Let l represent the multimodal representation vector of sample p. a Let τ represent the multimodal characterization vector of sample a, where τ is the temperature coefficient.

[0098] Furthermore, step B7 specifically includes the following steps:

[0099] Step B71: Convert the multimodal representation vector F obtained in step B6 into... out and the representation vector C of mainstream viewpoints main The concatenated input is fed into the classifier to obtain the post prediction result:

[0100]

[0101] Among them, FNC(·) is a rumor classifier, which consists of a two-layer fully connected network with ReLU activation function;

[0102] The cross-entropy loss function is used as the classification loss function to calculate the loss value. The calculation formula is as follows:

[0103]

[0104] Where y is the real label;

[0105] Step B72: Alignment loss L align And classification loss L classify Together, these parameters form the final loss function of the model. The Adam gradient optimization algorithm is used as the optimizer, and all parameters are iteratively updated through backpropagation to minimize the loss function during model training. The calculation formula is as follows:

[0106] loss = β a L align +β c L classify

[0107] Where, β a and β c It is used to balance the two types of losses.

[0108] The present invention also provides a multimodal rumor detection system that integrates multiple granular features, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-mentioned method steps.

[0109] Compared with the prior art, the present invention has the following beneficial effects:

[0110] 1. By combining the multimodal large model VisualGLM, BERT, and Swin-T, we have achieved multimodal and multigranular feature extraction of posts, which fully explores the information of images and text in posts, enabling the model to more comprehensively understand and analyze post content.

[0111] 2. By using a retrieval and selection mechanism, supplementary evidence and mainstream viewpoints are obtained from comments, and the information in the comments of posts is fully explored from different angles, which enriches the information sources, improves the model's ability to identify rumors, and enables it to more accurately judge the authenticity of post content;

[0112] 3. By calculating the similarity between images and text, the use of cross-modal fusion features is guided, which effectively alleviates the impact of image-text ambiguity, making the model more robust in cases of weak image-text correlation, enhancing the model's performance when processing posts with weak image-text correlation, and improving the overall reliability of detection.

[0113] 4. By contrastive learning, similar information of rumors of the same category is captured, which helps to correctly classify marginal samples, improves the model's ability to distinguish different types of rumors, and significantly improves the model's accuracy and robustness. It performs particularly well for complex rumor detection tasks. Attached Figure Description

[0114] Figure 1 This is a flowchart illustrating the method implementation of an embodiment of the present invention;

[0115] Figure 2 This is an architecture diagram of the multimodal rumor detection model M that integrates multiple granular features in an embodiment of the present invention. Detailed Implementation

[0116] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0117] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0118] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0119] like Figure 1 As shown in the figure, this embodiment provides a multimodal rumor detection method that integrates multiple granular features, including the following steps:

[0120] Step A: Collect multimedia posts from social media, extract the text, images, and comments from the posts, and label the posts with authenticity tags to construct the training dataset DS;

[0121] Step B: Construct a multimodal rumor detection model M that integrates multi-granular features. This model extracts multi-granular features from posts, enabling a more comprehensive and detailed understanding of the post content. It deeply integrates image and text features, enhancing the overall understanding of the content. Furthermore, it resolves ambiguity through image-text similarity weights, making the model more robust against posts with weak image-text correlation. Additionally, it fully mines information from comments, utilizing them from different angles to further enrich and strengthen content understanding. Finally, it captures similar information from similar social media posts through comparative learning, further improving the model's accuracy. The multimodal rumor detection model M is trained using the training dataset DS.

[0122] Step C: Input the text, images, and comments of the undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts.

[0123] Figure 2 This is an architecture diagram of the multimodal rumor detection model M that integrates multiple granular features in this embodiment. In this embodiment, the specific implementation steps of step B are as follows.

[0124] Step B1: Process all training samples in the training dataset DS to obtain supplementary evidence, mainstream viewpoints, and textual representations of images in the comments. Use the AMR graph tool to generate a graphical representation G of the text. t .

[0125] In this embodiment, step B1 specifically includes the following steps:

[0126] Step B11: Iterate through the sample data in the training dataset DS, perform word segmentation on both the text of the posts and the comment content, remove stop words and special characters, and delete posts containing invalid images; for each sample post in DS...

[0127] s i Represented as s i =(t i ,r i ,v i ,a); where t i and u i r represents the text and image of the post, respectively. i Indicates post s i The corresponding collection of comments, Indicates post s i The j-th comment; a is the authenticity tag corresponding to the post, a∈{non-rumor, rumor}.

[0128] The text of the post i Represented as:

[0129]

[0130] in, For the post text content t i The k-th word in the text, k = 1, 2, ..., n, where n is the post text t. i The number of words.

[0131] Comments on the post Represented as:

[0132]

[0133] in, For comment content The k-th word in the text, i = 1, 2, ..., m, where m is the comment r. j The number of words.

[0134] Step B12: Utilize social media posts to retrieve comments and obtain a set of supplementary evidence from the comments. Indicates post s i The kth piece of supplementary evidence.

[0135] Mainstream viewpoints are extracted using a similarity-based selection mechanism; BERT is first used to analyze comments. Encode the comments to obtain a comment representation vector. Represented as:

[0136]

[0137] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector.

[0138] An inter-sequence attention matrix is ​​obtained by calculating the similarity between each comment and other comments. Where N is the number of comments under a social media post; U m,n The similarity between comment m and comment n can be formalized as:

[0139]

[0140] After obtaining the inter-sequence attention matrix, the sum of similarity between each comment and all other comments is calculated. The top-K comments with the highest sum of similarity are selected as the mainstream viewpoints. Indicates post s i The kth mainstream viewpoint.

[0141] Step B13: Use the multimodal large model VisualGLM to translate the image and obtain the text description vt of the image. i , where i represents the image of the i-th social media post.

[0142] Step B14: Use the AMR graph interpreter to generate an abstract semantic graph representation G of the post's text. t Using BERT for G t Encoding is performed to obtain the abstract semantic representation H of the text. t .

[0143] Step B2: Use BERT to encode the text representations of the text, supplementary evidence, mainstream viewpoints, and images respectively, to obtain the fine-grained text representation vector T. fine Supplementary evidence representation vector C sup Mainstream viewpoint representation vector C main Image coarse-grained representation vector V coarse The image is encoded using Swin-T to obtain the fine-grained image representation vector V. fine BERT is used to encode the text graph representation, and the input to GCN yields a coarse-grained text representation vector T. coarse .

[0144] In this embodiment, step B2 specifically includes the following steps:

[0145] Step B21: Process the text obtained in step B11 The text t is obtained by encoding with BERT. i fine-grained representation vector T fine ;T fine Represented as:

[0146]

[0147] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, b, represents the dimension of the word embedding representation vector.

[0148] Image v in the post i The input is encoded in Swin-T to obtain the fine-grained image representation vector V. fine :

[0149]

[0150] in, This represents the hidden state of the k-th window in the input at the output of the last layer of the model, where n is the hidden state. p 's' is the batch size in Swin-T, and 's' is the hidden size of the visual embedding.

[0151] Step B22: Supplementary evidence set obtained in step B12 Supplementary evidence is obtained by encoding with BERT. initial representation vector

[0152]

[0153] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector.

[0154] The average vector of the initial representation vectors of all supplementary evidence is represented as the final supplementary evidence representation vector C. sup :

[0155]

[0156] The set of mainstream viewpoints obtained in step B12 Encoding with BERT yields mainstream viewpoints. initial representation vector

[0157]

[0158] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector.

[0159] The average vector of the initial representation vectors of all mainstream viewpoints is represented as the final mainstream viewpoint representation vector C. main :

[0160]

[0161] Step B23: Text description of the image obtained in step B13 (vt) i Encoding using BERT yields the image coarse-grained representation vector V. coarse V coarse Represented as:

[0162]

[0163] in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector.

[0164] Step B24: Abstract the semantic representation H of the text obtained in step B14. t The input graph convolutional neural network (GCN) produces a coarse-grained text representation vector T. coarse ;T coarse Represented as:

[0165] T coarse =GCN(H t ).

[0166] Step B3: Concatenate image coarse-grained representation vector V coarse and text coarse-grained representation vector T coarse Obtain the coarse-grained representation vector TV of the image and text coarse By fusing the fine-grained text representation vector T through a collaborative attention mechanism fine and image fine-grained representation vector V fine The fine-grained representation vector TV of the image and text is obtained. fine Cascaded coarse-grained representation vector TV coarse And fine-grained representation vector TV fine The cross-modal representation vector F is obtained by multiplying it by the image-text similarity weight. m .

[0167] In this embodiment, step B3 specifically includes the following steps:

[0168] Step B31: Process the fine-grained text representation vector T obtained in step B2. fine and image fine-grained representation vector V fine Cooperative attention transformer CT was used for fusion.

[0169] CT consists of a multi-head attention network and a feedforward neural network.

[0170] Different modal inputs are represented as I1 and I2; in CT, I1 is used as query Q, and I2 is used as key K and value V. CT will assign each head h i The collaborative attention matrix is ​​calculated as follows:

[0171]

[0172] Where Softmax represents the activation function and the projection matrix is... Corresponding to Q, K, V, and d respectively h =d m / m,d m 'm' represents the dimension of the CT scan, and 'm' represents the number of heads.

[0173] Multi-head attention H is the projection of the concatenation of all collaborative attention matrices:

[0174] H = (h1; h2; ...; h m W o

[0175] Where, the symbol ; denotes cascade operation and projection matrix.

[0176] Then, H and I1 are passed through a two-layer normalized feedforward neural network FFN to obtain an attention-based multimodal representation H':

[0177] H'=Norm(I1+FFN(Norm(I1+HI1)))

[0178] The multimodal representation H' is average-pooled into an eigenvector F as the output of CT.

[0179] Feature T fine and feature V fine They are mapped to the same dimension through a linear layer. I1 and I2 are input into a shared weighted CT in different orders to obtain output features, including a visual attention-weighted text feature F. vt A visual feature F weighted by text attention tv :

[0180]

[0181] in,

[0182] Step B32: Cascade the image coarse-grained representation vector V obtained in step B2 coarse and text coarse-grained representation vector T coarse The image is fed into a feedforward neural network FFN1 to obtain the coarse-grained representation vector TV. coarse The visual attention-weighted text features F in the cascade step B31 vt Visual features F weighted by text attention tv The vector TV is fed into a feedforward neural network FFN2 to obtain a fine-grained representation of the image and text. fine :

[0183]

[0184] Step B33: Calculate the image coarse-grained representation vector V coarse and text coarse-grained representation vector T coarse The cosine similarity is considered as a weight in the image-text similarity ratio and serves as a weighting coefficient for cross-modal features to guide the classifier's learning process. The cosine similarity is calculated as follows:

[0185]

[0186] Cascaded coarse-grained representation vector TV coarse And fine-grained representation vector TV fine And input the projection head Φ M A cross-modal representation is generated; each element in the output vector is multiplied by a text-image similarity weight, resulting in the final cross-modal representation vector F. m as follows:

[0187] F m =similarity·Φ M (TV coarse TV fine ).

[0188] Step B4: Cascaded aggregation of fine-grained text representation vectors T fine and the aggregated text coarse-grained representation vector T coarse The text single-modal representation vector F is obtained through projection head mapping. t Cascaded aggregation of fine-grained image representation vectors V fine and the aggregated image coarse-grained representation vector V coarse The image single-modal representation vector F is obtained through projection head mapping. v .

[0189] In this embodiment, the specific implementation method of step B4 is as follows: The text fine-grained representation vector T is processed at the word level. fine and text coarse-grained representation vector T coarse The features are aggregated and concatenated, then input into a projection head consisting of two fully connected networks with ReLU activation functions to obtain the text unimodal representation vector F. t The aggregated image fine-grained representation vector V fine and the aggregated image coarse-grained representation vector V coarse By concatenating the images, the single-modal representation vector F is obtained from the input projection head, which has the same branching structure as the text but different parameters. v :

[0190]

[0191] Step B5: Implement the supplementary evidence representation vector C using a collaborative attention mechanism. sup With the text single-modal representation vector F t Image single-modal representation vector F v Cross-modal representation vector F m The interaction yields the text comment interaction representation vector F. tc Image comment interaction representation vector F vc Cross-modal comment interaction representation vector F mc .

[0192] In this embodiment, the specific implementation method of step B5 is as follows:

[0193] Text unimodal representation vector F t and supplementary evidence representation vector C sup They are mapped to the same dimension through a linear layer. I1 and I2 are input into a shared weighted CT in different orders to obtain the output features, a comment attention-weighted text feature F. ts A text attention-weighted comment feature F st The text comment interaction representation vector F is obtained by feeding it into the feedforward neural network FFN3. tc :

[0194]

[0195] Among them W t ,

[0196] Image single-modal representation vector F v and supplementary evidence representation vector C sup They are mapped to the same dimension through a linear layer. As U1 and I2 are input into a shared weighted CT in different orders, the output features are obtained, and a commentary attention-weighted image feature F is generated. vs A comment feature F weighted by image attention sv The image comment interaction representation vector F is obtained by feeding it into the feedforward neural network FFN4. vc :

[0197]

[0198] Among them W t ,

[0199] Cross-modal representation vector F m and supplementary evidence representation vector C sup They are mapped to the same dimension through a linear layer. I1 and I2 are input into a shared weighted CT in different orders to obtain output features, a comment attention-weighted image feature F. ms A comment feature F weighted by image attention sm The cross-modal comment interaction representation vector F is obtained by feeding it into the feedforward neural network FFN5. mc :

[0200]

[0201] Among them W t,

[0202] Step B6: Cascaded text comment interaction representation vector F tc Image comment interaction representation vector F vc Cross-modal comment interaction representation vector F mc Obtain the multimodal representation vector F out Aligning multimodal representation vectors F through supervised contrastive learning out .

[0203] In this embodiment, the specific implementation method of step B6 is as follows:

[0204] Connect the text comment interaction representation vector F tc Image comment interaction representation vector F vc Cross-modal comment interaction representation vector F mc The multimodal representation vector F is obtained by feeding it into the feedforward neural network FFN6. out Aligning multimodal representation vectors F through supervised contrastive learning out A random sample g is selected as the anchor point. p, which shares the same label as the anchor point, is considered a positive multimodal sample, and a, which has a different label, is considered a negative multimodal sample. The supervised contrastive loss is denoted as L. align :

[0205] F out =FFN6(F tc ;F vc ;F mc )

[0206]

[0207] Where G is the small training set extracted during training, A(g) is the negative sample set in the small training set, and P(g) is the positive sample set in the small training set; sim(·) represents the cosine similarity function, l g Let l represent the multimodal representation vector of sample g. p Let l represent the multimodal representation vector of sample p. a Let τ represent the multimodal characterization vector of sample a, where τ is the temperature coefficient.

[0208] Step B7: Convert the multimodal representation vector F out and the representation vector C of mainstream viewpoints main The combined input is fed into a fully connected layer to obtain the prediction results of the posts; then, based on the target loss function, the gradients of each parameter in the multimodal rumor detection model are calculated using the backpropagation method, and all parameters of the model are iteratively updated using backpropagation.

[0209] In this embodiment, step B7 specifically includes the following steps:

[0210] Step B71: Convert the multimodal representation vector F obtained in step B6 into... out and the representation vector C of mainstream viewpoints main The concatenated input is fed into the classifier to obtain the post prediction result:

[0211]

[0212] FNC(·) is a rumor classifier, consisting of a two-layer fully connected network with ReLU activation function.

[0213] The cross-entropy loss function is used as the classification loss function to calculate the loss value. The calculation formula is as follows:

[0214]

[0215] Where y is the real label.

[0216] Step B72: Alignment loss L align And classification loss L classify Together, these parameters form the final loss function of the model. The Adam gradient optimization algorithm is used as the optimizer, and all parameters are iteratively updated through backpropagation to minimize the loss function during model training. The calculation formula is as follows:

[0217] loss = β a L align +β c L classify

[0218] Where, β a and β c It is used to balance the two types of losses.

[0219] Step B8: When the change in loss value generated by the multimodal rumor detection model in each iteration is less than the given threshold or the maximum number of iterations is reached, the training process of the model is terminated.

[0220] This embodiment also provides a multimodal rumor detection system that integrates multiple granular features, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0221] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0222] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0223] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0224] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0225] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A multimodal rumor detection method integrating multi-granularity features, characterized in that, Includes the following steps: Step A: Collect multimedia posts from social media, extract the text, images, and comments from the posts, and label the posts with authenticity tags to construct the training dataset DS; Step B: Construct a multimodal rumor detection model M that integrates multi-granularity features. The multimodal rumor detection model M extracts multi-granularity features from posts, deeply integrates image and text features, and resolves image and text ambiguity through image and text similarity weights. It also fully mines the information in comments, utilizes comments from different angles, and finally captures similar information of posts of the same category through comparative learning. A multimodal rumor detection model M was trained using the training dataset DS; Step C: Input the text, images, and comments of the undetected multimedia posts into the trained multimodal rumor detection model M to obtain the authenticity labels of the multimedia posts; Step B specifically includes the following steps: Step B1: Process all training samples in the training dataset DS to obtain supplementary evidence, mainstream viewpoints, and textual representations of images in the comments. Use the AMR graph tool to generate graphical representations of the text. ; Step B2: Use BERT to encode the text representations of the text, supplementary evidence, mainstream viewpoints, and images respectively, to obtain fine-grained text representation vectors. Supplementary evidence representation vector Mainstream viewpoint representation vector Image coarse-grained representation vector The image is encoded using Swin-T to obtain a fine-grained representation vector. BERT is used to encode the text graph representation, and the input to GCN yields a coarse-grained text representation vector. ; Step B3: Concatenating coarse-grained image representation vectors and text coarse-grained representation vector Obtain the coarse-grained representation vector of the image and text. By fusing fine-grained text representation vectors through a collaborative attention mechanism and image fine-grained representation vector This yields a fine-grained representation vector of the image and text. Cascaded graphic coarse-grained representation vector and fine-grained representation vectors of text and graphics The cross-modal representation vector is obtained by multiplying it by the image-text similarity weight. ; Step B4: Cascaded Aggregation of Fine-Grained Text Representation Vectors and aggregated text coarse-grained representation vector The text single-modal representation vector is obtained through projection head mapping. Cascaded aggregation of fine-grained image representation vectors and aggregated image coarse-grained representation vector The image single-modal representation vector is obtained through projection head mapping. ; Step B5: Implement supplementary evidence representation vectors separately through a collaborative attention mechanism. Text unimodal representation vector Image single-modal representation vector Cross-modal representation vector The interaction yields a text comment interaction representation vector. Image comment interaction representation vector Cross-modal comment interaction representation vector ; Step B6: Cascaded Text Comment Interaction Representation Vector Image comment interaction representation vector Cross-modal comment interaction representation vector Obtain the multimodal representation vector Aligning multimodal representation vectors through supervised contrastive learning ; Step B7: Convert the multimodal representation vector and mainstream viewpoint representation vector The combined input is fed into a fully connected layer to obtain the prediction results of the posts; then, based on the target loss function, the gradients of each parameter in the multimodal rumor detection model are calculated using the backpropagation method, and all parameters of the model are iteratively updated using backpropagation. Step B8: When the change in loss value generated by the multimodal rumor detection model in each iteration is less than the given threshold or the maximum number of iterations is reached, the training process of the model is terminated. Step B1 specifically includes the following steps: Step B11: Iterate through the sample data in the training dataset DS, perform word segmentation on both the text of the posts and the comment content, remove stop words and special characters, and delete posts containing invalid images; for each sample post in DS... Represented as ;in, and These represent the text and image of the post, respectively. Indicates post The corresponding collection of comments, , Indicates post The j-th comment; 'a' is the authenticity tag corresponding to the post. {Not a rumor, not a rumor}; Post text Represented as: in, For post text content The kth word in n is the text of the post The number of words; Comments on the post Represented as: in, For comment content The kth word in m is a comment The number of words; Step B12: Utilize social media posts to retrieve comments and obtain a set of supplementary evidence from the comments. , Indicates post The kth piece of supplementary evidence; Mainstream viewpoints are extracted using a similarity-based selection mechanism; BERT is first used to analyze comments. Encode the comments to obtain a comment representation vector. , is represented as: in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector; An inter-sequence attention matrix is ​​obtained by calculating the similarity between each comment and other comments. Where N is the number of comments under a social media post; The similarity between comment m and comment n can be formalized as: After obtaining the inter-sequence attention matrix, the sum of similarity between each comment and all other comments is calculated. The top-K comments with the highest sum of similarity are selected as the mainstream viewpoints. Indicates post The kth mainstream viewpoint; Step B13: Use the VisualGLM multimodal large model to translate the image and obtain its text description. , This represents the image of the i-th social media post; Step B14: Use the AMR graph interpreter to generate an abstract semantic graph representation of the post's text. Using BERT Encoding is performed to obtain an abstract semantic representation of the text. .

2. The multimodal rumor detection method integrating multi-granularity features according to claim 1, characterized in that, Step B2 specifically includes the following steps: Step B21: Process the text obtained in step B11 Text is obtained by encoding with BERT. fine-grained representation vector ; Represented as: in, Indicates the k-th word In BERT encoding, the output of the last hidden state, b, represents the dimension of the word embedding representation vector; Images from the post The input is encoded in Swin-T to obtain a fine-grained image representation vector. : in, This represents the hidden state of the k-th window in the input at the output of the last layer of the model. is the batch size in Swin-T, and s is the hidden size of the visual embedding; Step B22: Supplementary evidence set obtained in step B12 Supplementary evidence is obtained by encoding with BERT. initial representation vector : in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector; The average vector of the initial representation vectors of all supplementary evidence is represented as the final supplementary evidence representation vector. : The set of mainstream viewpoints obtained in step B12 Encoding with BERT yields mainstream viewpoints. initial representation vector : in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector; The average vector of the initial representation vectors of all mainstream viewpoints is represented as the final mainstream viewpoint representation vector. : Step B23: Text description of the image obtained in step B13 Encoding with BERT yields a coarse-grained representation vector of the image. ; Represented as: in, Indicates the k-th word In BERT encoding, the output of the last hidden state, d, represents the dimension of the word embedding representation vector; Step B24: Abstract the semantic representation of the text obtained in step B14. The input graph convolutional neural network (GCN) produces a coarse-grained representation vector of the text. ; Represented as: 。 3. The multimodal rumor detection method integrating multi-granularity features according to claim 2, characterized in that, Step B3 specifically includes the following steps: Step B31: Process the fine-grained text representation vector obtained in step B2. and image fine-grained representation vector Using collaborative attention transformer CT for fusion; CT consists of a multi-head attention network and a feedforward neural network; Representing different modal inputs as and In CT scans, Used as a query for Q, and Using K as the key and V as the value, CT will use each head The collaborative attention matrix is ​​calculated as follows: in, Represents the activation function and projection matrix. Corresponding to Q, K, and V respectively. , 'm' represents the dimension of the CT scan, and 'm' represents the number of heads. Multi-head attention H is the projection of the cascaded collaborative attention matrices: In this context, the symbol ; denotes a cascade operation and a projection matrix. ; After that, 𝐻 and A multimodal representation based on attention is obtained by using a two-layer normalized feedforward neural network (FFN). : Multimodal representation The feature vector F is obtained by average pooling and used as the output of CT. feature and characteristics They are mapped to the same dimension through a linear layer. As and Inputting the text features in different orders into a shared weighted CT yields output features, including a visual attention-weighted text feature. A visual feature weighted by text attention : in, , ; Step B32: Cascade the image coarse-grained representation vector obtained in step B2 and text coarse-grained representation vector Feed into feedforward neural network Obtain the coarse-grained representation vector of the image and text. Visual attention-weighted text features in cascade step B31 Visual features weighted by text attention Feed into feedforward neural network Obtain the fine-grained representation vector of the image and text. : Step B33: Calculate the image coarse-grained representation vector and text coarse-grained representation vector The cosine similarity is considered as the weight of image-text similarity and serves as a weighting coefficient for cross-modal features to guide the classifier's learning process. The cosine similarity is calculated as follows: Cascaded coarse-grained representation vector of images and text and fine-grained representation vectors of text and graphics And input the projection head A cross-modal representation is generated; each element in the output vector is multiplied by a text-image similarity weight, resulting in the final cross-modal representation vector. as follows: 。 4. The multimodal rumor detection method integrating multi-granularity features according to claim 3, characterized in that, The specific implementation method of step B4 is as follows: At the word level dimension, the text fine-grained representation vectors are respectively... and text coarse-grained representation vector The features are aggregated and concatenated, then input into a projection head consisting of two fully connected networks with ReLU activation functions to obtain the text unimodal representation vector. ; Aggregate image fine-grained representation vectors and aggregated image coarse-grained representation vector By concatenating the images, a single-modal representation vector is obtained from a projection head that has the same branching structure as the text but different parameters. : 。 5. The multimodal rumor detection method integrating multi-granularity features according to claim 4, characterized in that, The specific implementation method of step B5 is as follows: Text unimodal representation vector and supplementary evidence representation vector They are mapped to the same dimension through a linear layer. As and Inputting different sequences of text into a shared weighted CT yields output features, including comment attention-weighted text features. A text attention-weighted comment feature By feeding it into a feedforward neural network Obtain the text comment interaction representation vector : in ; Image single-modal representation vector and supplementary evidence representation vector They are mapped to the same dimension through a linear layer. As and Inputting the data in different orders into a shared weighted CT scan yields output features, and a commentary focuses on the attention-weighted image features. A comment feature weighted by image attention By feeding it into a feedforward neural network Obtain the image comment interaction representation vector : in ; Cross-modal representation vector and supplementary evidence representation vector They are mapped to the same dimension through a linear layer. As and Inputting the data in different orders into a shared weighted CT scan yields output features, and a commentary focuses on the attention-weighted image features. A comment feature weighted by image attention By feeding it into a feedforward neural network Obtain cross-modal comment interaction representation vector : in .

6. The multimodal rumor detection method integrating multi-granularity features according to claim 5, characterized in that, The specific implementation method of step B6 is as follows: Connect text comment interaction representation vector Image comment interaction representation vector Cross-modal comment interaction representation vector Feed into feedforward neural network Obtain the multimodal representation vector Aligning multimodal representation vectors through supervised contrastive learning A random sample g is selected as the anchor point. p, which shares the same label as the anchor point, is considered a positive multimodal sample, and a, which has a different label, is considered a negative multimodal sample. The supervised contrastive loss is expressed as... : Where G is the small training set extracted during the training process, A(g) is the negative sample set in the small training set, and P(g) is the positive sample set in the small training set. Represents the cosine similarity function. This represents the multimodal representation vector of sample g. This represents the multimodal representation vector of sample p. This represents the multimodal representation vector of sample a. This is the temperature coefficient.

7. The multimodal rumor detection method integrating multi-granularity features according to claim 6, characterized in that, Step B7 specifically includes the following steps: Step B71: Convert the multimodal representation vector obtained in step B6 into a multimodal representation vector. and mainstream viewpoint representation vector The concatenated input is fed into the classifier to obtain the post prediction result: in, This is a rumor classifier consisting of a two-layer fully connected network with ReLU activation function; The cross-entropy loss function is used as the classification loss function to calculate the loss value. The calculation formula is as follows: Where y is the real label; Step B72: Alignment loss and classification loss Together, these parameters form the final loss function of the model. The Adam gradient optimization algorithm is used as the optimizer, and all parameters are iteratively updated through backpropagation to minimize the loss function during model training. The calculation formula is as follows: in, and It is used to balance the two types of losses.

8. A multimodal rumor detection system integrating multi-granularity features, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-7.

Images