Intention-aware and evidence-scored fine-grained multimodal rumor detection method

Abstract

The application discloses a fine-grained multi-modal rumor detection method for intention perception and evidence scoring and belongs to the technical field of natural language processing and computer vision. The method comprises the following steps: performing feature extraction on post texts, post images and external evidence; refining text features and image features by using a multi-gate hybrid expert network; performing quality scoring on text evidence and image evidence and dynamically weighting and aggregating; constructing fine-grained semantic association between fine-grained text features and image regions, realizing cross-modal bidirectional alignment; mining propagation intention from text sentiment fluctuation, rhetorical features and visual emotional clues and performing gate fusion; dynamically distributing the weights of features by adaptive feature fusion; and finally inputting the fusion representation into a classifier to obtain a truth class prediction result. The application realizes collaborative reasoning of content semantics, evidence information and propagation intention, and improves the accuracy and robustness of multi-modal rumor detection.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and computer vision, and in particular to a fine-grained multimodal rumor detection method based on intent perception and evidence scoring. Background Technology

[0002] In recent years, rumor detection technology has made significant progress. Early research mainly relied on unimodal information such as text content, propagation structure, or user behavior for authenticity judgment. Early unimodal detection methods learned text semantics or propagation patterns through models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), Transformers, and graph neural networks (GNNs), which improved rumor detection performance to some extent. However, with the widespread dissemination of mixed text and image content on social media, multimodal rumor detection has gradually become a research hotspot. Existing methods, by fusing text and image information and combining techniques such as cross-modal attention, contrastive learning, and external knowledge retrieval, have achieved detection results superior to unimodal methods on multiple public datasets.

[0003] While existing rumor detection methods have achieved good performance across many mainstream technical approaches, they still have certain limitations: First, existing methods typically fuse extracted text and image features directly, making them susceptible to interference from redundant information and noisy features. Second, existing evidence-based rumor detection methods mainly focus on the semantic relevance of evidence to news content, lacking effective assessment of evidence quality and credibility, leading to low-quality or even erroneous evidence easily influencing the model's final decision. Third, current cross-modal interaction methods mostly focus on global semantic consistency, but fail to adequately capture the fine-grained semantic relationships between fine-grained text features and local image regions, limiting the model's ability to discriminate semantic conflicts and consistency between text and images. Fourth, rumors on social media often have specific dissemination purposes, such as creating panic, misleading the public, or attracting attention, while existing research focuses more on the content itself and acquires less high-level semantic information such as dissemination intent, resulting in insufficient understanding of rumor propagation mechanisms by the models. Summary of the Invention

[0004] This invention aims to address at least the problems in existing technologies, such as the difficulty in fully mining multimodal features and fine-grained cross-modal semantic associations, which affects the effective expression of information, and the insufficient capture of high-level semantic information such as the quality of external evidence and dissemination intent, which limits the model's ability to make truthful inferences. In particular, it innovatively proposes a fine-grained multimodal rumor detection method based on intent perception and evidence scoring.

[0005] To achieve the above-mentioned objectives of this invention, this invention provides a fine-grained multimodal rumor detection method based on intent perception and evidence scoring, comprising the following steps:

[0006] S1, extract features from post text, post image, and corresponding external text and image evidence to obtain corresponding text features, image features, and evidence features;

[0007] S2, the text features and the image features are input into a multi-gated hybrid expert network for refinement to obtain text enhancement features and image enhancement features;

[0008] S3: Evaluate the quality scores of each textual and image piece of evidence separately, and aggregate the features of each piece of evidence according to the quality weights to obtain the final evidence representation;

[0009] S4. Construct cross-modal associations between fine-grained text features and image region features, perform bidirectional semantic alignment, and obtain fine-grained modal alignment features;

[0010] S5, extract the text propagation intent and image propagation intent from the text enhancement features and the image enhancement features respectively, and obtain the propagation intent perception features through gating fusion;

[0011] S6. Project each feature onto a unified semantic space, dynamically allocate weights through a gating mechanism and perform weighted fusion to obtain the fused high-level semantic representation;

[0012] S7. Input the high-level semantic representation into the classifier to obtain the prediction result of the authenticity category to which the post belongs.

[0013] In a preferred embodiment of the above technical solution, S1 includes: using a pre-trained language model as a text encoder to encode the post text to obtain post text features, and using the text encoder with shared parameters to encode each piece of text evidence to obtain text evidence features; using a pre-trained visual model as an image encoder to encode the post image to obtain post image features, and using the image encoder with shared parameters to encode each piece of image evidence to obtain image evidence features; the dimensions of the post text features and the text evidence features are consistent, and the dimensions of the post image features and the image evidence features are consistent.

[0014] In a preferred embodiment of the above technical solution, step S2 includes: the multi-gated hybrid expert network comprising a shared expert pool, a gating network, and a task tower; inputting the text features and the image features into the shared expert pool, where multiple expert networks in the shared expert pool respectively mine potential information in the input features from different semantic perspectives to obtain multiple expert outputs; dynamically determining the importance weights of each expert output through the gating network, and performing weighted fusion of multiple expert outputs to obtain a shared representation; the task tower includes a first task tower and a second task tower, where the first task tower maps the shared representation to a unified semantic space for modal alignment to obtain an aligned representation, and the second task tower refines the shared representation to obtain authenticity discrimination features; the text enhancement features and the image enhancement features are respectively composed of the aligned representation and authenticity discrimination features of the corresponding modal.

[0015] In a preferred embodiment of the above technical solution, step S3 includes: for text evidence, concatenating the text enhancement features with the features of each text evidence to obtain a joint representation; calculating the quality score of each text evidence through a quality assessment network; filtering invalid text evidence using a masking method; normalizing the quality scores of each text evidence to obtain quality weights; and weighted summing of the features of each text evidence according to the quality weights to obtain aggregated text evidence features; for image evidence, obtaining aggregated image evidence features in the same way; and concatenating and projecting the aggregated text evidence features and the aggregated image evidence features to obtain the final evidence representation.

[0016] In a preferred embodiment of the above technical solution, step S4 includes: calculating the semantic similarity between each fine-grained text feature and each image region feature to obtain a cross-modal association matrix; normalizing the cross-modal association matrix along the image region dimension to obtain a text-to-image attention matrix, and obtaining the image semantic representation corresponding to each fine-grained text through attention weighting; normalizing the cross-modal association matrix along the fine-grained text dimension to obtain an image-to-text attention matrix, and obtaining the text semantic representation corresponding to each image region through attention weighting; performing average pooling on the image semantic representation and the text semantic representation respectively, and projecting the pooling results after concatenation to obtain the fine-grained modal alignment feature.

[0017] In a preferred embodiment of the above technical solution, the extraction of the text dissemination intent in S5 includes: performing sentiment prediction on the text sequence features to obtain the probability distribution of each fine-grained text belonging to three sentiment categories: positive, neutral, and negative; measuring the degree of text sentiment fluctuation based on the variance of the sentiment probability sequence to obtain sentiment fluctuation features; constructing a rhetoric dictionary, statistically analyzing the coverage of rhetorical expressions in the post text, and projecting the coverage onto a high-dimensional semantic space to obtain rhetorical features; and fusing the text enhancement features, the sentiment fluctuation features, and the rhetorical features to obtain the text dissemination intent.

[0018] In a preferred embodiment of the above technical solution, the extraction of image propagation intent in S5 includes: encoding sentiment prompts using the text encoder to construct a visual sentiment prototype matrix; mapping the image enhancement features to the semantic space of the text encoder and normalizing them, calculating the similarity between the normalized image representation and each sentiment prototype in the visual sentiment prototype matrix to obtain a visual sentiment similarity vector; calculating the cosine similarity between the text sentiment distribution and the visual sentiment similarity vector as a cross-modal sentiment consistency feature; fusing the image enhancement features, the visual sentiment similarity vector, and the cross-modal sentiment consistency feature to obtain the image propagation intent; mapping the text propagation intent and the image propagation intent to a unified space and calculating gating weights, adaptively integrating the two according to the gating weights to obtain the propagation intent perception feature.

[0019] In a preferred embodiment of the above technical solution, step S6 includes: projecting the alignment representation and authenticity discrimination feature in the text enhancement feature and the image enhancement feature, the final evidence representation, the fine-grained modality alignment feature, and the propagation intent perception feature onto a unified semantic space to construct a joint representation; using a gating network to learn the weight coefficients corresponding to each feature to adaptively measure the importance of different features; weighting each feature according to the weight coefficients, and summing the weighted features to obtain the fused high-level semantic representation.

[0020] In a preferred embodiment of the above technical solution, step S7 includes: inputting the high-level semantic representation into a fully connected classifier, obtaining the predicted probability of the post belonging to each real category through a normalization function, and taking the category corresponding to the maximum predicted probability as the prediction result; during model training, the cross-entropy loss between the prediction result and the real label is used as the main classification objective, and multiple auxiliary optimization objectives are introduced for joint optimization. The auxiliary optimization objectives include orthogonal constraint loss for feature enhancement, quality regularization loss for evidence scoring, constraint consistency loss for fine-grained modality alignment, and contrast loss for propagation intent perception.

[0021] In the preferred embodiment of the above technical solution, the authenticity category in S7 includes non-rumors, rumors, and unverified rumors.

[0022] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0023] This paper is the first to apply MMoE (Multi-Gated Hybrid Expert Network) to a multimodal rumor detection task. It learns semantic information through multiple expert networks and designs a gating mechanism to adaptively fuse representations from different experts, achieving more robust multimodal feature learning. An evidence scoring module is proposed to assess the quality of retrieved evidence and dynamically assign weights based on the credibility of the evidence, effectively addressing noise interference from low-quality evidence. A fine-grained modality alignment module is designed to enhance cross-modal semantic consistency by capturing the fine-grained semantic correspondence between fine-grained text features and local image regions. Finally, a multimodal rumor detection framework integrating evidence scoring, fine-grained modality alignment, and dissemination intent perception is constructed. Dissemination intent perception is introduced into the news and evidence-driven authenticity detection process, enabling collaborative reasoning of content semantics, evidence information, and dissemination behavior.

[0024] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0025] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0026] Figure 1 This is the overall flowchart of the present invention;

[0027] Figure 2 This is the overall architecture diagram of the present invention;

[0028] Figure 3 This is a schematic diagram of the processing procedure of the feature enhancement module of the present invention;

[0029] Figure 4 This is a schematic diagram of the evidence scoring module of the present invention;

[0030] Figure 5 This is a schematic diagram of the structure of the transmission intent perception module of the present invention;

[0031] Figure 6 The figure shows the experimental results of how different numbers of expert networks, learning rates, and batch sizes affect the model performance of this invention.

[0032] Figure 7 This is a graph showing the experimental results of how different numbers of sentiment cue words affect the model performance in this invention;

[0033] Figure 8 This is the loss weight parameter of the present invention. and Figure showing the experimental results affecting model performance;

[0034] Figure 9 This is a T-SNE visualization of the final post representation results learned by the present invention and its variants on two datasets. Detailed Implementation

[0035] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0036] like Figure 1 As shown, this invention provides a fine-grained multimodal rumor detection method (INFER) based on intent perception and evidence scoring. For example... Figure 2 As shown, the method consists of eight parts: feature extraction module, feature enhancement module, evidence scoring module, fine-grained modality alignment module, propagation intent perception module, adaptive feature fusion module, classification, and loss function.

[0037] First, the feature extraction module uses pre-trained language and vision models to extract semantic representations of post text, post images, and external evidence, respectively. The original features contain cross-modal semantic gaps and a large amount of noise. MMoE is used to refine the original features and construct a unified modal alignment space. Second, the evidence scoring module further evaluates the relevance and credibility of each piece of evidence to the original post. A dynamic weighting mechanism is used to obtain high-quality evidence features, enhancing the verification of content authenticity. To further explore the potential fine-grained semantic relationships between text and images, the model uses fine-grained text features and local image region information to establish a correspondence between them, addressing common issues of inconsistency and semantic conflict in rumor content. Then, the propagation intent perception module mines the implicit propagation intent during rumor dissemination based on the text and visual emotional cues of the post. The adaptive feature fusion module achieves effective fusion of multimodal features through dynamic weight allocation. Finally, the fused representation is input into a classifier to complete the final rumor prediction, thereby achieving more accurate and robust multimodal rumor detection.

[0038] Problem Description: This invention is a multimodal rumor detection method that combines post content and evidence information from social media. Given a post from social media... ,in Indicates the post text, Representing a post image, the model performs text retrieval and reverse image search on the post to obtain a set of textual evidence related to the post. and image evidence set The goal of this invention is to learn a mapping function. Predict the authenticity category of a post by utilizing the semantic relationship between the post content and its corresponding evidence:

[0039] ;

[0040] in Indicates post tags, 0 indicates that it is not a rumor, 1 indicates that it is a rumor, and 2 indicates that it is an unverified rumor.

[0041] S1 extracts features from the post text, post image, and corresponding external text and image evidence to obtain the corresponding text features, image features, and evidence features.

[0042] S1-1, extract features from the post text and the corresponding external text evidence to obtain text features and text evidence features.

[0043] The text content of posts typically contains key information such as event descriptions, opinions, and emotional biases, serving as crucial criteria in rumor detection. To fully capture the contextual semantic relationships within posts, this invention employs the pre-trained language model XLM-RoBERTa-Large as a unified text encoder. Compared to traditional word vector methods, it possesses stronger semantic capture capabilities and cross-linguistic generalization abilities, effectively learning rich contextual semantic information. For post text... The text features are obtained by inputting them into a text encoder:

[0044] ;

[0045] in The feature matrix representing the post text; The text features representing the l-th word; Represents a word index. The sequence length of the post text is determined by the actual number of lexical units after the input text is segmented. It is an upper bound parameter that varies with the samples rather than a fixed hyperparameter. The dimension of the text features is determined by the dimension of the hidden layer of the selected pre-trained language model and does not require additional manual setting. It represents a word consisting of L lexical units, each lexical unit having a dimension of 1. The real-valued matrix space.

[0046] Meanwhile, external textual evidence also contains rich semantic information. To ensure consistency between the semantic space of posts and evidence, a shared-parameter XLM-RoBERTa-Large text encoder is used to extract features from each piece of textual evidence. Represented as:

[0047] ;

[0048] in The feature matrix representing textual evidence; Indicate the characteristics of the nth textual evidence; Indicates evidence index, This represents the number of textual evidence retrieved, determined by the actual number of valid evidence items returned by the retrieval system. It is a data-driven dynamic upper bound, and its value range corresponds to the top-K return setting of the evidence retrieval system. This indicates the dimension of text features, which should be consistent with the dimension of the post text output. Indicates by The feature matrix space composed of textual evidence; This represents the encoding function of the XLM-RoBERTa-Large text encoder.

[0049] S1-2, extract features from the post image and the corresponding external image evidence to obtain image features and image evidence features.

[0050] Images on social media are susceptible to manipulation, splicing, and cross-scene reuse, necessitating image encoders with strong semantic understanding for feature extraction. This study employs the pre-trained visual model CLIP ViT-L / 14 as a unified image encoder for input post images. The image features are input into the image encoder to obtain the image features:

[0051] ;

[0052] in The feature matrix representing the image of the post; Represents the regional features of the m-th image region; M represents the post image region index; M represents the total number of post image regions, corresponding to the total number of grid regions, which is a natural quantity determined by the encoder structure. Indicates the dimension of image features; This represents the feature matrix space consisting of M image regions; This represents the encoding function of the CLIP ViT-L / 14 image encoder.

[0053] Similarly, to ensure that the post images and image evidence are in the same visual semantic space, a CLIP ViT-L / 14 image encoder with shared parameters is used to encode the retrieved image evidence, and the extracted image evidence features are... Represented as:

[0054] ;

[0055] in The feature matrix representing image evidence; Represents the features of the nth image evidence; This indicates the number of image evidences retrieved. This indicates the feature dimension of the image evidence, which is consistent with the output dimension of the post image. Indicates by The feature matrix space is composed of image evidence.

[0056] S2 inputs text features and image features into a multi-gated hybrid expert network for refinement, obtaining text enhancement features and image enhancement features.

[0057] Through feature extraction, the model obtains multi-level semantic representations of post content and its corresponding evidence. However, data sources in social media are complex and contain a large amount of noise information irrelevant to authenticity judgment. Furthermore, text and image modalities have different feature distributions and representation spaces. Direct evidence reasoning through encoder output is easily affected by the cross-modal semantic gap, limiting the model's discriminative ability. To address this, this invention designs a feature enhancement module, introducing MMoE to refine post text and image features, while simultaneously constructing a unified cross-modal alignment space. The feature enhancement process is as follows: Figure 3 As shown.

[0058] For the input text features and image features First, the input is fed into a shared expert pool to obtain deep semantic representations. Each expert network is treated as an independent semantic learner, aiming to extract latent information from the input features from different perspectives. The output representation of the k-th expert is shown below:

[0059] ;

[0060] in This represents the expert index; K represents the total number of experts, which is an adjustable hyperparameter. The features input to the expert network are text features. and image features Any vector in; Let the first-level linear transformation matrix of the k-th expert be represented. This represents the second-level linear transformation matrix of the k-th expert; This represents the first-layer bias parameters. Indicates the second-layer bias parameters; This represents a modified linear unit, which is a nonlinear activation function. This indicates that after the output of the k-th expert and the data processing by the expert network, the model obtains representations from different semantic perspectives. .

[0061] Different detection tasks often focus on different semantic information. For rumors involving the repurposing of old images, the model needs to focus on visual information, while for rumors exaggerating facts, the model focuses more on textual information. Therefore, a gating network is used to dynamically determine the importance of each expert. The gating network calculation for the j-th task tower is shown below:

[0062] ;

[0063] in This represents the expert weight vector corresponding to the j-th task tower; This indicates the mission tower number, corresponding to the dual mission tower setting; This represents the weight matrix of the gated network; The features of the gating network corresponding to the input task tower are text features. and image features Any vector in; Indicates the bias parameters of the gating network; This represents the normalization function.

[0064] The MMoE architecture requires each downstream subtask to be equipped with an independent gating network and task tower. This invention decomposes the feature refinement objective into two complementary subtasks: modality alignment and authenticity discrimination. Therefore, a dual task tower is set up. The two tasks correspond to two learning objectives: spatial consistency and task discriminability, respectively. This is the smallest nontrivial decomposition granularity in multi-task learning, which can avoid gradient conflicts under a single gating network and parameter redundancy caused by multiple towers.

[0065] Subsequently, the importance weights of experts are obtained through a gating network, and the outputs of multiple experts are weighted and fused:

[0066] ;

[0067] in This represents the shared representation obtained by the j-th task tower, which serves as the input to the projection layer within the task tower. This represents the expert weight vector corresponding to the j-th task tower. The importance weight corresponding to the k-th expert; This represents the output of the k-th expert; K represents the total number of experts.

[0068] After obtaining the shared representation, the model performs a modality alignment task in the first task tower to learn a unified cross-modal representation space. Since text and image modalities have different data distributions, semantic misalignment problems are prone to occur. Therefore, the shared representation is first mapped to a unified semantic space for alignment.

[0069] ;

[0070] in Representing modes Alignment representation; Represents either text modality or image modality; Indicates the first task tower mode The output of the input features is calculated separately for the text modality and the image modality. and These represent the projection matrices of the first and second layers in the first task tower, respectively. and These represent the offset parameters corresponding to the first and second layers in the first mission tower, respectively. Presentation layer normalization operation.

[0071] While cross-modal alignment can alleviate semantic discrepancies, determining information authenticity still requires a large number of task-related features. Therefore, this invention designs a second task tower to refine the shared representation and learn authenticity-discriminating features:

[0072] ;

[0073] in Representing modes A concise expression; Represents either text modality or image modality; Indicates the second task tower mode The output of the input features is calculated separately for the text modality and the image modality. and These represent the projection matrices of the first and second layers in the second task tower, respectively. and These represent the offset parameters corresponding to the first and second layers in the second mission tower, respectively. This represents the Gaussian error linear unit activation function.

[0074] Finally, we obtained the text enhancement features. and and image enhancement features and MMoE can not only effectively suppress noise interference, but also learn cross-modal shared knowledge and authenticity discrimination knowledge, providing the model with more robust and discriminative feature representations.

[0075] S3 evaluates the quality scores of each textual and image piece of evidence, and aggregates the features of each piece of evidence according to their quality weights to obtain the final evidence representation.

[0076] External evidence provides crucial factual basis for rumor detection; however, evidence found through search engines often varies in quality, with some evidence highly relevant to the target post while others contain noise or even false matches. Assigning equal importance to all evidence not only weakens the contribution of valid evidence but may also introduce misinformation, affecting the model's ability to determine truthfulness. Therefore, this invention designs an evidence scoring module to rate the quality of retrieved evidence and uses an attention aggregation mechanism to filter high-credibility evidence, constructing a reliable fact-verification representation. Text enhancement features are then implemented. Image enhancement features and textual evidence features Features of image evidence As input. The evidence scoring module, such as... Figure 4 As shown.

[0077] S3-1: Evaluate the quality score of each piece of textual evidence, and perform weighted aggregation of the corresponding textual evidence features based on the textual evidence quality weights.

[0078] For textual evidence, the relevance of different textual evidences to the target post varies significantly. To assess the quality of textual evidence, the text enhancement features are first concatenated with the features of each piece of textual evidence:

[0079] ;

[0080] in This represents the joint representation consisting of text enhancement features and the features of the nth text evidence. Represents text modality; Indicates feature concatenation operation; Indicates text enhancement features; This represents the feature of the nth piece of textual evidence.

[0081] Subsequently, a two-layer perceptron was used to score the textual evidence. A higher score indicates that the textual evidence can provide stronger support for verifying authenticity. The process of calculating the quality score of textual evidence is as follows:

[0082] ;

[0083] in This represents the quality score of the nth textual evidence. This represents a text evidence quality assessment network. Because the retrieved text evidence varies, invalid text evidence is filtered using a mask.

[0084] ;

[0085] in Indicates the mask value; This represents the actual number of valid textual evidences. Next, the quality weight of each piece of textual evidence is calculated using the Softmax function. In batch processing of variable-length evidence sets, to maintain the consistency of the tensor shape, the padding positions need to be masked; the scores of the masked positions are set to negative infinity, and after the Softmax exponentiation, their corresponding quality weights are strictly 0. This completely eliminates the contribution of invalid evidence in the weighted aggregation without affecting the normalized distribution of valid positions. In engineering implementations, negative infinity is usually approximated by a sufficiently small finite value, such as negative billions, to avoid floating-point overflow and ensure gradient stability.

[0086] ;

[0087] in This represents the quality weight of the nth textual evidence; This represents an exponential function; the summation index j iterates through all... Scoring of textual evidence; Indicates the amount of textual evidence.

[0088] After obtaining the quality weights of the textual evidence, a weighted summation is used to obtain the aggregated textual evidence features:

[0089] ;

[0090] in This represents the aggregated features of textual evidence; Indicates the weight of the quality of textual evidence; This indicates the characteristics of the corresponding textual evidence.

[0091] S3-2, evaluate the quality score of each piece of image evidence, and perform weighted aggregation of the corresponding image evidence features according to the image evidence quality weight.

[0092] Similarly, for image evidence, a joint representation of image enhancement features and image evidence features is first constructed.

[0093] ;

[0094] in This represents the joint representation consisting of the post's image enhancement features and the features of the nth image evidence; i represents the image modality; Indicates feature concatenation operation; Indicates image enhancement features in the post; This represents the feature of the nth image evidence.

[0095] Subsequently, a two-layer perceptron is used to calculate the image evidence quality score, and the quality weights of the image evidence are calculated using the Softmax function.

[0096] ;

[0097] ;

[0098] in This represents the quality score of the nth image piece of evidence; This represents the quality weight of the nth image piece of evidence; Indicates the quantity of visual evidence; This represents the image evidence quality assessment network. Image evidence at invalid locations is also filtered using a masking method.

[0099] The aggregated image evidence features are obtained by weighting and summing the quality weights of the image evidence:

[0100] ;

[0101] in Represents aggregated features of image evidence; Indicates the quality weight of image evidence; This represents the corresponding image evidence features.

[0102] Textual and visual evidence provide verification information from linguistic and visual perspectives, respectively. Fusing these two types of evidence yields complementary correlations between different modalities of evidence.

[0103] ;

[0104] in This indicates that the evidence is final. This represents the concatenation operation of aggregated textual evidence features and image evidence features; Represents the learnable projection matrix; This represents the bias parameter.

[0105] S4. Construct cross-modal associations between fine-grained text features and image region features, perform bidirectional semantic alignment, and obtain fine-grained modal alignment features.

[0106] While text-enhanced and image-enhanced features can effectively capture overall semantic information, they still lack a sufficient understanding of the local semantics of rumor content, thus affecting the model's ability to judge text-image consistency. To address this, this invention designs a fine-grained modality alignment module. By constructing local semantic associations between fine-grained text features and image regions, it achieves cross-modal fine-grained semantic alignment, uncovering more accurate cross-modal consistency information. After feature extraction, the model obtains word-level text features. This refers to fine-grained text features, which are region-level features output by the image encoder. This refers to image region features. First, local semantic associations are constructed between text and image. The semantic similarity between each fine-grained text feature and each image region feature is calculated to obtain the local semantic associations.

[0107] ;

[0108] in This represents the correlation score between the l-th fine-grained text feature and the m-th image region feature; This represents the l-th fine-grained text feature; Represents the features of the m-th image region; Indicates the feature dimension; Used to scale the dot product result. The resulting cross-modal correlation matrix. The elements in the matrix represent the semantic correlation between the corresponding fine-grained text features and image region features.

[0109] S4-1 performs fine-grained text feature alignment.

[0110] For each fine-grained text, the model dynamically searches for associated region features in the image, ensuring that each word in the text receives corresponding image context information. First, the cross-modal association matrix is ​​normalized along the image region dimension, as shown below:

[0111] ;

[0112] ;

[0113] in The attention matrix representing the text-to-image transition; Represents the cross-modal correlation matrix; Softmax is applied row-by-row along the image region dimension; Indicates attention weights. This represents the importance distribution of each region in the image corresponding to each fine-grained text. Then, attention-weighted representations are used to obtain the semantic representation of the image corresponding to the fine-grained text.

[0114] ;

[0115] in Let represent the image semantic representation corresponding to the l-th fine-grained text, and let .

[0116] S4-2, Perform image region feature alignment.

[0117] Some regions in an image may correspond to multiple text words simultaneously, so alignment from a textual perspective alone is insufficient; further, reverse alignment from an image perspective is required. Similarly, after obtaining the cross-modal association matrix, for each image region, the model searches for the best-matching text semantics. First, the cross-modal association matrix is ​​normalized along the fine-grained text dimension:

[0118] ;

[0119] ;

[0120] in The attention matrix representing the image-to-text transition; Represents the transpose of the cross-modal correlation matrix; Softmax is applied line by line along the fine-grained text dimension; This represents the importance of the l-th fine-grained text feature to the m-th image region feature. The sum of the corresponding weights is 1. Then, attention-weighted methods are used to obtain the textual semantic representation of the image region:

[0121] ;

[0122] in Let m represent the text semantic representation corresponding to the m-th image region. The bidirectional semantic alignment model can simultaneously learn the fine-grained relationships between text and images.

[0123] S4-3 performs bidirectional semantic alignment to obtain fine-grained modality alignment features.

[0124] After local interaction in the region, the model obtained and Calculate the alignment representations on the text side and the image side respectively:

[0125] ;

[0126] ;

[0127] in This indicates a fine-grained representation of the text. This represents a fine-grained representation of the image. This indicates the average pooling operation.

[0128] Finally, fine-grained modality alignment features are obtained through feature fusion:

[0129] ;

[0130] in This represents fine-grained modal alignment features; Indicates feature concatenation operation; Represents the projection matrix; This represents the bias parameter. The fine-grained modal alignment module effectively captures text-image consistency information and identifies potential cross-modal semantic conflicts by establishing semantic associations between fine-grained text features and image regions.

[0131] S5 extracts the text propagation intent and image propagation intent from the text enhancement features and image enhancement features respectively, and obtains the propagation intent perception features through gating fusion.

[0132] In social media platforms, the information posted and disseminated by users often carries specific dissemination intentions. These underlying intentions reflect the motivations and purposes behind information dissemination and are closely related to rumor propagation. After feature enhancement, evidence scoring, and fine-grained modality alignment modules, the model has acquired semantic information at multiple levels. Building on this, the dissemination intention perception module further mines the implicit dissemination intentions within post content, supplementing the content's semantic information from a dissemination behavior perspective and enhancing the model's understanding of rumor propagation mechanisms. The dissemination intention perception module is as follows: Figure 5 As shown.

[0133] S5-1, Extracting the textual dissemination intent from text enhancement features.

[0134] Text content is the primary vehicle for users to express their communication intentions. To capture emotional fluctuations and rhetorical tendencies within the text, sentiment prediction is first performed on the text sequence features. For the extracted text features... A three-category sentiment predictor is used to calculate the global sentiment probability distribution, where the three categories of sentiment include positive, neutral, and negative.

[0135] ;

[0136] in Represents the text sentiment probability matrix; Softmax normalizes along the sentiment category dimension; Represents the sentiment classification weight matrix; This indicates the bias term.

[0137] During the spread of rumors, user emotions often fluctuate more dramatically than with actual news. Therefore, we further use the variance of the sentiment probability sequence to measure the degree of text sentiment fluctuation:

[0138] ;

[0139] in Indicates characteristics of emotional fluctuations; This represents the probability sequence corresponding to the sentiment category of the rumor. This indicates variance; a larger variance suggests more pronounced emotional fluctuations and changes in expression within the text.

[0140] Rumors often employ exaggerated, manipulative, or inflammatory rhetoric. Therefore, this invention constructs a rhetorical dictionary to statistically analyze the rhetorical coverage of posts, where the rhetorical coverage... The higher the rhetorical coverage, the more rhetorical expressions a post contains, which are then projected into a high-dimensional semantic space:

[0141] ;

[0142] in Indicates rhetorical features; Represents the projection matrix; This represents the bias term. The rhetorical coverage rate is defined as the proportion of lexical units in a post that match the rhetorical dictionary to the total number of lexical units in the text. It is a proportion where the numerator is no greater than the denominator, naturally falling within the range of 0 to 1. The endpoint 0 indicates that the text contains no rhetorical expressions at all, while the endpoint 1 indicates that all lexical units are rhetorical expressions. Both endpoints are achievable and semantically clear. Normalization to the 0-1 interval eliminates the influence of text length differences, allowing direct comparison of the rhetorical intensity of posts of different lengths, and avoiding feature distribution drift caused by changes in the input scale with sample length when projected into a high-dimensional space.

[0143] Finally, the text enhancement features, sentiment fluctuation features, and rhetorical features are fused together:

[0144] ;

[0145] in Indicates the text's intended message; Indicates text enhancement features; Indicates characteristics of emotional fluctuations; Indicates rhetorical features; This indicates the intention to extract text from the network.

[0146] S5-2 extracts the image propagation intent from the image enhancement features and obtains propagation intent perception features through gating fusion.

[0147] Images can also convey a user's intention to spread information. Rumors often tend to use visually striking images to attract user attention. This invention further mines information about the intention to spread information from image modalities.

[0148] First, the sentiment cue words are encoded using the CLIP text encoder to construct a visual sentiment prototype matrix. ,in This indicates the number of sentiment cue words, with a value of 3. The three sentiment cue words correspond to positive, neutral, and negative sentiments, respectively. This represents the feature dimension of the text encoder. The image representation is then mapped to the encoder's semantic space, and the similarity between the image and the sentiment prototype is calculated:

[0149] ;

[0150] ;

[0151] in This represents the normalized image representation. Represents the mapping matrix; Indicates image enhancement features; Indicates L2 normalization; Represents a visual sentiment similarity vector; The three dimensions correspond one-to-one with the three categories of emotions: positive, neutral, and negative.

[0152] To measure the consistency of sentiment between text and images, the cosine similarity between the text sentiment distribution and the visual sentiment distribution is calculated:

[0153] ;

[0154] in Indicates cross-modal sentiment consistency characteristics; This represents the average sentiment distribution of the text; Represents a visual sentiment similarity vector; This represents the cosine similarity function.

[0155] Finally, the image enhancement features, visual sentiment similarity vectors, and cross-modal sentiment consistency features are fused together:

[0156] ;

[0157] in Indicates the intent to spread the image; Indicates image enhancement features; Represents a visual sentiment similarity vector; Indicates cross-modal sentiment consistency characteristics; This represents the network for extracting the intent of image propagation.

[0158] Because there are differences between text propagation intent and image propagation intent, a gated fusion mechanism is used to adaptively integrate the two. First, the two propagation intents are mapped to a unified space, and then the fusion weights are calculated:

[0159] ;

[0160] in Indicates the gating weight; Represents the Sigmoid function; Represents the gating parameter matrix; Represents the text propagation intent mapping matrix; Represents the image propagation intent mapping matrix; Indicates the text's intended message; Indicates the intent to spread the image; This indicates the bias term.

[0161] Ultimately, the characteristics of perceived communication intent are obtained:

[0162] ;

[0163] in To perceive characteristics of the intention to spread; This represents the Hadamard product, which is the product of corresponding elements of two vectors.

[0164] The model of perception module for dissemination intent can reflect users’ potential dissemination motives, and improve the model’s understanding of the patterns of rumor dissemination and its ability to judge the authenticity of rumors.

[0165] S6 projects each feature onto a unified semantic space, dynamically assigns weights through a gating mechanism, and performs weighted fusion to obtain the fused high-level semantic representation.

[0166] After passing through the feature enhancement module, evidence scoring module, fine-grained modality alignment module, and dissemination intent perception module, the model has obtained high-order semantic representations at multiple levels. These features describe rumor information from the content layer, fact layer, cross-modal layer, and behavioral layer, respectively, and have strong complementarity. However, different samples have different degrees of dependence on information, and simple feature concatenation or fixed-weight fusion cannot fully leverage the advantages of different features. Therefore, this invention designs an adaptive feature fusion module, which dynamically learns the importance of different features and adaptively allocates the contribution weight of each feature, thereby obtaining a more robust and discriminative fused representation.

[0167] Since the feature dimensions and semantic spaces output by different modules differ, the model first projects the outputs of each module into a unified semantic space:

[0168] ,

[0169] ,

[0170] ,

[0171] ,

[0172] ;

[0173] in This represents the mapped features after feature enhancement and alignment; This represents the enhanced and refined mapped features; This represents the evidence scoring mapping features; This represents fine-grained modality alignment mapping features; This indicates the characteristics of the perception mapping of the intended message. , , , and Represents the corresponding mapping matrix; , , , and This indicates the bias term.

[0174] Different samples have varying degrees of dependence on various semantic information. A gating mechanism is introduced to dynamically weight different features. First, the output features of each module are constructed into a joint representation. Then, the gating network is used to learn the corresponding weight coefficients, thereby adaptively measuring the importance of different features.

[0175] ;

[0176] ;

[0177] in A global semantic representation of the sample; This indicates the degree of importance corresponding to the five characteristics; This indicates feature enhancement alignment feature weights; This indicates feature enhancement and refinement of feature weights; Indicates the weight of the evidence scoring features; This represents the fine-grained modality alignment feature weights; Indicates the weight of features that represent the perceived intent to spread; Represents the weight matrix; This represents the bias term. After obtaining the weights corresponding to various features, weighted summation and adaptive feature fusion are performed:

[0178] ,

[0179] ,

[0180] ,

[0181] ,

[0182] ;

[0183] ;

[0184] in This indicates that the weighted feature enhancement alignment feature is used; This indicates that the weighted feature enhancement refinement feature is a feature that has been optimized. This indicates the weighted characteristics of the evidence score; This represents the weighted fine-grained modal alignment feature; This represents the weighted perceived characteristics of the propagation intent. This represents the high-level semantic representation after fusion. Through adaptive feature fusion, the model can learn the complex interaction relationships between different semantic features, thereby obtaining a more robust fused representation.

[0185] S7 inputs the high-level semantic representation into the classifier to obtain the prediction result of the authenticity category to which the post belongs.

[0186] After adaptive feature fusion, the model obtains a final representation containing multi-layered information. By simultaneously constraining the learning process of different modules through a joint optimization strategy, the model performs rumor authenticity judgment. The final fused representation... Input the data into the classifier to obtain the class probability distribution and predict the results:

[0187] ;

[0188] ;

[0189] ;

[0190] in This represents the classification score output by the classifier; Represents the classification weight matrix; Indicates the bias term; This indicates the predicted probability that a sample belongs to any category; Indicates the number of categories; Indicates the predicted category; This indicates the index corresponding to the maximum value.

[0191] After the adaptive feature fusion module, the model obtains the following mapping features: enhanced and aligned fused features, refined and enhanced features, evidence scoring mapping features, fine-grained modality alignment mapping features, and propagation intent perception mapping features. A fully connected classifier and a Softmax function are used to predict the probability distribution of samples. The categorical cross-entropy loss between the predicted results and the true labels is used as the primary classification objective, denoted as [missing information]. To further enhance the model's discriminative ability, this invention introduces several auxiliary optimization objectives: orthogonal constraint loss is used in the feature enhancement module. Constraining the differences between expert outputs; the fine-grained modal alignment module uses constraint consistency loss. Constraining the consistency of bidirectional alignment; the evidence scoring module uses quality regularization loss. Constrain the distribution of evidence quality scores; the propagation intent perception module uses contrastive loss. This constrains the class separability of features perceived by the propagation intent. Therefore, the INFER total loss function is expressed as follows:

[0192] ;

[0193] in The weighting coefficients representing the orthogonal constraint loss; The weighting coefficients representing the constraint consistency loss; The weighting coefficients representing the quality regularization loss; The weighting coefficient represents the loss of perceived communication intent.

[0194] The four auxiliary loss weight coefficients adjust the contribution of orthogonality constraint, alignment consistency, quality regularization, and intent contrast loss to the overall objective, and are all taken as non-negative real values. The coefficient of the main classification loss is fixed at 1 as the baseline scale, and the coefficients of the auxiliary terms are less than 1 to avoid the auxiliary objectives suppressing the main objective. Each coefficient should be determined through grid search or staged annealing strategy on the validation set, which is a standard parameter tuning practice for multi-objective joint optimization.

[0195] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0196] To verify the effectiveness and robustness of the proposed model INFER, this invention conducts extensive experiments on two publicly available multimodal rumor detection datasets. First, the two datasets used, the experimental setup, and the evaluation metrics are described. Next, INFER is compared with several state-of-the-art rumor detection methods. Subsequently, through complexity analysis, parameter sensitivity analysis, ablation experiments, and case studies, the rationality of each module design and its contribution to model performance are further verified.

[0197] 1. Dataset

[0198] To verify the effectiveness of INFER, the existing multimodal rumor detection dataset MR was used. 2 Conduct experiments, MR 2 It consists of two subsets: one from an English-language social media platform and the other from a Chinese-language social media platform. MR 2 In addition to the post text and corresponding images, it also provides external evidence information related to the post content. The posts in the two datasets are labeled into three categories: non-rumor, rumor, or unverified rumor. The original division of the training set, validation set, and test set of the two datasets is maintained for the experiment. Table 1 gives the detailed statistical information of the two datasets.

[0199] Table 1. Dataset Statistics

[0200]

[0201] 2. Experimental setup and evaluation indicators

[0202] The dataset was divided into training, validation, and test sets in an 8:1:1 ratio. XLM-RoBERTa-Large was used as the text encoder, and CLIP ViT-L / 14 as the image encoder. The feature enhancement module shared 10 expert networks with a learning rate of 3×10⁻⁶. -5 The batch size was set to 16, and the number of training epochs was set to 20. To mitigate overfitting, an early stopping strategy was adopted, with a patience value of 5 training epochs. The experiment was conducted on a hardware platform equipped with an Intel Xeon Gold 6226R processor with a base frequency of 3.90GHz, 32 cores and 64 threads, 128GB of memory, and two NVIDIA RTX A6000 graphics processing units (GPUs).

[0203] To comprehensively evaluate model performance, this invention uses macro accuracy (M_Acc), precision (Pre), recall (Rec), and F1 score (F1) as performance evaluation metrics. Classification performance is statistically analyzed across three rumor categories, as shown below:

[0204] ;

[0205] ;

[0206] ;

[0207] ;

[0208] ;

[0209] in Indicates category; Indicates the total number of categories; Representative category The total number of samples in the sample; , , They represent The categories are true positives, false positives, and false negatives.

[0210] 3. Baseline

[0211] To verify the effectiveness of the proposed method, 12 methods, including single-modal and multi-modal methods, were selected as comparison baselines.

[0212] This invention presents a unimodal rumor detection method. It selects four unimodal baseline models: Etching-based and Topic-based Rumor Detection (ETDR), Topic-enhanced Rumor Detection (TDRD), Dual-channel Convolutional Neural Network (DC-CNN), and Deep Sentiment Dual Attention Network (DEDA). ETDR utilizes a Word2vec model and a Long Short-term Memory (LSTM) network to learn text semantic representations; TDRD enhances text feature capture capabilities by fusing topic information; DC-CNN utilizes a dual-channel convolutional network to construct local semantic features and global contextual dependencies; and DEDA mines text sentiment information for rumor detection through a deep attention mechanism.

[0213] This invention presents a multimodal rumor detection method. The multimodal collaborative attention network (MCAN), hierarchical multimodal collaborative attention network (HMCAN), Transformer-based multimodal authenticity verification method (MTTV), image-text similarity method (ITS), hierarchical graph attention multimodal rumor detection method (HGA-MMRD), multimodal matching and collaborative attention network (MMCAN), content consistency network (CCN), and deep evidence-enhanced text semantic attention network (DEETSA) are selected as multimodal baseline models. MCAN and HMCAN employ collaborative attention and hierarchical contextual attention mechanisms, respectively, for interactive image-text rumor detection. MTTV uses Transformer to learn cross-modal representations. ITS utilizes image-text similarity for authenticity judgment. HGA-MMRD uses a graph attention network to capture intermodal and cross-modal semantic interaction relationships. MMCAN combines knowledge distillation and matching mechanisms to achieve fine-grained image-text alignment. CCN and DEETSA utilize the consistency relationship between post content and external evidence to judge the authenticity of news.

[0214] 4. Experimental Results

[0215] Table 2 shows the comparative experimental results of INFER on an English social media platform dataset and a Chinese social media platform dataset. The precision, recall, and F1 score of the model were calculated for three categories of rumors: rumors, non-rumors, and unverified rumors. Macro-accuracy was also calculated. In the rumor detection task, INFER's macro-accuracy outperformed the baseline model, achieving macro-accuracies of 0.968 and 0.959 on the English and Chinese social media platform datasets, respectively, representing improvements of 4.42% and 3.79% compared to the second-best baseline model. This fully validates the model's effectiveness, demonstrating that it can achieve more accurate and stable discrimination between different truthfulness categories.

[0216] Table 2 shows the comparative experimental results on the two datasets. The best performance results are shown in bold, and the second-best results are indicated by underline.

[0217]

[0218] Table 2 shows that all multimodal rumor detection methods outperform unimodal rumor detection methods. On an English social media platform dataset, the best-performing unimodal model, DEDA, has an M_Acc of 0.885, while INFER reaches 0.968. On a Chinese social media platform dataset, DEDA's M_Acc is 0.887, while INFER reaches 0.959. Similarly, INFER shows significant performance improvements across the three rumor categories: non-rumor, rumor, and unverified rumor. This is mainly because unimodal methods rely solely on text content for authenticity judgment, making it difficult to fully extract supplementary information from images. In contrast, INFER, by jointly capturing text, images, and external evidence, can understand post content from multiple perspectives, thus achieving stronger semantic expression and authenticity judgment capabilities.

[0219] Furthermore, INFER was compared with advanced multimodal rumor detection models such as MCAN, HMCAN, and DEETSA. Experimental results show that while these models can improve detection performance through cross-modal feature fusion, INFER still achieved the best results on two datasets. On an English social media platform dataset, INFER's M_Acc reached 0.968, a 4.42% improvement over the best-performing baseline model, DEETSA; on a Chinese social media platform dataset, M_Acc reached 0.959, a 3.79% improvement over DEETSA. Particularly in the Rumor category detection task, INFER's F1 scores reached 0.936 and 0.938 respectively, both significantly outperforming existing multimodal models. This indicates that relying solely on text-image interaction and cross-modal attention mechanisms is insufficient to fully characterize the complex semantic relationships in the rumor propagation process, while introducing high-quality evidence, propagation intent information, and fine-grained modal alignment methods can further improve the model's ability to distinguish rumor content.

[0220] 5. Complexity Analysis

[0221] To evaluate the computational efficiency and practical feasibility of the proposed INFER model, this invention conducts a complexity analysis. With a fixed pre-trained text and image encoder, the main computational overhead of the model comes from four modules: feature enhancement, evidence scoring, fine-grained modality alignment, and propagation intent perception. Specifically, the feature enhancement (FEM) module structure includes... A shared expert and A task expert, assuming the input feature dimension is... Its time complexity is expressed as In the evidence scoring module, the model needs to compare the post content with... The relevance and quality of the candidate evidence are calculated, and the fusion feature dimension is set as follows. Then its time complexity is For the fine-grained modal alignment module, let the text sequence length be... The number of image regions is This module establishes word-level and region-level semantic associations through a cross-modal attention mechanism, with a time complexity of O(n log n). Furthermore, the communication intent perception module mainly includes emotion distribution prediction, emotion prototype matching, and gating fusion operations, assuming the number of emotion prototypes is... Its complexity is Therefore, the overall time complexity of INFER is expressed as: Although INFER incorporates multiple modules, including feature extraction, feature enhancement, evidence scoring, fine-grained modality alignment, propagation intent awareness, and adaptive feature fusion, it effectively controls the model parameter size and computational cost through a shared expert mechanism, gated fusion strategy, and a limited number of candidate evidence selections. Therefore, INFER maintains good computational efficiency while ensuring high detection performance, making it a promising candidate for large-scale social media multimodal rumor detection.

[0222] 6. Parameter sensitivity analysis

[0223] 6.1 Number of Expert Networks

[0224] To analyze the impact of the number of expert networks in the FEM module structure on model performance, experiments were conducted on an English social media platform dataset and a Chinese social media platform dataset with different numbers of expert networks. The results are as follows: Figure 6 (a) and Figure 6 As shown in (d) in the figure, where Figure 6 (a) in the figure shows the results of an expert network experiment on a dataset from a certain English social media platform. Figure 6 Figure (d) shows the experimental results of the expert network on a Chinese social media platform dataset. As the number of expert networks increases, the overall model performance shows a gradual improvement trend. On an English social media platform dataset, M_Acc increased from 94.9% to 96.8%, and on a Chinese social media platform dataset, M_Acc increased from 93.2% to 95.9%. This indicates that increasing the number of expert networks can enhance the model's ability to learn different semantic patterns, enabling it to extract richer text and image features from multiple perspectives. However, an excessive number of expert networks can lead to feature redundancy. Therefore, considering both model performance and computational cost, this invention ultimately sets the number of expert networks to 10.

[0225] 6.2 Learning Rate Size

[0226] The learning rate is a crucial hyperparameter affecting model convergence speed and optimization performance. Therefore, this invention tested the model performance with different learning rates on two datasets, and the experimental results are as follows: Figure 6 (b) and Figure 6 As shown in (e), where Figure 6 (b) in the figure shows the learning rate experiment results on a dataset from a certain English social media platform. Figure 6 (e) in the figure shows the experimental results of learning rate on a Chinese social media platform dataset. When the learning rate starts from 1×10... -5 Increased to 3×10 -5 The model's performance continuously improves initially, but begins to decline as the learning rate increases further. Specifically, on a dataset from a certain English social media platform, the model performs poorly at 3×10... -5 The highest M_Acc was achieved at that time, which is also within 3×10 on a certain Chinese social media platform dataset. -5 It achieves optimal performance at that time.

[0227] 6.3 batch size

[0228] To investigate the impact of batch size on model performance, this invention conducted experiments with different batch sizes on an English social media platform dataset and a Chinese social media platform dataset. The results are as follows: Figure 6 (c) and Figure 6 As shown in (f), where Figure 6 (c) in the figure shows the batch experiment results on a dataset from a certain English social media platform. Figure 6 Figure (f) shows the batch experiment results on a Chinese social media platform dataset. As the batch size changes, the model performance initially increases and then decreases. When the batch size is small, although the model parameters are updated more frequently, the gradient fluctuations are larger, easily affecting training stability. Conversely, when the batch size is too large, the model's generalization ability decreases, leading to performance degradation. The model achieves optimal performance on both datasets when the batch size is 16.

[0229] 6.4 Number of Emotion Categories

[0230] To analyze the impact of the number of sentiment categories in the communication intent perception module on model performance, this invention conducted experiments on an English social media platform dataset and a Chinese social media platform dataset, setting different numbers of sentiment prompt words. The results are as follows: Figure 7 As shown, where Figure 7 In the table, (a) represents the macro accuracy score on a dataset from a certain English social media platform. Figure 7 (b) in the table represents the non-rumor F1 score on a dataset from a certain English social media platform. Figure 7 (c) in the table represents the F1 score for rumors on a dataset from a certain English social media platform. Figure 7 (d) represents the F1 score of an unverified rumor on a dataset from a certain English social media platform. Figure 7 In the table, (e) represents the macro accuracy score on a dataset from a certain Chinese social media platform. Figure 7 In the table, (f) represents the non-rumor F1 score on a certain Chinese social media platform dataset. Figure 7 In the dataset, (g) represents the F1 score for rumors on a certain Chinese social media platform. Figure 7 In the table, (h) represents the F1 score of an unverified rumor on a Chinese social media platform dataset. Num represents the number of sentiment categories, with values ​​of 1, 2, 3, 4, and 5. Overall, the model performance gradually improves with the increase in the number of sentiment categories. When the number of sentiment categories is set to 5, the model achieves the best performance on both datasets. On both datasets, when only a small number of sentiment categories are used, the model struggles to fully capture the complex and diverse emotional expressions during rumor dissemination, thus limiting the quality of the dissemination intent representation. With an increase in the number of sentiment categories, the model can learn richer emotional semantic information, effectively enhancing its understanding of dissemination intent; therefore, M_Acc and the F1 scores for each category are significantly improved.

[0231] 6.5 Analysis of Loss Function Parameters

[0232] This invention focuses on analyzing two loss weights that contribute significantly to the overall optimization objective: the orthogonal constraint loss weight of the feature enhancement module. Consistency loss weights The experimental results are as follows Figure 8 As shown, where Figure 8 (a) in the figure shows the experimental results of the loss function parameters on a dataset from a certain English social media platform. Figure 8 Figure (b) shows the experimental results of the loss function parameters on a Chinese social media platform dataset. The figure shows that the model performance is affected by... and The combined influence of these factors shows a clear trend. On a dataset from a certain English social media platform and a dataset from a certain Chinese social media platform, when... and When the values ​​are small, the model struggles to fully learn the differentiated representations between expert networks and the semantic consistency between text and images, resulting in relatively low overall performance. As the parameters gradually increase, the model performance continuously improves, indicating that appropriate orthogonal constraints help enhance the complementarity of features learned by different experts, while cross-modal constraints can effectively reduce the semantic differences between text and image features, thereby improving the quality of multimodal representations. However, when the constraint weights increase further, the model performance begins to decline, suggesting that overly strong constraints limit the degrees of freedom of feature representation, causing the model to overemphasize feature decoupling or modal alignment, thus weakening the discriminative information needed for authenticity judgment. Combining the experimental results from both datasets, when... It is 0.15 and When the value is 0.05, the model achieves the best performance, indicating that the model can achieve the best balance between feature difference learning and cross-modal semantic consistency modeling.

[0233] 7. Ablation test

[0234] To verify the contribution of each module to the model performance, 10 ablation experiments were designed on an English social media platform dataset and a Chinese social media platform dataset. First, the experiments were conducted by removing individual modules one by one: (1) with FEM: removing the feature enhancement module; (2) with ES: removing the evidence scoring module; (3) with FMA: removing the fine-grained modality alignment module; (4) with PIA: removing the propagation intent perception module. The experimental results on the two datasets are shown in Table 3. INFER achieved the best performance on both datasets, with M_Acc reaching 0.968 and 0.959 on the English social media platform dataset and the Chinese social media platform dataset, respectively, which is significantly better than all ablation variants. This shows that each of the designed modules can improve the model's ability to represent rumor information from different perspectives. The model performance decreased to varying degrees after removing any module. The most significant performance drop was observed after removing the fine-grained modality alignment module. On an English social media platform dataset and a Chinese social media platform dataset, M_Acc dropped to 0.934 and 0.907, respectively, indicating that fine-grained modality alignment plays an important role in capturing semantic consistency and conflict relationships between images and text.

[0235] To further analyze the synergistic effects between modules, this invention combines different modules in pairs and conducts experiments on two datasets. Table 3 shows that when only some modules are retained, the model performance is significantly lower than the complete model. Meanwhile, the performance differences achieved by different module combinations reflect strong complementarity and synergistic gains among the modules. For example, on an English social media platform dataset, the ES+FMA combination model achieved an M_Acc of 0.962, indicating a strong synergistic effect between high-quality evidence and cross-modal alignment; while on a Chinese social media platform dataset, the FEM+FMA combination achieved a relatively better result with an M_Acc of 0.944, indicating that feature enhancement can further improve the representation ability of fine-grained modal interactions. However, regardless of the module combination used, its performance did not exceed that of the complete model INFER, indicating that a single module or a portion of the modules is insufficient to fully characterize the complex features of rumors. Only by simultaneously integrating multi-dimensional information such as content semantics, evidence quality, cross-modal relationships, and dissemination intent can we fully explore the discriminative clues in the rumor dissemination process, thereby obtaining better detection results.

[0236] Table 3 shows the ablation experimental results on the two datasets, with the best performance results shown in bold.

[0237]

[0238] To more intuitively verify the impact of each module on the model, T-SNE was used to perform a visualization analysis of the multimodal rumor representations learned by INFER and its variants. The results are as follows: Figure 9 As shown, where Figure 9 (a) in the figure is a T-SNE visualization of the final post representation result learned by the INFER model of this invention on a dataset of an English social media platform. Figure 9 (b) in the figure is a T-SNE visualization of the final post representation learned by the feature enhancement module on an English social media platform dataset after removing the feature enhancement module. Figure 9 (c) in the figure is a T-SNE visualization of the final post representation learned by the evidence scoring module on a dataset from an English social media platform after removing the evidence scoring module. Figure 9 (d) in the figure is a T-SNE visualization of the final post representation learned on an English social media platform dataset after removing the fine-grained modality alignment module. Figure 9 (e) in the figure is a T-SNE visualization of the final post representation learned by the propagation intent awareness module on a certain English social media platform dataset after removing the propagation intent awareness module. Figure 9 In the diagram, (f) is a T-SNE visualization of the final post representation learned by the INFER model of this invention on a Chinese social media platform dataset. Figure 9 In the diagram, (g) is a T-SNE visualization of the final post representation learned by the feature enhancement module on a Chinese social media platform dataset. Figure 9 (h) in the figure is a T-SNE visualization of the final post representation learned by the evidence scoring module on a Chinese social media platform dataset. Figure 9 In the diagram, (i) is a T-SNE visualization of the final post representation learned from a Chinese social media platform dataset after removing the fine-grained modality alignment module. Figure 9Figure (j) shows the T-SNE visualization of the final post representation learned on a Chinese social media platform dataset after removing the intention-to-spread module. In the complete INFER model, the three categories of samples—non-rumor, rumor, and unverified rumor—form a relatively clear cluster structure, indicating that the fused features have stronger discriminative power and better separability. However, after removing different modules, varying degrees of overlap appear between samples. After removing the FEM module, although the three categories of samples can form basic clusters, the intra-cluster distribution becomes significantly more dispersed, especially on the Chinese social media platform dataset, where the non-rumor category exhibits significant dispersion. After removing the Evidence Scoring (ES) module, both datasets show significant category crossover, indicating that the Evidence Scoring module can effectively filter high-quality evidence and reduce the interference of noisy evidence on feature learning. After removing the Fine-Grained Modality Alignment (FMA) module, although the overall cluster structure still exists, some category boundaries become blurred, especially on the Chinese social media platform dataset, where the interval between different categories is significantly reduced, indicating that the Fine-Grained Modality Alignment module can effectively enhance the semantic consistency representation between text and images and improve cross-modal discriminative ability. After removing the Propagation Intent Awareness (PIA) module, the overall clustering effect on both datasets is closer to the complete model than other ablation models. However, there are still some class boundary shrinkage and a small number of sample aliasing phenomena, indicating that the Propagation Intent Awareness module can further supplement the discriminative information at the level of propagation behavior and improve the model's ability to identify complex rumors.

[0239] Based on the combined experimental results and T-SNE visualization results, all four modules contribute positively to the model's performance. Specifically, the ES and FMA modules show more significant optimization effects on class boundaries, while the FEM module primarily improves the compactness of intra-class features, and the PIA module further enhances the discriminative ability between classes. The modules synergistically improve the model's feature representation capabilities, enabling the complete INFER model to achieve optimal clustering and detection performance on both datasets.

[0240] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A fine-grained multimodal rumor detection method based on intent perception and evidence scoring, characterized in that, Includes the following steps: S1, extract features from post text, post image, and corresponding external text and image evidence to obtain corresponding text features, image features, and evidence features; S2, the text features and image features are input into a multi-gated hybrid expert network for refinement to obtain text enhancement features and image enhancement features; the multi-gated hybrid expert network includes a shared expert pool, a gating network, and a task tower; the text features and image features are input into the shared expert pool, and multiple expert networks in the shared expert pool mine the potential information in the input features from different semantic perspectives to obtain multiple expert outputs; the importance weight of each expert output is dynamically determined by the gating network, and the multiple expert outputs are weighted and fused to obtain a shared representation; the task tower includes a first task tower and a second task tower, the first task tower maps the shared representation to a unified semantic space for modal alignment to obtain an aligned representation, and the second task tower refines the shared representation to obtain authenticity discrimination features; The text enhancement features and the image enhancement features are respectively composed of alignment representations of the corresponding modalities and authenticity discrimination features; S3, evaluate the quality scores of each piece of text evidence and each piece of image evidence respectively, and aggregate the features of each piece of evidence according to the quality weights to obtain the final evidence representation, including: for text evidence, concatenating the text enhancement features with the features of each piece of text evidence to obtain a joint representation, calculating the quality score of each piece of text evidence through a quality assessment network, filtering invalid text evidence using a masking method, normalizing the quality scores of each piece of text evidence to obtain quality weights, and weighting and summing the features of each piece of text evidence according to the quality weights to obtain the aggregated text evidence features; for image evidence, obtain the aggregated image evidence features in the same way; concatenate and project the aggregated text evidence features and the aggregated image evidence features to obtain the final evidence representation; S4. Construct cross-modal associations between fine-grained text features and image region features, perform bidirectional semantic alignment, and obtain fine-grained modal alignment features; S5, extract textual and image propagation intentions from the text enhancement features and image enhancement features respectively, and obtain propagation intention perception features through gating fusion; wherein, the extraction of textual propagation intention includes: performing sentiment prediction on text sequence features to obtain the probability distribution of each fine-grained text belonging to three sentiment categories: positive, neutral, and negative; measuring the degree of text sentiment fluctuation based on the variance of the sentiment probability sequence to obtain sentiment fluctuation features; constructing a rhetoric dictionary, statistically analyzing the coverage of rhetorical expressions in the post text, and projecting the coverage onto a high-dimensional semantic space to obtain rhetorical features; fusing the text enhancement features, the sentiment fluctuation features, and the rhetorical features to obtain the textual propagation intention; the extraction of image propagation intention includes: using the text... The encoder encodes sentiment prompts to construct a visual sentiment prototype matrix; the image enhancement features are mapped to the semantic space of the text encoder and normalized; the similarity between the normalized image representation and each sentiment prototype in the visual sentiment prototype matrix is ​​calculated to obtain a visual sentiment similarity vector; the cosine similarity between the text sentiment distribution and the visual sentiment similarity vector is calculated as a cross-modal sentiment consistency feature; the image enhancement features, the visual sentiment similarity vector, and the cross-modal sentiment consistency feature are fused to obtain the image propagation intent; the text propagation intent and the image propagation intent are mapped to a unified space and a gating weight is calculated; the two are adaptively integrated according to the gating weight to obtain the propagation intent perception feature; S6. Project each feature onto a unified semantic space, dynamically allocate weights through a gating mechanism and perform weighted fusion to obtain the fused high-level semantic representation; S7. Input the high-level semantic representation into the classifier to obtain the prediction result of the authenticity category to which the post belongs.

2. The method according to claim 1, characterized in that, S1 includes: using a pre-trained language model as a text encoder to encode the post text to obtain post text features, and using the text encoder with shared parameters to encode each piece of text evidence to obtain text evidence features; using a pre-trained visual model as an image encoder to encode the post image to obtain post image features, and using the image encoder with shared parameters to encode each piece of image evidence to obtain image evidence features; the dimensions of the post text features and the text evidence features are consistent, and the dimensions of the post image features and the image evidence features are consistent.

3. The method according to claim 1, characterized in that, S4 includes: calculating the semantic similarity between each fine-grained text feature and each image region feature to obtain a cross-modal association matrix; normalizing the cross-modal association matrix along the image region dimension to obtain a text-to-image attention matrix, and obtaining the image semantic representation corresponding to each fine-grained text through attention weighting; normalizing the cross-modal association matrix along the fine-grained text dimension to obtain an image-to-text attention matrix, and obtaining the text semantic representation corresponding to each image region through attention weighting; performing average pooling on the image semantic representation and the text semantic representation respectively, and projecting the pooling results after concatenation to obtain the fine-grained modal alignment feature.

4. The method according to claim 1, characterized in that, S6 includes: projecting the alignment representation and authenticity discrimination feature in the text enhancement feature and the image enhancement feature, the final evidence representation, the fine-grained modality alignment feature, and the propagation intent perception feature into a unified semantic space to construct a joint representation; using a gating network to learn the weight coefficients corresponding to each feature to adaptively measure the importance of different features; weighting each feature according to the weight coefficients, and summing the weighted features to obtain the fused high-level semantic representation.

5. The method according to claim 1, characterized in that, S7 includes: inputting the high-level semantic representation into a fully connected classifier, obtaining the predicted probability of the post belonging to each real category through a normalization function, and taking the category corresponding to the maximum predicted probability as the prediction result; during model training, the cross-entropy loss between the prediction result and the real label is used as the main classification objective, and multiple auxiliary optimization objectives are introduced for joint optimization. The auxiliary optimization objectives include orthogonal constraint loss for feature enhancement, quality regularization loss for evidence scoring, constraint consistency loss for fine-grained modality alignment, and contrast loss for propagation intent perception.

6. The method according to claim 1, characterized in that, The authenticity category in S7 includes non-rumor, rumor, and unverified rumor.

Images