Rumor detection method based on social media

Abstract

The invention discloses a social media-based rumor detection method, and belongs to the technical field of natural language processing and deep learning, and the method comprises the steps: obtaining and preprocessing an image, a text and user interaction behavior data on a social media; image-text modal features are extracted, and social interaction sub-graph nodes are initialized; for a target node in the sub-graph, generating a candidate node set through random walk, and inferring and adding a potential edge by synthesizing a PageRank influence score and a vector similarity of the candidate node set to realize enhancement of the social interaction sub-graph; extracting social interaction modal features from the heterogeneous social network formed by merging after enhancement based on an edge level attention mechanism; and fusing the visual, text and social interaction modal features by using a bidirectional fusion mechanism based on a selective state space model to obtain a final multi-modal fusion representation for rumor detection. The rumor detection accuracy can be improved.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and deep learning technology, and in particular relates to a rumor detection method based on social media. Background Technology

[0002] In recent years, deep learning technology has made significant progress in the field of social media rumor detection. Existing methods mainly focus on identifying rumors using single or partial modalities of information. One type of method is multimodal rumor detection, which extracts and analyzes image and text features from posts, using the semantic consistency or inconsistency between visual and textual information to help determine the authenticity of the information. Another type of method focuses on utilizing social modalities, constructing user interaction networks (such as forwarding and commenting relationships) to analyze information dissemination patterns, user influence, and group behavior characteristics to identify suspicious information. These technologies have improved the automation level of rumor detection to some extent, reduced the burden of purely manual verification, and provided preliminary technical solutions for information governance on social media platforms.

[0003] However, existing rumor detection technologies still have significant limitations and shortcomings. First, traditional multimodal methods typically only integrate image and text information, neglecting the crucial dimension of social relationships and group effects in the information dissemination process. Second, methods that rely solely on social modalities often overlook the intuitive evidence provided by media content such as images, leading to insufficient information utilization. Furthermore, when modeling user interaction behavior, existing methods are mostly limited to direct interactions within a single post, making it difficult to capture potentially complex user relationships across posts, thus failing to construct a complete social interaction environment. Simultaneously, existing methods often employ manually defined patterns to model interaction behavior, introducing subjective bias and lacking flexibility, making it difficult to adapt to the diverse user behavior patterns across different platforms. Finally, when integrating multiple heterogeneous modalities such as images, text, and social media, existing cross-modal fusion mechanisms struggle to fully characterize and utilize the complex nonlinear relationships and complementary information between these modalities, limiting the model's ability to deeply understand and discriminate comprehensive information representations. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a rumor detection method based on social media, thereby resolving the issues present in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for detecting rumors based on social media, comprising:

[0006] S1: Acquire image data, text data, and user interaction behavior data from social media;

[0007] S2: Extract visual modal features and text modal features based on the image data and text data;

[0008] S3: Construct an initial social interaction subgraph based on user interaction behavior data, and initialize the node features in the subgraph using the text modality features;

[0009] S4: For the target node in the initial social interaction subgraph, a candidate node set is generated by random walk, and the PageRank influence score of the nodes in the candidate node set and their vector similarity with the target node are calculated. Based on the calculation results, potential edges are inferred and added to the target node to obtain the edge-enhanced social interaction subgraph.

[0010] S5: Based on the edge-level attention mechanism, extract social interaction modality features from the heterogeneous social network formed by merging the edge-enhanced subgraphs;

[0011] S6: Using a bidirectional fusion mechanism based on a selective state-space model, the visual modal features, text modal features, and social interaction modal features are fused to obtain the final multimodal fusion representation for rumor detection.

[0012] Preferably, in step S1, the image data undergoes scale normalization, channel format conversion, and color channel standardization; the text data undergoes word segmentation, stop word filtering, and word form normalization.

[0013] Preferably, in step S4, the formula for comprehensively calculating the PageRank influence score of nodes in the candidate node set is as follows:

[0014] ;

[0015] ;

[0016] in, Let this be a vector representing the PageRank values ​​of each node at the (t+1)th iteration; Let be the PageRank value vector of each node at the t-th iteration; For the initial social interaction subgraph Let s be the adjacency matrix of the source node and t be the target node. Adjacency matrix The diagonal matrix is ​​denoted by d, where d is the damping factor and E is the set of edges of the subgraph.

[0017] Preferably, the vector similarity is a cosine similarity calculated based on the node embedding vector.

[0018] Preferably, in step S5, the edge-level attention mechanism is a heterogeneous attention mechanism, which defines independent learnable projection matrices and attention weight matrices for different types of nodes and edge relationships.

[0019] Preferably, the heterogeneous attention mechanism calculates the relevance of each attention head based on relation triples, using the following formula:

[0020] ;

[0021] Where K(s) and Q(t) represent the Key matrix of the source node's attention and the Query matrix of the target node's attention, respectively, T represents the transpose operation of the matrix, and W is the learnable parameter. This represents the importance coefficient of the relation triple. This is the attention weight matrix. is the scaling factor, representing the dimension of the vector; i represents the i-th attention head.

[0022] Preferably, in step S6, the bidirectional fusion mechanism based on the selective state-space model is implemented through the Co-mamba module. After the Co-mamba module performs selective scanning encoding on the two input modal branches, bidirectional information exchange and collaborative updates are performed through a symmetrical cross-connection structure.

[0023] Preferably, after step S6, a loss calculation step is also included: inputting the final multimodal fusion representation into the classifier to obtain the prediction result, and calculating the cross-entropy loss between the prediction result and the true label to optimize the model.

[0024] Preferably, in the loss calculation step, the total loss function further includes the modality alignment loss between text modality features and social interaction modality features.

[0025] Secondly, the present invention also provides a rumor detection device based on social media, comprising:

[0026] The data processing module is used to execute steps S1 to S3 in the first aspect;

[0027] The social graph enhancement and feature extraction module is used to perform steps S4 and S5 in the first aspect;

[0028] The multimodal fusion and detection module is used to perform step S6 in the first aspect.

[0029] Compared with the prior art, the present invention has the following advantages and technical effects:

[0030] This invention employs a technical solution that involves "performing random walks to generate a set of candidate nodes for target nodes in an initial social interaction subgraph, and comprehensively calculating the PageRank influence scores of the candidate nodes and their vector similarity to the target nodes to infer and add potential edges to the target nodes." This approach enables the automatic and data-driven discovery of unobserved potential connections between users and posts. This technique overcomes the problem of incomplete social environment representation caused by existing technologies that only utilize explicit and local social interaction behaviors. It achieves a more comprehensive and deeper modeling of user social interaction networks, effectively enhancing the richness and representativeness of social modality features.

[0031] This invention employs a technical solution of "extracting social interaction modal features using a graph neural network based on an edge-level attention mechanism after obtaining edge-enhanced social interaction subgraphs and merging them into heterogeneous social networks." This approach adapts differentiated attention calculations to different types of nodes and edge relationships within the network, enabling the precise capture of complex semantic information and structural differences in heterogeneous social interactions. This effectively avoids the subjective bias and lack of flexibility introduced by manually defined interaction patterns in existing technologies, thus improving the adaptability and accuracy of social feature extraction.

[0032] This invention employs a technical solution that "introduces a bidirectional fusion mechanism based on a selective state-space model to fuse visual, textual, and social interaction modal features when fusing multimodal information." This dynamically models and integrates complex nonlinear associations and long-sequence dependencies between different modalities. This addresses the shortcomings of existing cross-modal fusion mechanisms in fully characterizing the deep relationships among multi-source heterogeneous information such as images and text, and social media, thereby generating a more discriminative and comprehensive information representation. Attached Figure Description

[0033] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 This is a flowchart of a rumor detection method based on social media according to an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram illustrating the mining of social interaction behaviors according to an embodiment of the present invention. Detailed Implementation

[0036] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0038] Example 1

[0039] like Figure 1 As shown, this embodiment provides a rumor detection method based on social media, including:

[0040] S1: Acquire image data, text data, and user interaction behavior data from social media;

[0041] Furthermore, in step S1, scale normalization, channel format conversion, and color channel standardization are performed on the image data; word segmentation, stop word filtering, and word form normalization are performed on the text data.

[0042] Specifically, this embodiment reads multimodal data from social media, including images, text, and user interaction behavior, and performs data preprocessing operations. For images, scale normalization is first performed, and the storage format is transformed into a directly computable channel × height × width. Then, mean removal and variance scaling are performed on the three color channels of the image to obtain a standardized image tensor that meets the model input specifications. For text data, standardization preprocessing is performed, including word segmentation, stop word filtering, and lexical normalization, to remove noise information and improve the consistency of text feature expression. For user interaction behavior, the association between posts, users, and comments is established using information such as the IDs within the constructed dataset to construct an initial social interaction subgraph.

[0043] Define posts on social media as The text visual encoder uses a Chinese CLIP extraction model, and the extraction process is as follows:

[0044] (1)

[0045] (2)

[0046] For the feature initialization of social interaction nodes, posts and comments are initialized using their corresponding text features to obtain the node features of posts and comments. For user nodes, the features are initialized based on the features of their associated post and comment nodes.

[0047] S2: Extract visual modal features and text modal features based on the image data and text data;

[0048] S3: Construct an initial social interaction subgraph based on user interaction behavior data, and initialize the node features in the subgraph using the text modality features;

[0049] Specifically, a text-image preprocessing training model is used to extract image-text modal features from the preprocessed image and text, and the social interaction node features are initialized based on the text features.

[0050] As one specific implementation method, the nodes include post nodes P, user nodes C, and comment nodes U;

[0051] A post node P is a single piece of information in social media (such as a post on Weibo or Twitter, or a text / video post on Douyin). Each post node corresponds to a specific information instance on the platform that is "published, searchable, and has a unique ID", rather than an abstract concept of "post". Its associated data is all structured data that can be directly extracted from the platform's backend and has a clear physical carrier (text / image / video files stored on the server).

[0052] User node C corresponds to "a registered account instance on a social media platform", including Weibo registered user instances and Xiaohongshu registered user instances. Each user node corresponds to "a specific user instance that has been registered and has a unique account ID" on the platform. The associated data are all user behavior records and account attributes that can be traced in the platform's backend, and have a clear subject reference (real registered user or institutional account).

[0053] Comment node U corresponds to "a comment instance posted by a user for a specific post". Each comment node corresponds to "a specific interactive content instance with a unique comment ID for a specific post". The associated data is clearly bound to "posting user - post - comment content". It is a structured interactive record that can be directly extracted from the platform backend, rather than an abstract concept of "comment".

[0054] For post node P and comment node C, this embodiment initializes them using their content. For user node U, it initializes using its published posts and comments.

[0055] The initial social interaction subgraph is a network model of the explicit interaction relationships among "user-post-comment" and consists of a "node set + edge set": the node set is the user nodes, post nodes, and comment nodes covered by the subgraph (such as all interactive users and comments surrounding a suspected rumor post); the edge set is the explicit interaction relationships between nodes (such as the forwarding / commenting relationship between "user-post" and the subordinate relationship between "comment-post"). Each edge is labeled with the relationship type and interaction strength (such as forwarding relationship strength = number of forwards / total number of interactions for the post), which is the technical basis for subsequent subgraph enhancement and feature extraction.

[0056] The initial social interaction subgraph serves as a "carrier" for capturing social interaction modal features. Through the subgraph structure, it can quantitatively represent the local patterns of information dissemination (such as the participation of high-influence users and the distribution of sentiment tendencies in comments), providing structured input for subgraph enhancement (potential edge addition) in step S4 and social interaction feature extraction (edge-level attention mechanism) in step S5, ultimately achieving the association and fusion of social modal information and text-image modal information.

[0057] S4: For the target node in the initial social interaction subgraph, a candidate node set is generated by random walk, and the PageRank influence score of the nodes in the candidate node set and their vector similarity with the target node are calculated. Based on the calculation results, potential edges are inferred and added to the target node to obtain the edge-enhanced social interaction subgraph.

[0058] Specifically, each constructed social interaction subgraph is traversed. For the target node in the subgraph, a set of candidate nodes is obtained through multiple graph random walks. Then, PageRank and local vector similarity are used to infer potential interaction behaviors between users, thus completing the incremental enhancement of potential edges on the social interaction subgraph. Figure 2 As shown.

[0059] Step S4 specifically includes:

[0060] S401: For the initial social interaction network constructed It includes three interactive behaviors: user posting, user commenting, and commenting on posts. For the subgraph... The set of candidate nodes for the target node t in the algorithm All remaining nodes can be defined as candidate nodes. Nodes *s* that can be indirectly reached from the target node *t* are identified from the topology using random walks, where *l* is the random walk length. For each subgraph, this embodiment uses PageRank to compute candidate entity nodes. The influence of PageRank vector. The calculation is as follows:

[0061] (3)

[0062] (4)

[0063] in, This is the PageRank value vector of each node at the (t+1)th iteration; Let be the PageRank value vector of each node at the t-th iteration; For the initial social interaction subgraph Let s be the adjacency matrix of the source node and t be the target node. Adjacency matrix The diagonal matrix is ​​denoted by d, where d is the damping factor and E is the set of edges of the subgraph.

[0064] S402: For the target node t and the source node s, use their node embedding vectors. and To calculate cosine similarity as a measure of local similarity:

[0065] (5)

[0066] S403: Obtain the final selection score between the current node and potential nodes through a comprehensive evaluation function F. Considering both network topology and semantic information, select the top K nodes with the highest scores as the final set of potential nodes. and potential edge set The formula is:

[0067] (6)

[0068] (7)

[0069] Where F(s,t) represents the potential selection score between source node s and target node t. and Indicates an adjustable parameter. Represents the union of sets.

[0070] S5: Based on the edge-level attention mechanism, extract social interaction modality features from the heterogeneous social network formed by merging the edge-enhanced subgraphs;

[0071] Furthermore, in step S5, the edge-level attention mechanism is a heterogeneous attention mechanism, which defines independent learnable projection matrices and attention weight matrices for different types of nodes and edge relationships.

[0072] Specifically, the enhanced social interaction subgraphs are merged into a complete social interaction network, which contains edge and type information between heterogeneous nodes. Then, a graph feature extraction network based on edge-level attention is used to extract social interaction features.

[0073] Step S5 specifically includes:

[0074] S501: The set of potential nodes obtained from node t Subsequently, in this embodiment, the adjacency matrix dimension of the subgraph is set to N×N, where N is the total number of nodes. For the subgraph... In this embodiment, firstly Initialize it as an N×N zero matrix, and then according to... and Set the potential direct neighbors of the target node t in the corresponding subgraph to... In the middle. Specifically, if a node is a potential direct neighbor of the target node, the corresponding element in the matrix is ​​set to 1. Finally, the new subgraph. The enhanced adjacency matrix becomes And according to each subgraph Enhanced adjacency matrix in Synthesizing new heterogeneous social networks :

[0075] (8)

[0076] in, , This indicates an element-wise OR operation on the matrix.

[0077] S502: Given a target node t, all its neighboring nodes, and a triplet of relationships between them. ,in Indicates the node type. This indicates the type of heterogeneous relationship. In this embodiment, the representation of the target node t is updated based on the characteristics of the source node s.

[0078] First, for each edge e=(s,t), this embodiment defines independent linear projection matrices for different types of nodes and relations. The features of the source and target nodes are mapped to the K and Q spaces respectively through type-dependent linear transformations, i.e.:

[0079] (9)

[0080] in This represents the i-th attention head. This represents the nodes of the previous layer. Subsequently, to model the semantic differences between different types of relationships, a hierarchy is established for each edge type. Introduce a dedicated attention weight matrix And calculate the relevance of each attention head:

[0081] (10)

[0082] Where K(s) and Q(t) represent the Key matrix of the source node's attention and the Query matrix of the target node's attention, respectively, T represents the transpose operation of the matrix, and W is the learnable parameter. This represents the importance coefficient of the relation triple. Here is the attention weight matrix. is the scaling factor, representing the dimension of the vector; i represents the i-th attention head.

[0083] Multi-head heterogeneous attention can be achieved by normalizing attention weights through the Softmax operation:

[0084] (11)

[0085] To obtain the feature differences between different nodes and relation types, the source node features are mapped to the message space. For each attention head i, the message is computed as follows:

[0086] (12)

[0087] in, This is the projection matrix related to the edge type. After concatenating all attention head messages, the overall multi-head message representation is obtained:

[0088] (13)

[0089] Subsequently, in the target-specific aggregation phase, messages from neighboring nodes are weighted and aggregated according to attention weights to obtain the intermediate representation of the target node:

[0090] (14)

[0091] Finally, to remap the aggregated features back to the spatial distribution corresponding to the target node type, a type-specific linear transformation is applied, and residual connections are introduced to enhance stability:

[0092] (15)

[0093] in, This represents a non-linear activation function.

[0094] S6: Using a bidirectional fusion mechanism based on a selective state space model, the visual modal features, text modal features, and social interaction modal features are fused to obtain the final multimodal fusion representation for rumor detection;

[0095] Furthermore, in step S6, the bidirectional fusion mechanism based on the selective state-space model is implemented through the Co-mamba module. After the Co-mamba module performs selective scanning encoding on the two input modal branches respectively, bidirectional information exchange and collaborative updates are performed through a symmetrical cross-connection structure.

[0096] Specifically, the extracted visual modal features, text features, and social interaction features are fused using a bidirectional fusion mechanism based on a selected state space to obtain the final comprehensive information modal representation.

[0097] Step S6 specifically includes:

[0098] S601: The Co-mamba module aims to build a bidirectional collaborative interaction mechanism between two feature branches, so as to maintain the independence of their respective representations and achieve dynamic enhancement across branches. Let the input features be... and Within the module, each path first passes through a local-sequence hybrid encoder consisting of convolution, SiLU activation, and selective scanning operators to extract its local patterns and long sequence dependencies. This process can be represented as:

[0099] (16)

[0100] in, This represents a non-linear activation function.

[0101] S602: Subsequently, the two branches exchange semantics bidirectionally through a cross-connection structure: the intermediate representation from branch a, after undergoing a set of linear mappings and nonlinear transformations, is sent to branch b as supplementary information, and vice versa, thus forming a symmetric collaborative update mechanism, which can be described as:

[0102] (17)

[0103] in and This represents a nonlinear mapping used for cross-branch information injection. Through this bidirectional interaction mechanism, each network not only maintains its own structured features but also dynamically adjusts itself based on the contextual features of the other branch, thereby effectively modeling joint dependencies in cross-modal or cross-perspective scenarios.

[0104] S603: After completing the cross-branch collaborative update, both branches are mapped back to the original channel space through the output projection layer, and residual connections are added to enhance training stability and semantic consistency. The final output is:

[0105] (18)

[0106] in and This is the output mapping function. Through the coupling of local convolutional feature extraction, sequence modeling, bidirectional collaborative interaction, and residual enhancement, the Co-mamba module achieves deep fusion and consistent representation of multi-source inputs.

[0107] For simplicity, the above process can be summarized as follows:

[0108] (19)

[0109] (20)

[0110] For text modalities, visual modalities, and social modalities, the Co-mamba module is used to fuse them to obtain separate results. , and The final multimodal features can be represented as:

[0111] (twenty one)

[0112] S7: Input the final multimodal fusion representation into the classifier to obtain the prediction result, and calculate the cross-entropy loss between the prediction result and the true label to optimize the model.

[0113] Furthermore, in the loss calculation step, the total loss function also includes the modality alignment loss between text modality features and social interaction modality features.

[0114] Specifically, a loss function is defined and calculated to optimize the accuracy of the detection model, and the model parameters are optimized by minimizing the cross-entropy loss function.

[0115] This embodiment designs a This provides a feedback mechanism for model training. By calculating the loss value, we can know how well the model performs in the current state.

[0116] Before fusing textual and social modal features, heterogeneous modalities are first aligned. Specifically, for and Transform them to the same modal feature space:

[0117] (twenty two)

[0118] in and These are learnable parameters. Then, modality alignment is performed using the distance between them and the MSE loss to obtain optimized text features. and social modality features :

[0119] (twenty three)

[0120] During the training phase, the final multimodal data is fed into a fully connected layer to predict whether it is a rumor:

[0121] (twenty four)

[0122] in This represents the probability that a post is predicted to be a rumor. This example then uses the cross-entropy loss function to express the prediction loss:

[0123] (25)

[0124] The overall training of the model is accomplished by the total loss:

[0125] (26)

[0126] in, This represents the alignment loss between text features and social modality features. This represents the loss of classification results during training.

[0127] This example calculates the accuracy of rumor detection on three datasets containing social modalities and one dataset containing a large-scale text and image modal. The three datasets containing social modalities include Weibo and Twitter. The large-scale text and image modal dataset includes Weibo, Twitter, and TikTok.

[0128] The results show that this embodiment has a high detection accuracy on datasets containing social modalities or large-scale image and text modalities. The rumor detection accuracy results on different datasets are shown in Table 1.

[0129] Table 1

[0130]

[0131] This embodiment verifies the effectiveness of social interaction behavior inference, social modality feature extraction, and selective state space-based bidirectional fusion mechanism through evaluation studies.

[0132] In inferring social interaction behavior in a draw, we analyzed the impact on rumor detection accuracy by systematically removing key components from the model. First, removing the PageRank component from the inference process revealed that the algorithm could not effectively utilize the node with the greatest global influence, leading to poor inter-node correlation and insufficient extraction of interaction features in subsequent local inferences. Next, removing the calculation of local vector similarity in the inference process showed that the accuracy improvement was lower. This is because the enhanced similarity links between nodes can individually strengthen the representation of interactive nodes.

[0133] For social modality feature extraction, we systematically and manually define interaction behavior patterns to analyze their impact on detection efficiency. By manually constructing interaction behavior patterns for feature extraction, we found that the detection accuracy varies significantly across different datasets. This is because different datasets involve different social platforms, and users exhibit different preferred interaction behaviors. Manually defined patterns have significant biases and cannot accurately reflect the internal characteristics of different behavior patterns, thus reducing the generalization ability of the method.

[0134] For the selective state-space bidirectional fusion mechanism, the effect was observed by systematically removing the Co-mamba fusion method from the model. Visualized node representations after removal revealed that the model struggles to effectively distinguish between genuine and fake posts of different types, and faces difficulties in leveraging the complex relationships between multiple modalities on social platforms.

[0135] These experiments revealed that rumor detection in social modalities is significantly affected by module removal, indicating that this embodiment is well-suited for rumor detection on social platforms with rich social interactions. Overall, the complete model, including all components, delivers the best results, validating the effectiveness of these components in the model. Their combined effect ensures high accuracy in fact-finding when handling rumor detection on social platforms.

[0136] This embodiment can improve the efficiency of rumor detection while ensuring high accuracy, providing a reliable solution for information governance on social platforms. It has significant advantages in detecting complex post information and user associations, and reduces the workload of manual fact-checking.

[0137] Example 2

[0138] This embodiment provides a rumor detection device based on social media, including:

[0139] The data processing module is used to execute steps S1 to S3 of Embodiment 1;

[0140] Specifically, the data acquisition module is used to read multimodal post sequences on social media, including text, images, and user social interaction information, and to perform data preprocessing operations, such as normalizing images and cleaning and segmenting text, so that the subsequent model can extract and analyze features.

[0141] The social graph enhancement and feature extraction module is used to perform steps S4 and S5 of Embodiment 1;

[0142] Specifically, the social graph augmentation module performs tensor augmentation on the initially constructed social interaction subgraph. First, it obtains a candidate node set through multiple random walks on the subgraph. Then, it uses PageRank and local similarity calculations to globally and locally filter the nodes, obtaining a potential node set. Finally, it establishes connections between the target node and the candidate nodes. The feature extraction module comprises three parts: text feature extraction, image feature extraction, and social interaction feature extraction. For text and image features, it extracts features using the pre-trained CLIP model. Based on the extracted text features, it initializes the user's social interaction features, then infers interaction behavior and extracts the social interaction features.

[0143] The multimodal fusion and detection module is used to perform step S6 of Embodiment 1.

[0144] Specifically, the multimodal fusion and detection module is used to fuse features from multiple modalities. Based on a selectable state space, it receives dual-stream input and performs symmetrical fusion to learn the complex relationships between multiple modalities on social media and obtain a complete representation of the information.

[0145] The loss evaluation module is used to define and calculate loss functions to optimize the accuracy of the detection model. It optimizes model parameters using MSE loss and cross-entropy loss functions.

[0146] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A rumor detection method based on social media, characterized in that, Includes the following steps: S1: Acquire image data, text data, and user interaction behavior data from social media; S2: Extract visual modal features and text modal features based on the image data and text data; S3: Construct an initial social interaction subgraph based on user interaction behavior data, and initialize the node features in the subgraph using the text modality features; S4: For the target node in the initial social interaction subgraph, a candidate node set is generated by random walk, and the PageRank influence score of the nodes in the candidate node set and their vector similarity with the target node are calculated. Based on the calculation results, potential edges are inferred and added to the target node to obtain the edge-enhanced social interaction subgraph. S5: Based on the edge-level attention mechanism, extract social interaction modality features from the heterogeneous social network formed by merging the edge-enhanced subgraphs; S6: Using a bidirectional fusion mechanism based on a selective state-space model, the visual modal features, text modal features, and social interaction modal features are fused to obtain the final multimodal fusion representation for rumor detection.

2. The method according to claim 1, characterized in that, In step S1, scale normalization, channel format conversion, and color channel normalization are performed on the image data; word segmentation, stop word filtering, and word form normalization are performed on the text data.

3. The method according to claim 1, characterized in that, In step S4, the formula for comprehensively calculating the PageRank influence score of nodes in the candidate node set is as follows: ； ； in, Let this be a vector representing the PageRank values ​​of each node at the (t+1)th iteration; Let be the PageRank value vector of each node at the t-th iteration; For the initial social interaction subgraph Let s be the adjacency matrix of the source node and t be the target node. Adjacency matrix The diagonal matrix is ​​denoted by d, where d is the damping factor and E is the set of edges of the subgraph.

4. The method according to claim 1, characterized in that, The vector similarity is a cosine similarity calculated based on the node embedding vectors.

5. The method according to claim 1, characterized in that, In step S5, the edge-level attention mechanism is a heterogeneous attention mechanism, which defines independent learnable projection matrices and attention weight matrices for different types of nodes and edge relationships.

6. The method according to claim 5, characterized in that, The heterogeneous attention mechanism calculates the relevance of each attention head based on relation triples, using the following formula: ； Where K(s) and Q(t) represent the Key matrix of the source node's attention and the Query matrix of the target node's attention, respectively, T represents the transpose operation of these matrices, and W is the learnable parameter. This represents the importance coefficient of the relation triple. Here is the attention weight matrix. is the scaling factor, representing the dimension of the vector; i represents the i-th attention head.

7. The method according to claim 1, characterized in that, In step S6, the bidirectional fusion mechanism based on the selective state space model is implemented through the Co-mamba module. After the Co-mamba module performs selective scanning encoding on the two input modal branches, bidirectional information exchange and collaborative updates are performed through a symmetrical cross-connection structure.

8. The method according to claim 1, characterized in that, After step S6, a loss calculation step is also included: inputting the final multimodal fusion representation into the classifier to obtain the prediction result, and calculating the cross-entropy loss between the prediction result and the true label to optimize the model.

9. The method according to claim 8, characterized in that, In the loss calculation step, the total loss function also includes the modality alignment loss between text modality features and social interaction modality features.

10. A rumor detection device based on social media, characterized in that, include: The data processing module is used to perform steps S1 to S3 in claim 1; A social graph enhancement and feature extraction module is used to perform steps S4 and S5 in claim 1; A multimodal fusion and detection module is used to perform step S6 in claim 1.

Images