A multi-modal rumor detection method and system based on implicit cue enhancement

Abstract

The present application relates to the field of artificial intelligence and network information security, and particularly relates to a multi-modal content rumor detection technology. The method comprises: obtaining multi-modal data containing text information and corresponding image information; performing feature extraction on the text information and the image information respectively to obtain text feature representation and image feature representation; performing multi-modal feature modeling based on the text feature representation and the image feature representation, including obtaining graphic-text semantic consistency features through cross-modal interaction, extracting rumor clue features through semantic encoding and multi-label classification and fusing with image features, and performing semantic enhancement on the text features and fusing with image features; fusing the features to obtain comprehensive feature representation, and performing classification processing based on the comprehensive feature representation to output rumor detection results. The present application is suitable for social platform content review, network information management, public opinion monitoring, risk early warning and network information security protection and other scenes, and provides more stable and accurate technical support for multi-modal false information identification.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and network information security, specifically to a multimodal content rumor detection technology and system. Background Technology

[0002] With the development of the internet and social media platforms, information dissemination has evolved from single-text to multimodal dissemination, incorporating text, images, videos, and other formats. Information posted by users on social networks typically includes both text descriptions and accompanying images. While this multimodal expression improves the efficiency and readability of information dissemination, it also provides a more covert and complex medium for the spread of misinformation and rumors. Rumors often use a combination of text and images to enhance their credibility and spread, thereby causing social panic, a crisis of public trust, and even threatening network information security. Therefore, identifying these diverse rumors is crucial to reducing the negative impact of social networks. Existing technologies mainly rely on image-text semantic alignment, explicit rumor clue extraction, or general multimodal feature fusion, and suffer from the following core bottlenecks:

[0003] (1) Image-text semantic alignment methods mainly detect rumors by judging whether the text content and the corresponding image are semantically consistent. However, in real-world scenarios, there are often cases where the image and text information are consistent but the content itself is distorted. Relying solely on the consistency of image and text makes it difficult to identify rumors that are consistent in image and text but lack the authenticity of the event, lack factual basis, or have misleading expressions.

[0004] (2) Explicit rumor clue methods usually rely on superficial features such as emotional expressions, exaggerated words, and obvious conflict descriptions, making it difficult to capture more discriminative implicit clues in rumors, such as the narrative style, semantic tendency, content nature, degree of common sense violation, and potential misleading patterns of the text. Therefore, they are less effective in identifying rumors that are more concealed and disguised.

[0005] (3) Most existing multimodal methods perform coarse-grained fusion of images and text, lacking a fine-grained collaborative modeling mechanism for rumor detection tasks. They are difficult to simultaneously take into account the complementary effects between image-text matching relationships, implicit clue information, and overall semantic expression, resulting in insufficient adaptability of the model to complex image-text combination scenarios.

[0006] (4) When the text lacks obvious rumor clues, the relationship between the text and the image is complex, or the clues in the sample are relatively sparse, the existing methods are prone to over-reliance on single modal features or surface features, which may lead to missed detections, false detections, or a decrease in the model's generalization ability. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal rumor detection method and system based on implicit cue enhancement, so as to solve at least one of the problems existing in the above-mentioned methods in extracting and utilizing implicit rumor cues.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A multimodal rumor detection method based on implicit cue enhancement includes the following steps:

[0010] S1. Obtain the multimodal data to be detected, wherein the multimodal data includes text information and corresponding image information;

[0011] S2. Extract features from text and image information to obtain text feature representations and image feature representations;

[0012] S3. Based on text feature representation and image feature representation, perform the following parallel feature processing to obtain multiple semantic features, where there is no sequential order among the processing steps:

[0013] S31. Perform cross-modal fusion of text feature representation and image feature representation to obtain modality-shared features;

[0014] S32. Semantically encode the text feature representation and perform multi-label classification to obtain rumor clue features; perform cross-modal fusion of the rumor clue features and image feature representation to obtain clue enhancement features for assisting in rumor identification; wherein, the rumor clue features are information categories that reflect the potential semantic features of the text, including: spam, pornography, advertising, insults, confidentiality, politics or religion, prohibited, terrorism, negative, exaggeration, fabrication, and compliance.

[0015] S33. Perform contextual semantic enhancement processing on the text feature representation, fuse the enhanced text features and image features through a cross-modal interaction mechanism, and perform semantic modeling based on the fusion result to obtain multimodal semantic enhancement features;

[0016] S4. Concatenate or weightedly fuse the cue enhancement features, modality sharing features, and multimodal semantic enhancement features to obtain a comprehensive feature representation;

[0017] Furthermore, in step S2, the process of extracting features from text information includes: converting the text into an embedded vector sequence through a text segmenter to obtain a text feature representation;

[0018] The process of feature extraction from image information includes: segmenting the image information into multiple image blocks of equal size; performing linear projection on each image block to obtain an embedding vector, adding position encoding to each embedding vector, inputting the sequence of embedding vectors with added position encoding into a Transformer encoder, and processing the sequence through a multi-layer self-attention mechanism and a feedforward network to obtain the image feature representation.

[0019] Furthermore, step S31 also includes:

[0020] Based on modal sharing features, the semantic consistency between text and image is calculated. Consistency is used to characterize the degree of matching between text and image. It means that the text content and image content are highly matched in terms of semantics, objects, scenes and logical relationships. The core information expressed by the two is consistent and there is no obvious conflict or ambiguity.

[0021] Furthermore, the process of semantically encoding and multi-label classification of the text feature representation in step S32 includes:

[0022] The text feature representation is processed using a BERT-based cue encoder to obtain the cue feature representation;

[0023] Based on the cue feature representation, a classifier is used to classify the cue features into rumor categories, thus obtaining rumor cue features.

[0024] Furthermore, the process of cross-modal fusion of the rumor clue features and image feature representation in step S32 includes:

[0025] The cue feature representation and the image feature representation are input into the cross-attention layer. The attention mechanism is used to establish a cross-modal semantic association between the rumor cue feature and the image feature, resulting in cue enhancement features.

[0026] A multimodal rumor detection system based on implicit cue enhancement, which is applied to the above-mentioned multimodal rumor detection method, includes: a feature encoding module, a rumor detection module, a feature fusion module, and a classification module;

[0027] The feature encoding module is used to extract features from text information and image information to obtain text feature representation and image feature representation, and then input them into the rumor detection module respectively;

[0028] The rumor detection module consists of three parallel sub-networks: a rumor clue perception network, a graph-text semantic consistency network, and a multimodal semantic enhancement network; wherein:

[0029] The rumor clue perception network is used to extract rumor clue features based on text feature representation and fuse them with image feature representation to obtain clue enhancement features. The rumor clue features include: spam, pornography, advertising, insults, confidentiality, political or religious, prohibited, terrorism, negative, exaggerated, fabricated, and compliant. The image-text semantic consistency network is used to perform cross-modal interaction between text feature representation and image feature representation to obtain modality-shared features. The multimodal semantic enhancement network is used to semantically enhance text feature representation and then fuse it with image feature representation to obtain multimodal semantic enhancement features.

[0030] The feature fusion module is used to fuse the cue enhancement features, modality sharing features, and multimodal semantic enhancement features to obtain a comprehensive feature representation;

[0031] The classification module is used to output rumor detection results based on the comprehensive feature representation.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) This invention proposes a multimodal rumor detection method based on implicit cue enhancement, which not only considers the semantic matching relationship between text and image, but also further mines the implicit rumor cue in the text and combines it with the image content for joint discrimination, thereby improving the ability to identify complex rumors, hidden rumors and rumors with strong disguise.

[0034] (2) The present invention constructs an implicit clue category template, which can extract clue information reflecting potential misleading patterns, content nature and semantic tendencies from multimodal content, and work together with image information to improve the ability to identify rumors that are consistent with the text but distort the facts.

[0035] (3) This invention improves the robustness of the model in complex image and text scenarios, weak clue scenarios and sparse clue scenarios by synergistic integration of image and text semantic consistency modeling, implicit clue perception modeling and multimodal semantic enhancement modeling, which helps to reduce the risk of missed detection in rumor detection.

[0036] (4) This invention can be applied to scenarios such as social platform content review, network information governance, public opinion monitoring, risk warning and network information security protection, providing more stable and accurate technical support for multimodal false information identification. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the multimodal rumor detection method based on implicit cue enhancement according to the present invention;

[0039] Figure 2 This is a schematic diagram of the workflow of the rumor detection module of the present invention;

[0040] Figure 3 This is a schematic diagram of the image feature representation and text feature representation extraction process in an embodiment. Detailed Implementation

[0041] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0042] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0043] Example 1

[0044] Reference Figure 1 This embodiment provides a multimodal rumor detection method based on implicit cue enhancement, which includes the following steps:

[0045] S1. Data Acquisition: Acquiring data containing... A set of multimodal data to be detected; the multimodal data is a collection of image and text data. This includes the corresponding image information 205a and text information 205b, denoted as follows: and ;

[0046] S2. Feature Extraction: The feature encoding module 210 is used to extract features from the image information 205a and the text information 205b respectively, resulting in image feature representation 215a and text feature representation 215b, denoted as follows: and In this embodiment, the feature encoding module 210 includes a text feature extraction module 212 and an image feature extraction module 211, the feature extraction of which is described in detail below. Figure 3 The processing of image information 205a is implemented by image feature extraction module 211, and its implementation process includes:

[0047] The image information 205a is segmented into There are 305(n) image blocks of equal size, denoted as... .

[0048] Each image block 305(n) is transformed into an embedding vector 315a(n) using a linear projection layer 310, denoted as... :

[0049] ;

[0050] in, This represents the vector obtained by flattening image patch 305(n). It is the weight matrix of the linear projection. It is a bias term.

[0051] To preserve the spatial information of the image patch, a positional code 315b(n) is added to the embedding vector 315a(n), denoted as The position-encoded embedding vector 315c(n) is calculated and denoted as . :

[0052] ;

[0053] All embedding vectors 315c(n) constitute the embedding vector sequence 315c, denoted as .

[0054] The embedded vector sequence 315c(n) is input into the image Transformer encoder 320. Contextual modeling of the embedded vector sequence 315c(n) is performed through a multi-layer self-attention mechanism and a feedforward network to establish the dependencies between different locations in the image. The resulting contextual enhancement feature 325(n) is calculated and denoted as... :

[0055] ;

[0056] in, Indicates the feedforward network's first... The response function of the layer, Indicates the first The response function of the self-attention layer. Represents the embedding vector go through The feature values ​​are extracted from the self-attention layer and the feedforward layer.

[0057] All context-enhanced features 325(n) constitute the context-enhanced feature sequence 325, denoted as ,in, The summary representation corresponding to a specific “category” of image patches or the entire sequence is a compressed embedding vector rich in global image features, which will serve as the image feature representation 215a.

[0058] The text feature extraction module 212 performs text information feature extraction as follows:

[0059] The text feature extraction module 212 is a text segmenter. The text segmenter is used to convert the text information 205b into an embedded vector sequence 215b, which is the text feature representation.

[0060] S3, see reference Figure 2 The image features 215a and text features 215b are respectively input into the rumor clue perception network 220, the rumor image-text semantic consistency sub-network 230, and the multimodal semantic enhancement network 240 for collaborative fusion, specifically including:

[0061] S31, the rumor cue perception network 220, processes image and text features as follows: text features 215b are input into a cue encoder 221 based on the BERT model, and the cue feature representation 225a is calculated, denoted as... :

[0062] ;

[0063] in, This represents the response function of the BERT model. Text feature representation 215b.

[0064] Clue feature representation 225a is input into clue classifier 222 to calculate rumor clue feature 225b, denoted as :

[0065] ;

[0066] in, It is the weight matrix of the linear layer. This is a bias term. Rumor clue categories include: spam, pornography, advertising, abusive language, confidentiality, political or religious, prohibited, terrorism, negative, exaggerated, fabricated, and compliant, totaling 12 labels. Simultaneously, clue feature representation 225a and image feature 215a are input into a cross-attention layer, utilizing an attention mechanism to enhance the interaction between rumor clue features and image features, capturing the correlation and complementarity between the two types of information. The fused feature representation 225c is calculated, denoted as... :

[0067] ;

[0068] in, This represents the response function of the cross-attention layer. Image feature representation 215a.

[0069] S32, the rumor image-text semantic consistency subnetwork 230, its process for processing image-text features is as follows: text features 215b are input into a self-attention layer 231 to capture long-distance dependencies within the text sequence, and the attention layer output 235a is calculated, denoted as... :

[0070] ;

[0071] in, Image feature representation 215a, Text feature representation 215b, , , These are the query mapping matrix, key mapping matrix, and value mapping matrix of the self-attention layer. It is the dimension of the key vector in the self-attention layer, and the softmax function is defined as:

[0072] ;

[0073] The output 235a of the self-attention layer and the image feature 215a are input into a cross-attention layer to calculate the rumor modality cross feature 235b, denoted as... :

[0074] ;

[0075] in, , , These are the query mapping matrix, key mapping matrix, and value mapping matrix of the cross-attention layer. It is the dimension of the key vector in the cross-attention layer.

[0076] Cross feature 235b is input into feedforward network 232 to calculate rumor modality shared feature 235c, denoted as :

[0077] ;

[0078] in, and It is the weight matrix of the linear projection. and It is a bias term, and the definition of the max function is:

[0079] ;

[0080] The rumor modality sharing feature 235c is input into the image-text consistency evaluation module 233 to calculate the consistency score 235d of the image-text pair, denoted as... :

[0081] ;

[0082] in, This is the weight matrix of the image-text consistency evaluation module 233. It is a bias term.

[0083] The consistency score of the image-text pair 235d was processed by SoftMax to obtain the consistency judgment result, which includes 0 and 1, where 0 indicates that the image and text match and 1 indicates that the image and text do not match.

[0084] S33, the rumor multimodal semantic enhancement subnetwork 240, its process for processing image and text features is as follows: text features 215b are input into the embedding layer for preprocessing, converting words or characters in the text into vector representations 245a, denoted as... :

[0085] ;

[0086] To enhance the model's understanding of context and increase its non-linear expressive power, 245a is input into the self-attention layer 241 and the feedforward network, and the context-enhanced feature 245b output by the self-attention layer is calculated, denoted as... :

[0087] ;

[0088] The output 245b of the self-attention layer and image feature 215a are combined and cross-attention processing is performed to achieve deep semantic fusion between text and image, resulting in cross-feature 245c, denoted as... :

[0089] ;

[0090] The cross-feature 245c is passed through the feedforward network 242 to calculate the score 245d for each word, denoted as... :

[0091] ;

[0092] in, This represents the response function of the linear layer of the feedforward network. Representation layer normalization, Represents a non-linear activation function. This represents the feature projection function. The vocabulary score of 245d is used for prediction in subsequent masked language modeling tasks.

[0093] S4. The output features of the rumor clue perception network 220, the rumor image-text semantic consistency sub-network 230, and the multimodal semantic enhancement network 240 are fused. Specifically, the fused feature representation 225c in the rumor clue perception sub-network 220 is concatenated with the rumor modality shared feature 235c in the rumor image-text semantic consistency sub-network 230 to form a comprehensive feature vector 235e. Since the rumor modality shared feature 235c represents the image-text matching feature, and the fused feature representation 225c contains the matching information between the image and the rumor clue, the multimodal rumor detection model 200 based on implicit clue enhancement will be influenced by both image information and text information when making a rumor judgment, thus obtaining a comprehensive rumor prediction result.

[0094] S5. Classify rumors based on fused features and output the detection results. The specific process is as follows: The comprehensive feature vector 235e is input into the rumor classification module 234, and the rumor probability score 235f is calculated, denoted as... :

[0095] ;

[0096] The probability score of the predicted rumor is 235f. After processing by SoftMax, the rumor judgment result is obtained, which includes 0 and 1, where 0 represents a rumor and 1 represents a non-rumor.

[0097] Example 2:

[0098] This embodiment provides a multimodal rumor detection system based on implicit cue enhancement. The system is applied to the aforementioned multimodal rumor detection method and includes a feature encoding module, a rumor detection module, a feature fusion module, and a classification module.

[0099] The feature encoding module is used to extract features from text information and image information to obtain text feature representation and image feature representation, and then input them into the rumor detection module respectively;

[0100] The rumor detection module consists of three parallel sub-networks: a rumor clue perception network, a graph-text semantic consistency network, and a multimodal semantic enhancement network; wherein:

[0101] The rumor clue perception network is used to extract rumor clue features based on text feature representation and fuse them with image feature representation to obtain clue enhancement features. The rumor clue features include: spam, pornography, advertising, insults, confidentiality, political or religious, prohibited, terrorism, negative, exaggerated, fabricated, and compliant. The image-text semantic consistency network is used to perform cross-modal interaction between text feature representation and image feature representation to obtain modality-shared features. The multimodal semantic enhancement network is used to semantically enhance text feature representation and then fuse it with image feature representation to obtain multimodal semantic enhancement features.

[0102] The feature fusion module is used to fuse the cue enhancement features, modality sharing features, and multimodal semantic enhancement features to obtain a comprehensive feature representation;

[0103] The classification module is used to output rumor detection results based on the comprehensive feature representation.

[0104] The multimodal rumor detection system is trained using the following method:

[0105] S1. Dataset Collection: This embodiment collected three datasets from real social network environments for experiments, specifically including: (1) MediaEval dataset. This dataset consists of approximately 17,000 Twitter tweets covering different events. Each tweet contains text content, images or videos, and other social media information. The dataset is divided into a development set and a test set. The training set contains approximately 9,000 rumor tweets and 6,000 non-rumor tweets from 17 rumor-related events; the test set contains approximately 2,000 tweets from another batch of 35 rumor-related events. To ensure that the tweets in the two sets cover different events, there are no identical rumor events between the tweets in the development set and the test set; (2) CHECKED dataset. This dataset is a Chinese dataset targeting COVID-19 rumors, containing 2,104 verified Weibo posts related to COVID-19 identified through keyword search from December 2019 to August 2020; (3) Weibo-2017 dataset. This dataset includes fake news verified by Weibo's official community management center and real news from authoritative Chinese news sources such as Xinhua News Agency, covering the period from May 2012 to January 2016.

[0106] S2. Rumor Clue Preprocessing. The original three datasets only contain rumor labels. For complex multimodal rumor detection tasks, it is necessary to process data with finer granularity to improve the performance of the multimodal rumor detection model. In this embodiment, two additional data labels are added to the three datasets, specifically including: (1) Image-text matching labels, which are used to measure the semantic consistency between images and text. In this embodiment, CLIP method is used to generate them. (2) Rumor clue labels. In this embodiment, 12 types of rumor clue labels are constructed by using Baidu Content Review Platform, Alibaba Cloud Content Security Platform, Tencent Cloud Text Content Security Platform and keyword matching. In this embodiment, in order to improve the accuracy of clue labeling, a weighted majority voting algorithm is used to fuse the labeling results from the three review platforms, reduce the impact of labeling errors from a single platform, and improve the consistency and reliability of the final rumor clue labeling.

[0107] S3. Model Training and Optimization Strategy. In this embodiment, the training process focuses on coordinating information from text and images to accurately identify and classify multimodal rumor content. To this end, this embodiment constructs three key sub-networks: a rumor image-text semantic consistency sub-network, a rumor cue perception sub-network, and a rumor multimodal semantic enhancement sub-network. Each sub-network is optimized for a specific model task. The specific loss calculation is as follows:

[0108] S31. Calculate the loss of the rumor image-text semantic consistency sub-network. The rumor image-text semantic consistency sub-network focuses on evaluating the consistency between text and images, and the corresponding loss function is... Aimed at minimizing consistency errors between text and images:

[0109] ;

[0110] in, It is the first The true labels for each sample are 0 or 1, representing whether the image and text match. The probability of image-text matching predicted by the subnetwork.

[0111] S32. Calculate the loss of the rumor clue perception subnetwork. The rumor clue perception subnetwork identifies potential rumor clues by analyzing text content, and the corresponding loss function is... Aimed at improving the accuracy of rumor clue classification:

[0112] ;

[0113] in, This refers to the number of rumor categories, which is 12 in this example. It is the first Does the sample belong to the ? The tags for rumor-related clues range from 0 to 11. It is the first sub-network prediction Does the sample belong to the ? The probability of being a rumor-like clue.

[0114] S33. Calculate the loss of the rumor multimodal semantic enhancement sub-network. The rumor multimodal semantic enhancement sub-network further processes the features of text and images, and the corresponding loss function is... Committed to improving the semantic expressive power of multimodal features:

[0115] ;

[0116] in, It is the sample size. It represents the number of words that are masked in each sample. It is the first In the nth sample A masked word, It is the first The context of each sample It is the model's prediction of the first In the nth sample The probability of a word being masked.

[0117] S34. Calculate the loss of the rumor prediction layer. The rumor prediction layer predicts whether the multimodal information of the image-text-clue is a rumor, and the corresponding loss is calculated. :

[0118] ;

[0119] in, It is the first The true label for each sample is either 0 or 1, indicating whether the image or text is a rumor. This is the probability predicted by the rumor prediction layer that this image and text is a rumor.

[0120] S35. Calculate the total loss. Each subnetwork is optimized using its specific loss function, and finally, a total loss function is designed. This will allow for joint optimization of the entire multimodal rumor detection model. The total loss function... It is a weighted sum of four loss functions:

[0121] ;

[0122] in, , , and These are hyperparameters, representing the weights of the loss function in the rumor image-text semantic consistency subnetwork, the rumor clue perception subnetwork, the rumor multimodal semantic enhancement subnetwork, and the rumor prediction layer, respectively. Therefore, the prediction model based on the implicit clue enhancement multimodal rumor detection method can not only understand and analyze multimodal content from different perspectives, but also achieve accurate detection of rumors by comprehensively considering the consistency between text and images, the salience of rumor clues, and the enhanced information of multimodal semantics.

[0123] S4. Parameter optimization. Based on the total loss... Calculate gradient Update parameters ,in The learning rate is used. During training, the total loss function is iteratively optimized. This improves the model's performance on multimodal rumor detection tasks.

[0124] In this embodiment, Deit-base is used as the Vit module in the image part of data processing. This model utilizes... Pixel image block processing The input image consists of pixels. For text, the rumor text encoder and rumor clue encoder use BERT-base or its variants, with a hidden layer size of 768. For the cross-attention layer fusing data from different modalities, 12 attention heads are used to capture different contextual relationships. This embodiment achieves a recall of 0.831, precision of 0.772, and F1 scores of 0.789 for rumor samples and 0.752 for non-rumor samples on the MediaEval dataset. The proposed method exhibits high recall and F1 score, indicating that it can identify as many rumors as possible, avoiding false negatives (i.e., "missed" rumors), and its overall performance is relatively balanced. Furthermore, this embodiment achieves precision, accuracy, recall, and F1 score all above 0.97 on the CHECKED dataset. Additionally, this embodiment achieves precision of 0.827 on the Weibo-2017 dataset. The proposed method demonstrates greater stability in complex multimodal rumor detection tasks. A high recall rate indicates that as many rumors as possible can be identified, avoiding false negatives, i.e., "missed" rumors. This embodiment demonstrates the superior performance of the method proposed in this invention in detecting rumors.

[0125] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal rumor detection method based on implicit cue enhancement, characterized in that, The rumor detection method includes: S1. Obtain the multimodal data to be detected, wherein the multimodal data includes text information and corresponding image information; S2. Extract features from text and image information to obtain text feature representations and image feature representations; S3. Based on text feature representation and image feature representation, perform the following parallel feature processing to obtain multiple semantic features, where there is no sequential order among the processing steps: S31. Perform cross-modal fusion of text feature representation and image feature representation to obtain modality-shared features; S32. Semantically encode the text feature representation and perform multi-label classification to obtain rumor clue features; perform cross-modal fusion of the rumor clue features and image feature representation to obtain clue enhancement features for assisting in rumor identification; wherein, the rumor clue features are information categories that reflect the potential semantic features of the text, including: spam, pornography, advertising, insults, confidentiality, politics or religion, prohibited, terrorism, negative, exaggeration, fabrication, and compliance. S33. Perform contextual semantic enhancement processing on the text feature representation, fuse the enhanced text features and image features through a cross-modal interaction mechanism, and perform semantic modeling based on the fusion result to obtain multimodal semantic enhancement features; S4. Concatenate or weightedly fuse the cue enhancement features, modality sharing features, and multimodal semantic enhancement features to obtain a comprehensive feature representation.

2. The method according to claim 1, characterized in that, In step S2, the process of extracting features from text information includes: converting the text into an embedded vector sequence through a text segmenter to obtain a text feature representation; The process of feature extraction from image information includes: segmenting the image information into multiple image blocks of equal size; performing linear projection on each image block to obtain an embedding vector, adding position encoding to each embedding vector, inputting the sequence of embedding vectors with added position encoding into a Transformer encoder, and processing the sequence through a multi-layer self-attention mechanism and a feedforward network to obtain the image feature representation.

3. The method according to claim 1, characterized in that, Step S31 further includes: The semantic consistency between text and image is calculated based on modality-shared features, which is used to characterize the degree of matching between text and image.

4. The rumor detection method according to claim 1, characterized in that, The process of semantically encoding and multi-label classification of the text feature representation in step S32 includes: The text feature representation is processed using a BERT-based cue encoder to obtain the cue feature representation; Based on the cue feature representation, a classifier is used to classify the cue features into rumor categories, thus obtaining rumor cue features.

5. The rumor detection method according to claim 1, characterized in that, The process of cross-modal fusion of the rumor clue features and image feature representation in step S32 includes: The cue feature representation and the image feature representation are input into the cross-attention layer. The attention mechanism is used to establish a cross-modal semantic association between the rumor cue feature and the image feature, resulting in cue enhancement features.

6. A multimodal rumor detection system based on implicit cue enhancement, characterized in that, The system is applied to the multimodal rumor detection method according to any one of claims 1 to 5, comprising: a feature encoding module, a rumor detection module, a feature fusion module, and a classification module; The feature encoding module is used to extract features from text information and image information to obtain text feature representation and image feature representation, and then input them into the rumor detection module respectively; The rumor detection module consists of three parallel sub-networks: a rumor clue perception network, a graph-text semantic consistency network, and a multimodal semantic enhancement network; wherein: The rumor clue perception network is used to extract rumor clue features based on text feature representation and fuse them with image feature representation to obtain clue enhancement features. The rumor clue features include: spam, pornography, advertising, insults, confidentiality, political or religious, prohibited, terrorism, negative, exaggerated, fabricated, and compliant. The image-text semantic consistency network is used to perform cross-modal interaction between text feature representation and image feature representation to obtain modality-shared features. The multimodal semantic enhancement network is used to semantically enhance text feature representation and then fuse it with image feature representation to obtain multimodal semantic enhancement features. The feature fusion module is used to fuse the cue enhancement features, modality sharing features, and multimodal semantic enhancement features to obtain a comprehensive feature representation; The classification module is used to output rumor detection results based on the comprehensive feature representation.

Images