An adaptive multi-modal fake news detection method and model based on double features

Abstract

The application relates to the fields of network content security, network space security technology and the like, and discloses a self-adaptive multi-modal false message detection method and model based on double features, which is realized by using a self-adaptive multi-modal false message detection model based on double features and comprises the following steps: an adaptive tweet processing module is used to obtain natural language content, visual content and marking information of a tweet; a data enhancement module is used to obtain the natural language content of the tweet and input the natural language content into a large language model of the data enhancement module to rewrite the text; a multi-modal feature construction module is used to obtain multi-modal mode features and fact features; and a false message detection module is used to combine a multi-expert network containing a gate network to aggregate tweet features and realize classification of false messages, so that the false information detection is accurately performed, and the accuracy of the existing false information detection method in a real scene is improved.

Description

Technical Field

[0001] The present invention relates to the fields of network content security, cyberspace security technology, and more specifically, to an adaptive multimodal false message detection method and model based on dual features. Background Art

[0002] In the mobile internet era, the speed and scale of information dissemination far surpass those of traditional media. Platforms like Weibo and TikTok have become primary channels for information dissemination. While citizens gain rapid access to information, they are often exposed to false news. False news, as misleading information, rapidly spreads and proliferates on online social platforms, leveraging their low barriers to entry, rapid dissemination, and wide reach. This can have serious consequences for individuals, organizations, and even society as a whole. To mitigate the potential harm caused by false news, false news detection technology is urgently needed, with broad application prospects in social platform management, cyberspace public opinion governance, and other areas.

[0003] Traditionally, identifying false information relies primarily on analyzing the text within tweets. This approach focuses on mining the statistical properties and deep semantics of text. For example, the authenticity of tweets is determined by calculating word frequency, detecting patterns in the use of specific symbols, and extracting semantics using deep neural networks. However, with the increasing abundance of visual content such as images and videos on social networks, relying solely on text analysis has become insufficient to cope with the increasing complexity of false information. Today, multimodal approaches that fuse visual content such as images and videos with textual information have become a major trend in false information detection on social platforms. Due to its direct visibility and high appeal, visual content can often more intuitively reveal the nature of false information. Combined with text analysis techniques, this approach effectively improves the accuracy of false information identification. These approaches not only consider the textual content of tweets but also delve into the underlying meaning of their visual expressions, fully leveraging the multimodal nature of tweets and providing a comprehensive solution for detecting the complexities of false information on social networks. However, existing multimodal false information detection methods for social platforms still face the following challenges:

[0004] 1. Existing multimodal methods fail to effectively capture information representative of tweet content. During model training or testing, tweets are assumed to consist solely of text and images. This assumption differs significantly from tweets shared on Weibo. Tweets come in a variety of formats, including pure text, pure images, videos, image-text pairs, and text with multiple images. Focusing solely on image-text pairs in model development while ignoring other tweet formats will result in detection models that are ineffective in real-world applications.

[0005] 2. Current methods primarily focus on uncovering the complex relationships between deep semantic features of images and text in disinformation, but often overlook the importance of shallower features embedded in disinformation, such as syntax, vocabulary, and surface visual semantics. Improving disinformation detection through pre-trained models has led to a widespread trend of representing tweets using the final output features of frozen text and visual models. However, these methods often overlook shallow outputs, resulting in inaccurate representations of multimodal tweets and poor cross-modal fusion.

[0006] 3. Existing methods ignore the importance of factual details in tweets, such as time reference, location, and personal identity, which are key indicators of tweet authenticity. Relying solely on pre-trained text and image models for feature extraction may not be able to effectively capture these factual elements. Although research has used named entity recognition (NER) to extract factual entities from text, there is still a gap in addressing the factual features present in images. Some methods focus on extracting visual entities from images to improve the search for multimodal clues. However, extracting entities only from images may lead to the loss of the relational context between entities, thereby hindering the comprehensive representation of factual features in the image and affecting the detection performance of the model.

[0007] 4. With the rapid development of large language models, fake news generated by these models has emerged. These tweets are diverse in style and more deceptive. However, existing multimodal fake news datasets are based on manually generated tweets collected from social platforms and lack tweets generated by large language models. Therefore, existing methods have difficulty capturing the unique characteristics of fake news generated by large language models, resulting in biased detection results. Summary of the Invention

[0008] The present invention aims to address the shortcomings of the existing technology by providing an adaptive multimodal false information detection method and model based on dual features. This method addresses the problems of multimodal false information detection methods used in social media environments, including the inability to handle the diverse organizational structure of tweets, inaccurate representation of multimodal tweets, neglect of factual features in multimodal false information, and the lack of false information generated by large language models in the training dataset. Furthermore, this method aims to improve the accuracy of existing false information detection methods in real-world scenarios.

[0009] The present invention is implemented through the following technical solution: an adaptive multimodal false news detection method based on dual features, which is implemented using an adaptive tweet processing module, a data enhancement module, a multimodal feature construction module, and a false news detection module. The method comprises the following steps:

[0010] 1) The adaptive tweet processing module collects tweets from three types of online platforms: authoritative platforms, rumor-debunking websites, and other websites, and forms a training data set with tweets that have been marked for authenticity, and a test data set with tweets that have not been marked for authenticity. The collected tweets are then processed to obtain the natural language content, visual content, and tag information of the tweets. That is, in the adaptive tweet processing module, the present invention first extracts the natural language content and visual content in the tweets; secondly, an adaptive algorithm is used to remove redundancy and noise in the natural language content and visual content; finally, the modal type of the tweet (only natural language content, only visual content, and both natural language and visual content) is marked to obtain the natural language content, visual content, and tag information of the tweets.

[0011] 2) The data enhancement module performs data enhancement on the training dataset: the natural language content of the tweets in the training dataset is obtained, and the natural language content of the tweets is input into the large language model of the data enhancement module to rewrite the text. The new tweets are composed of the visual content and tag information of the tweets to expand the training dataset. That is, in the data enhancement module, the present invention first obtains the natural language content of the tweets in the training dataset, inputs it into the large language model of the data enhancement module, rewrites the text, and changes the text style while maintaining content consistency. To reflect reality, the data enhancement module uses two different prompts for true and false tweets.

[0012] 3) The multimodal feature construction module extracts multimodal pattern features in the natural language content and visual content, as well as fact features in the natural language content and visual content through a multimodal dual feature extraction method; that is, in the multimodal feature construction module, the present invention proposes a multimodal dual feature extraction method, which includes two parts: pattern feature extraction and fact feature extraction, to extract multimodal pattern features and fact features in the natural language content and visual content in the original tweet.

[0013] 4) The fake news detection module combines a hybrid expert network with a gating network to aggregate tweet features and classify fake news. Specifically, within the fake news verification module, the present invention proposes a multi-expert network fake news detection method, which includes a hybrid expert network with a gating network. The adaptive tweet processing module obtains tweet tag information, and the gating network can flexibly process all types of tweets, ultimately achieving the goal of classifying tweets into true and false messages. A classifier is added after the hybrid expert network to verify the authenticity of tweets. A global loss is set to the classifier's loss, so that constraints are propagated back during the gradient backpropagation phase to coordinate the representations of each expert.

[0014] To further better implement the adaptive multimodal false news detection method based on dual features described in the present invention, the present invention proposes an adaptive false news processing method. The organization forms of tweets on social platforms are different. This method adaptively extracts natural language content and visual content based on the different existing organizational forms, so that subsequent modules can adopt different detection strategies according to the organizational type of tweets, effectively improving the accuracy of the method in modality-missing scenarios. At the same time, the semantic similarity algorithm is used to remove noise content, maintain the integrity of the image semantics, and thus improve the detection effect. In particular, the following setting method is adopted: the step 1) includes the following steps:

[0015] 1.1) The adaptive tweet processing module collects tweets from online platforms, labels tweets published by authoritative platforms as true, and tweets published by rumor-debunking websites as false. This labeled true and false tweets serves as the training dataset, while tweets published by other websites serve as the test dataset to be tested.

[0016] 1.2) For images with natural language information in tweets, the natural language information in the images will be extracted and combined with the text in the tweet to serve as the natural language content of the tweet. The extracted images and pure images in the tweets will serve as the visual content of the tweet.

[0017] 1.3) Using an adaptive algorithm to calculate semantic similarity between the acquired natural language information and visual content, filtering out redundancy and noise in the natural language and visual content;

[0018] 1.4) After step 1.3), the filtered tweets are categorized and labeled as containing only natural language content, only visual content, or both natural language and visual content;

[0019] 1.5) Save the natural language content, visual content, and tag information of tweets.

[0020] To further enhance the dual-feature-based adaptive multimodal false news detection method described in the present invention, a false news data augmentation method based on a large language model is proposed. This method utilizes the large language model to transform the style of tweets, enriching the content of a single training set and improving the model's ability to detect tweets of different styles. Specifically, the following configuration is employed: Step 2) includes the following steps:

[0021] 2.1) During the training phase of the dual-feature-based adaptive multimodal fake news detection model, the natural language content of tweets marked as real in the training dataset is obtained. The style transfer hint and the natural language content of the tweet are input into the large language model to obtain the natural language content rewritten with style transfer. The visual content and tag information of the tweet are combined with the natural language content rewritten with style transfer to form a new tweet, which is added to the training dataset.

[0022] 2.2) During the training phase of the dual-feature-based adaptive multimodal false news detection model, for each tweet marked as false in the training dataset, the natural language content of the tweet is obtained. The style formalization hints and the natural language content of the tweet are input into the large language model to obtain the natural language content after the style formalization. The visual content and tag information of the tweet are combined with the natural language content rewritten by style transfer to form a new tweet, which is added to the training dataset.

[0023] In order to further better realize the adaptive multimodal false news detection method based on dual features described in the present invention, the present invention proposes a dual false news feature extraction method to extract multimodal pattern features and multimodal fact features in false messages. For multimodal pattern features, the present invention integrates unimodal representations at different levels. For multimodal fact features, the present invention first uses a large language model to generate a text summary, then uses the text summary to guide the visual language model to generate an accurate picture description, and finally integrates the text summary and the picture description as a multimodal fact feature. The method effectively extracts multimodal false news features from the pattern and fact levels, further improving the prediction performance and generalization ability. In particular, the following setting method is adopted: the step 3) includes the following steps:

[0024] 3.1) Constructing a pattern feature extraction network: This paper proposes a pattern fusion method for multimodal pattern feature extraction. For natural language content, the BERT model is used to obtain the output of each layer; for visual content, the ViT model is used to obtain the output of each layer;

[0025] Then, for tweets with both natural language and visual content, the present invention uses a learnable bridge layer to connect the various levels of output of natural language content and the various levels of output of visual content to obtain fused multimodal pattern features;

[0026] For tweets with only natural language content or only visual content, the present invention does not fuse features and directly uses the output of each layer as the pattern feature.

[0027] 3.2) Fact Extraction:

[0028] For tweets with both natural language and visual content, the multimodal feature construction module first uses the large language model to obtain a text summary of the natural language content. It then uses the similarity module of the large visual language model (BLIP) to calculate the similarity between the text summary and the visual content. GradCAM is used to locate image regions related to the visual content and the text summary. Finally, the large visual language model (BLIP) is used to generate image descriptions of these image regions.

[0029] For tweets with labeled information containing only natural language content, the multimodal feature construction module only extracts text summaries;

[0030] For tweets with only visual content, the multimodal feature construction module directly uses the large visual language model BLIP to obtain image descriptions.

[0031] 3.3) Constructing fact feature extraction network:

[0032] This paper proposes a fact fusion method for extracting multimodal fact features. For text summaries obtained from tweets with both natural language and visual content, the multimodal feature construction module uses the BERT model to obtain outputs at each level. For image descriptions obtained from tweets with both natural language and visual content, the multimodal feature construction module also uses the BERT model to obtain outputs at each level. This paper uses a learnable bridge layer to connect the outputs of the text summaries and image descriptions at each level to obtain fused multimodal fact features.

[0033] For tweets with only natural language content, the dual-feature-based adaptive multimodal false news detection model directly uses the output of each level of the text summary as factual features.

[0034] For tweets with labeled information containing only visual content, the dual-feature-based adaptive multimodal false news detection model directly uses the output of each level of image description as factual features.

[0035] To further enhance the dual-feature-based adaptive multimodal false news detection method described in the present invention, a multi-expert network false news detection method is proposed. This method utilizes a gated network to guide the multi-expert network to learn corresponding representations for different types of false news content, thereby improving the model's ability to detect false news from different organizational types. Specifically, the following configuration is employed: Step 4) includes the following specific steps:

[0036] 4.1) Constructing a hybrid expert network: First, pattern features and fact features are concatenated to obtain multimodal joint features. The multimodal joint features are then input into multiple expert networks composed of Bi-LSTM networks to obtain multiple sequence fusion features. Simultaneously, the multimodal joint features are input into a gating network, which first performs average pooling on the multimodal joint features and concatenates them with a vector constructed from the tweet's tag information. After passing through a fully connected layer, the output is normalized using the softmax function to finally obtain a weight ratio vector output by each Bi-LSTM network. The multiple sequence fusion features and the weight ratios are weightedly added to obtain the final tweet representation. Finally, a fully connected layer is added as a classifier to complete the construction of the hybrid expert network.

[0037] 4.2) Model Training and Testing: If the model is currently in the training phase, the output of the fully connected layer of the classifier is first processed by the sigmoid activation function to obtain a prediction result, and the BCE loss is calculated on the prediction result. If the model is currently in the inference phase, the prediction result will be used as the inference result of the adaptive multimodal fake news detection model based on dual features.

[0038] An adaptive multimodal false news detection model based on dual features, including

[0039] The adaptive tweet processing module collects tweets from three types of online platforms: authoritative platforms, rumor-debunking websites, and other websites. Tweets posted by authoritative platforms are marked as true, and tweets posted by rumor-debunking websites are marked as false. The tweets marked as true and false serve as the training dataset, and tweets posted by other websites serve as the test dataset to be tested. The module also extracts information from different carriers of tweets to obtain natural language content and visual content. Using a similarity-based adaptive algorithm, the content is denoised and classified, resulting in three types of tweets: those containing only natural language content, those containing only visual content, and those containing both natural language and visual content.

[0040] The data augmentation module takes the natural language content of tweets in the training dataset as raw input, uses a large language model to rewrite and reconstruct the text using two types of real and fake tweet prompts, and changes the text style while maintaining content consistency. The new tweets after data augmentation are then added to the training dataset;

[0041] The multimodal feature construction module includes a pattern feature module and a fact feature module, which are used to obtain tweets and extract two types of features: pattern and fact;

[0042] The fake news detection module adopts a fake news detection method based on a hybrid expert network that includes a gating network to aggregate tweet features to achieve fake news classification.

[0043] In order to further better realize the adaptive multimodal false news detection model based on dual features described in the present invention, the following setting method is particularly adopted: the pattern feature module extracts the features of each level of single modality through a pre-trained model, and uses a weighted full cross-modal bridging layer (learnable bridging layer) to fuse the single modalities of each level to obtain pattern features; the fact feature module first uses a large graphic and text model to extract the fact content in the natural language content to obtain a fact content summary, and then uses the fact content summary to guide the large graphic and text model to generate a visual content description, and finally uses the pattern feature method to obtain the fact feature.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] (1) Existing multimodal false information detection methods are unable to handle the diverse organizational structures of tweets and are therefore not well suited for application in social media environments. To address these shortcomings, the present invention proposes an adaptive tweet processing method. The proposed method involves extracting natural language and visual content from tweets, thereby standardizing various tweet formats into a unified structure. Subsequently, a similarity-based algorithm is used to address issues related to noisy content and situations where natural language or visual content may be missing, thereby improving the overall effectiveness of the detection model. In addition, the present invention also proposes a multi-expert network that helps the expert network learn effective features through labels, thereby improving the detection rate of various types of false information.

[0046] (2) Considering that current false information detection methods prioritize deep semantic features over shallow features, resulting in inaccurate representation of multimodal tweets, the present invention designs a weighted bridging layer (i.e., a learnable bridging layer, also a fully cross-modal bridging layer with weights) based on a joint attention mechanism to fuse different levels of features extracted by pre-trained image and text models. By introducing multiple learnable bridging layers between the text and visual models, different levels of the unimodal model can be connected to the cross-modal module. This achieves effective cross-modal alignment and fusion between different levels of visual and text representations. Combined with learnable weights, the module improves the modal representation of multimodal false information and enhances the false message detection capability.

[0047] (3) To address the problem that current methods ignore factual features in multimodal false information, this paper proposes a zero-shot multimodal factual feature extraction method (a fact fusion method) based on a large-scale visual language model. First, the factual content in the text is extracted using the understanding ability of the large-scale visual language model to obtain a text summary; then, the text summary is used as a prompt word to guide the large-scale visual language model to generate a title for the key part of the image; finally, the text summary and the image title are combined to obtain factual features, and the factual features are used to enhance the false information detection capability.

[0048] (4) To address the problem that current datasets lack large-scale language models to generate fake information, which leads to reduced detection accuracy, this paper proposes a multimodal fake information data augmentation scheme based on large language models to obtain a dataset similar to real-world scenarios. This scheme uses large language models to imitate the writing style of real tweets and integrates image descriptions to make fake tweets more complex.

[0049] (5) The method described in the present invention divides the false news detection task into four modules: an adaptive tweet processing task, a data enhancement task, a multimodal feature construction task, and a false news detection classification task. The method adaptively processes tweets appearing on social platforms, avoiding the impact of modal missing or noise content on false news detection. At the same time, it will enhance the detection ability of the method for messages with diverse styles through data enhancement, and accurately capture the patterns and factual characteristics of false news, thereby achieving accurate false news detection. The model divides the false news detection task into four modules: an adaptive tweet processing module, a data enhancement module, a multimodal feature construction module, and a false news detection module. The model adaptively processes tweets appearing on social platforms, avoiding the impact of modal missing or noise content on false news detection. At the same time, it will enhance the detection ability of the method for messages with diverse styles through data enhancement, and accurately capture the patterns and factual characteristics of false news, thereby achieving accurate false news detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The overall flow chart for fake news detection on social networks.

[0051] Figure 2 Processing flow chart for building modules for multimodal features.

[0052] Figure 3 Diagram of the modality fusion structure in the multimodal feature building module.

[0053] Figure 4 Flowchart of fact extraction in the multimodal feature building module.

[0054] Figure 5 Diagram of the fact fusion structure in the multimodal feature building module.

[0055] Figure 6 This is the structural diagram of the false information detection module.

[0056] In the figure, Bridge is the bridging layer; Query is the query matrix in the attention mechanism; Key is the key matrix in the attention mechanism; Value is the value matrix in the attention mechanism; Gate is the gated network; Expert is the expert network; MSA is the multi-head self-attention layer; FNN is the feedforward network layer. DETAILED DESCRIPTION

[0057] The present invention will be further described in detail below with reference to the examples, but the embodiments of the present invention are not limited thereto.

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0060] Glossary:

[0061] BERT model: Bidirectional Encoder Representations from Transformers (BERT), a pre-trained model for natural language processing (NLP)

[0062] ViT model: Visual Transformer (ViT), a pre-training model for the visual field

[0063] BLIP: Bootstrapping language-image pre-training, a technology used for training large text vision models.

[0064] GradCAM: Grad-CAM (Gradient-weighted Class Activation Mapping) is a method for explaining predictions of deep learning models.

[0065] Bi-LSTM network: Bidirectional Long Short-Term Memory network (Bi-LSTM) is a deep learning model and a variant of recurrent neural network (RNN).

[0066] BCE: BCE loss function is the abbreviation of Binary Cross-Entropy loss function. In binary classification problems, BCE loss function is usually used to measure the difference between model output and actual label.

[0067] OCR: OCR stands for Optical Character Recognition. OCR is a technology that converts printed or handwritten text (such as books, documents, and images) into editable text data. OCR technology recognizes characters, numbers, and symbols in images and converts them into a computer-readable text format.

[0068] AdamW: Adam (Adaptive Moment Estimation) is a gradient descent optimization algorithm. AdamW is a variant of the Adam optimization algorithm. It improves the problem of learning rate attenuation in the Adam algorithm.

[0069] Example 1:

[0070] The present invention designs an adaptive multimodal false news detection model based on dual features, combining Figure 1 Shown, including

[0071] The adaptive tweet processing module (i.e., adaptive message processing module) collects tweets from three types of online platforms: authoritative platforms, rumor-debunking websites, and other websites. Tweets posted by authoritative platforms are marked as true, and tweets posted by rumor-debunking websites are marked as false. The tweets marked as true and false serve as the training dataset, and tweets posted by other websites serve as the test dataset to be tested. The module then extracts information from the different carriers of the tweets to obtain natural language content and visual content. Using a similarity-based adaptive algorithm, the module denoises and classifies the content, resulting in three types of tweets: those containing only natural language content, those containing only visual content, and those containing both natural language and visual content.

[0072] The data augmentation module augments the tweets in the database to make them more consistent with real-world scenarios. The module uses the natural language content of tweets in the training dataset as raw input, and uses a large language model to rewrite and reconstruct the text using two types of real and fake tweets. This method alters the text style while maintaining content consistency, and then adds the augmented tweets to the training dataset.

[0073] The multimodal feature construction module is equipped with a pattern feature module and a fact feature module, which are used to obtain tweets and extract two types of features: pattern and fact. The pattern feature module extracts the features of each level of unimodality through a pre-trained model and fuses the unimodal levels at each level using a weighted full cross-modal bridging layer (a learnable bridging layer) to obtain pattern features. The fact feature module first uses the large image and text model to extract the fact content from the natural language content to obtain a fact content summary, then uses the fact content summary to guide the large image and text model to generate visual content descriptions, and finally uses the pattern feature method to obtain fact features.

[0074] The fake news detection module adopts a fake news detection method based on a hybrid expert network that includes a multi-expert network and a gated network to aggregate tweet features to achieve fake news classification (distinguishing between real news and fake news).

[0075] Example 2:

[0076] An adaptive multimodal false news detection method based on dual features, combined with Figures 1 to 6 As shown in FIG, an adaptive multimodal false news detection model based on dual features is implemented, which is composed of an adaptive tweet processing module, a data enhancement module, a multimodal feature construction module, and a false news detection module, and includes the following steps:

[0077] 1) The adaptive tweet processing module collects tweets from three types of online platforms: authoritative platforms, rumor-debunking websites, and other websites, and forms a training data set with tweets that have been marked for authenticity, and a test data set with tweets that have not been marked for authenticity. The collected tweets are then processed to obtain the natural language content, visual content, and tag information of the tweets. That is, in the adaptive tweet processing module, the present invention first extracts the natural language content and visual content in the tweets; secondly, an adaptive algorithm is used to remove redundancy and noise in the natural language content and visual content; finally, the modal type of the tweet (only natural language content, only visual content, and both natural language and visual content) is marked to obtain the natural language content, visual content, and tag information of the tweets.

[0078] 2) The data enhancement module performs data enhancement on the training dataset: the natural language content of the tweets in the training dataset is obtained, and the natural language content of the tweets is input into the large language model of the data enhancement module to rewrite the text. The new tweets are composed of the visual content and tag information of the tweets to expand the training dataset. That is, in the data enhancement module, the present invention first obtains the natural language content of the tweets in the training dataset, inputs it into the large language model of the data enhancement module, rewrites the text, and changes the text style while maintaining content consistency. To reflect reality, the data enhancement module uses two different prompts for true and false tweets.

[0079] 3) The multimodal feature construction module extracts multimodal pattern features in the natural language content and visual content, as well as fact features in the natural language content and visual content through a multimodal dual feature extraction method; that is, in the multimodal feature construction module, the present invention proposes a multimodal dual feature extraction method, which includes two parts: pattern feature extraction and fact feature extraction, to extract multimodal pattern features and fact features in the natural language content and visual content in the original tweet.

[0080] 4) The fake news detection module combines a hybrid expert network with a gating network to aggregate tweet features and classify fake news. Specifically, within the fake news verification module, the present invention proposes a multi-expert network fake news detection method, which includes a hybrid expert network with a gating network. The adaptive tweet processing module obtains tweet tag information, and the gating network can flexibly process all types of tweets, ultimately achieving the goal of classifying tweets into true and false messages. A classifier is added after the hybrid expert network to verify the authenticity of tweets. A global loss is set to the classifier's loss, so that constraints are propagated back during the gradient backpropagation phase to coordinate the representations of each expert.

[0081] Example 3:

[0082] This embodiment is further optimized on the basis of the above embodiment, and the similarities with the above technical solution are not repeated here. Figures 1 to 6 As shown, the adaptive tweet processing module includes the following steps when processing the received tweets:

[0083] S1: The adaptive tweet processing module collects tweets from online platforms, where tweets published by authoritative platforms are marked as true and tweets published by rumor-debunking websites are marked as false. The tweets marked as true and false are used as the training dataset, and tweets published by other websites are used as the test dataset to be tested.

[0084] S2: For the pictures in the tweet, extract the natural language information and combine it with the text in the tweet as the natural language content of the tweet. For the pictures in the tweet, use them as the visual content of the tweet. Specifically, for a tweet m = [T, I1, ..., I n ], n≥0, where T is the text, I i Represents one of the pictures, then the natural language content C l =[T], visual content C v =[I1, ..., I n ]. By observing the natural language content embedded in the image (such as subtitles, text screenshots, etc.), it contains important clues to judge the authenticity of the tweet. Therefore, the adaptive tweet processing module uses OCR to extract the natural language content in the image. I , get the natural language content C l =[T, T I ].

[0085] S3: Calculate the semantic similarity between the natural language content and visual content of the tweets and filter out irrelevant visual content. l and C v , if C l If θ is empty, it means that there is no natural language content in the tweet, and the content type θ of the tweet is marked as only visual content type θ v If C l Not empty, for C v Each picture in I i , use BLIP model to calculate image-text similarity sim(C l , I i ). If the image-text similarity is greater than the threshold 0, then the image I i Add to the candidate list P. If the final candidate list P is empty, that is, the visual modalities in the tweet are all noise, then the content type of the original tweet is recorded as only the natural language content type θ l If the set P is not empty, that is, the tweet contains both visual content and natural language content, then the modality type of the tweet is marked as the natural language and visual content coexistence type θ f , and C v Update to P to eliminate the impact of noise on subsequent tasks. Finally, C v The images in the image are stitched vertically into a large image.

[0086] Example 4:

[0087] This embodiment is further optimized on the basis of embodiment 2 or 3, and the same points as the above technical solutions are not repeated here. Figures 1 to 6As shown, the data enhancement module includes the following steps when performing processing:

[0088] S1: During the training phase of the dual-feature-based adaptive multimodal fake news detection model, the natural language content of the tweets marked as true in the training dataset is obtained. Style transfer is used to prompt: "Rewriting task, please use your ability to rewrite the text to be rewritten with high quality. Make sure to retain the core information of the original text. Use different vocabulary and sentence structures so that the new version is not too similar to the original. Eliminate traces of direct copying and adapt to different audiences." The natural language content and prompt are input into the large language model to obtain the output, which is used as the tweet after style transformation and added to the training dataset.

[0089] S2: During the training phase of the adaptive multimodal false news detection model based on dual features, the natural language content of the tweets marked as false in the training dataset is obtained, and the style rewriting prompt is used to "rewrite and package a piece of false information text, so as to make it more concealed and credible by reconstructing the text and transforming the expression while retaining the false information in the text, so that readers will find it difficult to identify the false information in the original text when reading. The original information in the text (time, place, people, identity, events, positions, opinions, emotions) is retained. Avoid colloquialism and use written language. Use the skills of the language model to enrich the details of the event." The natural language content and prompts are input into the large language model to obtain the output, which is added to the training dataset as the tweet after style transformation.

[0090] Example 5:

[0091] This embodiment is further optimized on the basis of embodiment 2, 3 or 4, and the same points as the above technical solutions are not repeated here. Figures 1 to 6 As shown, the multimodal feature construction module includes the following steps when processing:

[0092] S1: Extracting pattern features from natural language content. This paper uses the natural language content with input length L as T0 to pre-train the BERT model. In the 12-layer BERT model, each layer consists of a multi-head self-attention layer (MSA) and a feed-forward network layer (FFN). The i-th layer of the BERT model is simplified as Then the output feature of the i-th layer can be expressed as T i is a 768-dimensional sequence of length L. For content type θ l For the message, the present invention takes T 12 As its pattern feature r p .

[0093] S2: Extracting pattern features from visual content. The present invention inputs visual content into the pre-trained ViT model. V ∈R H×W×C , where (H, W) is the resolution of the input image and C is the number of channels, reshaped into a two-dimensional sequence of blocks As input V0, where (Q, Q) is the image block resolution, N = HW / P 2 is the number of blocks. The i-th layer of the ViT model is simplified as The output features of the i-th layer can be expressed as V i is a 768-dimensional sequence of length N. For content type θ v As for the message, the present invention takes V 12 As its pattern feature r p .

[0094] S3: Obtain multimodal pattern features. The present invention adopts a common attention mechanism to fuse the pattern features of visual content and the pattern features of natural language content, and adopts a bridging method to align the single-modal features of each layer in cross-modal fusion to achieve the fusion of cross-modal features. At the same time, in order to prevent the weight of a single modality from being too large, the present invention introduces a learnable weight to learn the proportion of each modality in the fusion feature. The present invention defines the i-th layer of the cross-modal fusion model as It consists of a visual module and a text module. Each part consists of a MSA, a multi-head cross attention layer (MCA) and a FFN, such as Figure 3 Calculate f i v is the visual output representation of the i-th layer, f i t is the natural language output representation of the i-th layer,

[0095]

[0096] in Intermediate variables in Calculated as:

[0097]

[0098] where i represents the index in the natural language or vision model. It will serve as the input of the text module MSA and the K and V matrices of the visual module MCA, and vice versa.

[0099] Among them, Bridge(x, y)=LayerNorm(x+y);

[0100] Finally, calculate the fusion features for:

[0101]

[0102] Where α is the learnable weight, satisfying the constraint For content type θ f For the message, the present invention takes As its pattern characteristics rp .

[0103] S4: Extracting factual features from natural language content. The present invention obtains the natural language content of the tweet and uses style transfer to prompt "Summarize. Please use summarization techniques to summarize the provided text in a high-quality manner. Ensure that the key content of the original text (who, what, where, when, why, and how) remains unchanged during the summarization process. Eliminate irrelevant language and focus on the key content." The natural language content and prompt are input into the large language model to obtain the output as a text summary C. s . The text summary C s As S0 input pre-trained BERT model, the i-th layer of the BERT model is simplified as Then the output feature of the i-th layer can be expressed as S i is a 768-dimensional sequence. For content type θ l For the message, this method directly converts S 12 As a factual feature r f .

[0104] S5: Extract fact features from visual content. v , using the BLIP model to obtain a two-dimensional block sequence Where N is the number of blocks, D v is the visual feature dimension. s The feature representation of Where M is C s Length, D t is the text feature dimension. For each cross attention head, there is a parameter matrix and Calculate the cross attention A∈R M×K :

[0105]

[0106] The jth row of A indicates the attention amount assigned to all 2D blocks by the jth text tag. The image-text similarity sim(C V , C s) and the derivative of A and element-wise multiply the gradient matrix by A. Then, the relevance rel(i) of the ith two-dimensional block is:

[0107]

[0108] where H is the number of attention heads, and (H) denotes the index of the attention head. The present application employs a random top-K strategy to sample the two-dimensional block N times according to the relevance, and obtains N picture descriptions C c For messages with content type θ v , the present application directly uses the BLIP model to obtain the picture description. The text summary C c is input into the pre-trained BERT model, and the ith layer of the BERT model is simplified as The features of the output of the ith layer can be represented as C i is a 768-dimensional sequence. For messages with content type θ v , the present application directly inputs C 12 into the pre-trained BERT model as the fact feature r f .

[0109] S6: Obtain multi-modal fact features. The present application employs a common attention mechanism for fusing visual content fact features and natural language content fact features, and the present application defines the ith layer of the cross-modal fusion model as It is composed of a visual module and a text module. Each part is composed of an MSA, a multi-head cross-attention layer (MCA) and an FFN, as shown in Figure 5 The calculation of f i C is the ith layer visual output representation, f i S is the ith layer natural language output representation,

[0110]

[0111] where the intermediate variable is calculated as:

[0112]

[0113] where i represents the index in the natural language or visual model. The input of the text module MSA and the K and V matrices of the visual module MCA are input into the Bridge function, and vice versa.

[0114] where Bridge(x, y) = LayerNorm(x + y);

[0115] Finally, calculate the fusion fact features for:

[0116]

[0117] Where α is the learnable weight, satisfying the constraint For content type θ f As for the message, the present invention takes Its factual characteristics f .

[0118] Example 6:

[0119] This embodiment is further optimized on the basis of embodiment 2 or 3 or 4 or 5, and the same points as the above technical solutions are not repeated here. Figures 1 to 6 As shown, the false information detection process includes the following steps:

[0120] S1: Constructing a hybrid expert network. The present invention first obtains the pattern feature r p and factual features r f Concatenate to obtain the joint representation of multimodal sequence features r c Then multiple expert networks are used to extract the representation of fake news. A single expert network can be used to i (1≤i≤N) represents, N is the number of expert networks. Through this expert network, we can get the representation r i :r i =Z i (r c );

[0121] The present invention uses Bi-LSTM network as the expert network. Then, a gated network is constructed to obtain the output weights of each expert network. First, a learnable vector e is constructed. g Used to indicate different message types. c The joint representation e obtained by average pooling f , and e g After concatenation, the network passes through the fully connected layer G(·), and the output is normalized using the softmax function to finally obtain the weight ratio vector a output by each expert network:

[0122] Then, the final message characteristics can be obtained:

[0123] S2: If the model is currently in the training phase, the prediction result of the adaptive multimodal false news detection model based on dual features is Where P() is a fully connected layer (MLP layer) with a sigmoid activation function. The final loss function of the adaptive multimodal fake news detection model based on dual features can be expressed as:

[0124] Among them, Loss() is the BCE loss function. Finally, the gradient descent method of AdamW is used to optimize all the learnable weights and parameters in the network. After the training of the adaptive multimodal false news detection model based on dual features is completed, it will be placed in Figure 1 The false news detection module in

[15] is used to achieve the purpose of detection. The classification result of the classifier is used as the final classification result of the adaptive multimodal false news detection model based on dual features.

[0125] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. An adaptive multimodal false news detection method based on dual features, characterized by: The dual-feature-based adaptive multimodal false news detection model, which is composed of an adaptive tweet processing module, a data enhancement module, a multimodal feature construction module, and a false news detection module, is implemented, including the following steps: 1) The adaptive tweet processing module collects tweets from three types of online platforms: authoritative platforms, rumor-debunking websites, and other websites. Tweets that have been labeled for authenticity form a training dataset, while tweets that have not been labeled for authenticity form a test dataset to be tested. The collected tweets are then processed to obtain their natural language content, visual content, and tag information. 2) The data augmentation module performs data augmentation on the training dataset: it obtains the natural language content of tweets in the training dataset, feeds this content into the large language model of the data augmentation module, rewrites the text, and combines it with the visual content and tag information of the tweet to form a new tweet, thus expanding the training dataset. 3) The multimodal feature construction module extracts multimodal pattern features from natural language content and visual content, as well as factual features from natural language content and visual content through a multimodal dual feature extraction method; The steps include: 3.1) Constructing a pattern feature extraction network: For natural language content, the BERT model is used to obtain outputs at each level; for visual content, the ViT model is used to obtain outputs at each level; Then, for tweets with both natural language and visual content, a learnable bridge layer is used to connect the various levels of natural language content output and the various levels of visual content output to obtain fused multimodal pattern features. For tweets with only natural language content or only visual content, the output of each layer is directly used as the pattern feature; 3.2) Fact Extraction: For tweets with both natural language and visual content, the multimodal feature construction module first uses the large language model to obtain a text summary of the natural language content. It then uses the similarity module of the large visual language model (BLIP) to calculate the similarity between the text summary and the visual content. GradCAM is used to locate image regions related to the visual content and the text summary. Finally, the large visual language model (BLIP) is used to generate image descriptions of these image regions. For tweets with labeled information containing only natural language content, the multimodal feature construction module only extracts text summaries; For tweets with only visual content, the multimodal feature construction module directly uses the large visual language model BLIP to obtain image descriptions. 3.3) Constructing fact feature extraction network: For text summaries obtained from tweets with both natural language and visual content, the multimodal feature construction module uses the BERT model to obtain outputs at each level. For image descriptions obtained from tweets with both natural language and visual content, the multimodal feature construction module also uses the BERT model to obtain outputs at each level. A learnable bridge layer is used to connect the outputs of the text summaries and image descriptions at each level to obtain fused multimodal factual features. For tweets with only natural language content, the dual-feature-based adaptive multimodal false news detection model directly uses the output of each level of the text summary as factual features. For tweets with only visual content, the dual-feature-based adaptive multimodal false news detection model directly uses the output of each level of the image description as factual features. 4) The fake news detection module combines a hybrid expert network including a gating network to aggregate tweet features to achieve fake news classification.

2. The method for detecting fake news based on dual features according to claim 1, characterized in that: The step 1) includes the following steps: 1.1) The adaptive tweet processing module collects tweets from online platforms, marking tweets published by authoritative platforms as true and tweets published by rumor-debunking websites as false. This labeled true and false tweets serves as the training dataset, while tweets published by other websites serve as the test dataset. 1.2) For images with natural language information in tweets, the natural language information in the image is extracted and combined with the text in the tweet to form the natural language information content of the tweet; For the extracted images and pure images in tweets as the visual content of tweets; 1.3) Using an adaptive algorithm to calculate semantic similarity between the acquired natural language information and visual content, filtering out redundancy and noise in the natural language and visual content; 1.4) After step 1.3), the filtered tweets are categorized and labeled as containing only natural language content, only visual content, or both natural language and visual content; 1.5) Save the natural language content, visual content, and tag information of tweets.

3. The method for detecting fake news based on dual features according to claim 1, characterized in that: The step 2) includes the following steps: 2.1) During the training phase of the dual-feature-based adaptive multimodal fake news detection model, the natural language content of tweets marked as true in the training dataset is obtained. The style transfer hint and the natural language content of the tweet are input into the large language model to obtain the natural language content rewritten by style transfer. Combine the visual content and tag information of the tweet with the natural language content rewritten by style transfer to form a new tweet and add it to the training dataset; 2.2) During the training phase of the dual-feature-based adaptive multimodal false news detection model, the natural language content of tweets marked as false in the training dataset is obtained. The style formalization hints and the natural language content of the tweets are then fed into the large language model to obtain the natural language content after the style formalization. The visual content and tag information of the tweet are combined with the natural language content rewritten by style transfer to form a new tweet and added to the training dataset.

4. The method for detecting fake news based on dual features and adaptive multimodal messaging according to claim 1, 2 or 3, wherein: The step 4) includes the following specific steps: 4.1) Constructing a Hybrid Expert Network: First, the pattern features and fact features are concatenated to obtain multimodal joint features. These multimodal joint features are then fed into multiple expert networks composed of Bi-LSTM networks to obtain multiple sequence fusion features. Simultaneously, these multimodal joint features are fed into a gating network, which first average-pools the multimodal joint features and concatenates them with a vector constructed from the tweet's tag information. After passing the vector through a fully connected layer, the output is normalized using the softmax function to obtain a weighted ratio vector output by each Bi-LSTM network. The multiple sequence fusion features and the weighted ratios are weightedly added together to obtain the final tweet representation. Finally, a fully connected layer is added as a classifier to complete the construction of the hybrid expert network. 4.2) Model Training and Testing: If the model is currently in the training phase, the output of the fully connected layer of the classifier is first processed by the sigmoid activation function to obtain a prediction result, and the BCE loss is calculated on the prediction result. If the model is currently in the inference phase, the prediction result will be used as the inference result of the adaptive multimodal false news detection model based on dual features.

5. An adaptive multimodal fake news detection model based on dual features, characterized by: include The adaptive tweet processing module collects tweets from three types of online platforms: authoritative platforms, rumor-debunking websites, and other websites. Tweets posted by authoritative platforms are marked as true, while tweets posted by rumor-debunking websites are marked as false. The tweets marked as true and false serve as the training dataset, while tweets posted by other websites serve as the test dataset to be tested. The information in different carriers of tweets is extracted to obtain natural language content and visual content. A similarity-based adaptive algorithm is used to denoise and classify the content, obtaining three types of tweets with labeled information: those with only natural language content, those with only visual content, and those with both natural language and visual content. The data augmentation module takes the natural language content of tweets in the training dataset as raw input, uses a large language model to rewrite and reconstruct the text using two types of real and fake tweet prompts, and changes the text style while maintaining content consistency. The new tweets after data augmentation are then added to the training dataset; A multimodal feature construction module is provided with a pattern feature module and a fact feature module, which extracts multimodal pattern features in natural language content and visual content, as well as fact features in natural language content and visual content through a multimodal dual feature extraction method; The steps include: 3.1) Constructing a pattern feature extraction network: For natural language content, the BERT model is used to obtain outputs at each level; for visual content, the ViT model is used to obtain outputs at each level; Then, for tweets with both natural language and visual content, a learnable bridge layer is used to connect the various levels of natural language content output and the various levels of visual content output to obtain fused multimodal pattern features. For tweets with only natural language content or only visual content, the output of each layer is directly used as the pattern feature; 3.2) Fact Extraction: For tweets with both natural language and visual content, the multimodal feature construction module first uses the large language model to obtain a text summary of the natural language content. It then uses the similarity module of the large visual language model (BLIP) to calculate the similarity between the text summary and the visual content. GradCAM is used to locate image regions related to the visual content and the text summary. Finally, the large visual language model (BLIP) is used to generate image descriptions of these image regions. For tweets with labeled information containing only natural language content, the multimodal feature construction module only extracts text summaries; For tweets with only visual content, the multimodal feature construction module directly uses the large visual language model BLIP to obtain image descriptions. 3.3) Constructing fact feature extraction network: For text summaries obtained from tweets with both natural language and visual content, the multimodal feature construction module uses the BERT model to obtain outputs at each level. For image descriptions obtained from tweets with both natural language and visual content, the multimodal feature construction module also uses the BERT model to obtain outputs at each level. A learnable bridge layer is used to connect the outputs of the text summaries and image descriptions at each level to obtain fused multimodal factual features. For tweets with only natural language content, the dual-feature-based adaptive multimodal false news detection model directly uses the output of each level of the text summary as factual features. For tweets with only visual content, the dual-feature-based adaptive multimodal false news detection model directly uses the output of each level of the image description as factual features. The fake news detection module adopts a fake news detection method based on a hybrid expert network that includes a gating network to aggregate tweet features to achieve fake news classification.

6. The dual-feature-based adaptive multimodal false information detection model according to claim 5, characterized in that: The pattern feature module extracts features of each level of single modality through a pre-trained model, and uses a weighted full cross-modal bridging layer to fuse the single modalities at each level to obtain pattern features; the fact feature module first uses a large graphic and text model to extract fact content from natural language content to obtain a fact content summary, then uses the fact content summary to guide the graphic and text model to generate a visual content description, and finally uses the pattern feature method to obtain fact features.

Images