Self-adaptive multi-modal false message detection method and model based on dual features

Abstract

The invention relates to the technical field of network content security, network space security and the like, and discloses a self-adaptive multi-mode false message detection method and model based on double features, which are realized by adopting the self-adaptive multi-mode false message detection model based on the double features. The self-adaptive tweet processing module is used for obtaining natural language content, visual content and mark information of a tweet; the data enhancement module obtains natural language content of the tweet and inputs the natural language content into a large language model of the data enhancement module to rewrite the text; the multi-modal feature construction module is used for obtaining multi-modal mode features and fact features; the false message detection module combines a multi-expert network aggregation tweet feature containing a gating network to realize classification of false messages, finally achieves the beneficial effect of accurately performing false message detection, and aims at improving the accuracy of an existing false message detection method in a real scene.

Description

Technical Field

[0001] The present invention relates to the fields of network content security, cyberspace security technology, and more specifically, to an adaptive multimodal false message detection method and model based on dual features. Background Art

[0002] In the era of mobile Internet, the speed and scale of information dissemination far exceeds that of the traditional media era. Weibo, Douyin and other platforms have become the main channels for information dissemination. While people can quickly obtain information, they are often affected by false information. As a kind of misleading information, false information quickly breeds and spreads due to the low threshold for publishing, fast dissemination and wide range of influence on online social platforms, which will have a serious impact on individuals, organizations and even the entire society. In order to reduce the harm that false information may cause, false information detection technology has become a technology that urgently needs to be developed. It has broad application prospects in many aspects such as social platform management and cyberspace public opinion governance.

[0003] Traditionally, the identification of false information mainly relies on analyzing the text in the tweets themselves. This method focuses on mining the statistical characteristics and deep semantics of the text. For example, the authenticity of tweets is judged by calculating word frequency, detecting the usage patterns of specific symbols, and extracting semantics through deep neural networks. However, with the enrichment of visual content such as pictures and videos in social networks, it is difficult to cope with increasingly complex false information by relying solely on text analysis. Nowadays, multimodal methods that integrate visual content such as images and videos with text information have become the main trend in false information detection on social platforms. Due to its direct visibility and high attractiveness, visual content can more intuitively reveal the essence of false information in many cases. Combined with text analysis technology, it effectively improves the accuracy of false information identification. This type of method not only considers the text content in the tweets, but also deeply explores the potential meaning of its visual expression, making full use of the multimodal characteristics of tweets, and providing a comprehensive solution for the intricate false information detection in social networks. However, the existing multimodal false information detection methods on social platforms still have the following problems:

[0004] 1. Existing multimodal methods cannot effectively capture information that represents the content of tweets. During model training or detection, it is assumed that tweets only contain text and images. This assumption is very different from tweets shared on Weibo. Tweets come in many types, such as plain text, plain images, videos, image-text pairs, and text with multiple images. Focusing only on image-text pairs in model development and ignoring other tweet formats will result in detection models that cannot be effectively applied in reality.

[0005] 2. Current methods mainly focus on revealing the intricate relationship between the deep semantic features of images and text in false information, but often ignore the importance of shallow features contained in false information, such as syntax, vocabulary, and surface visual semantics. Improving false information detection capabilities through pre-trained models has led to a general trend of using parameter-frozen text models and the final layer output features of visual models to represent tweets. However, these methods usually ignore shallow outputs, resulting in inaccurate representation of multimodal tweets and poor cross-modal fusion effects.

[0006] 3. Existing methods ignore the importance of factual details in tweets, such as time reference, location, and personal identity, which are key indicators of tweet authenticity. Relying solely on pre-trained text and image models for feature extraction may not be able to effectively capture these factual elements. While research uses named entity recognition (NER) to extract factual entities from text, there is still a gap in addressing factual features present in images. Some methods focus on extracting visual entities from images to improve the search for multimodal clues. However, extracting entities only from images may lead to the loss of relational context between entities, thereby hindering the comprehensive representation of factual features in images, which in turn affects the detection performance of the model.

[0007] 4. With the rapid development of large language models, fake news generated by large language models have emerged. These tweets are diverse in style and more confusing. However, existing multimodal fake information datasets are manually generated tweets collected from social platforms and lack tweets generated by large language models. Therefore, existing methods find it difficult to capture the unique characteristics of fake information generated by large language models, resulting in biased detection results. Summary of the invention

[0008] The purpose of the present invention is to address the deficiencies of the prior art and provide an adaptive multimodal false information detection method and model based on dual features to solve the problems existing in multimodal false information detection methods used in social media environments, including the inability to handle the diverse organizational structure of tweets, inaccurate representation of multimodal tweets, neglect of factual features in multimodal false information, and lack of false information generated by large language models in the training data set. At the same time, this method aims to improve the accuracy of existing false information detection methods in real-world scenarios.

[0009] The present invention is implemented by the following technical scheme: an adaptive multimodal false news detection method based on dual features, which is implemented by an adaptive tweet processing module, a data enhancement module, a multimodal feature construction module and a false news detection module, and comprises the following steps:

[0010] 1) The adaptive tweet processing module collects tweets from three types of network platforms, namely, authoritative platforms, rumor-debunking websites and other websites, and forms a training data set with tweets that have been marked for authenticity, and forms a test data set to be tested with tweets that have not been marked for authenticity. The collected tweets are processed to obtain the natural language content, visual content and marking information of the tweets. That is, in the adaptive tweet processing module, the present invention first extracts the natural language content and visual content in the tweets; secondly, uses an adaptive algorithm to remove the redundant and noisy parts in the natural language content and visual content; finally, the modal type of the tweet (only natural language content, only visual content and coexistence of natural language and visual content) is marked to obtain the natural language content, visual content and marking information of the tweet.

[0011] 2) The data enhancement module performs data enhancement on the training data set: the natural language content of the tweets in the training data set is obtained, the natural language content of the tweets is input into the large language model of the data enhancement module to rewrite the text, and a new tweet is formed with the visual content and tag information of the tweet to expand the training data set; that is, in the data enhancement module, the present invention first obtains the natural language content of the tweets in the training data set, inputs it into the large language model of the data enhancement module, rewrites the text, and changes the text style while maintaining content consistency; to reflect reality, the data enhancement module uses two different prompts for true and false tweets.

[0012] 3) The multimodal feature construction module extracts multimodal pattern features in natural language content and visual content, as well as factual features in natural language content and visual content through a multimodal dual feature extraction method; that is, in the multimodal feature construction module, the present invention proposes a multimodal dual feature extraction method, which includes two parts: pattern feature extraction and factual feature extraction, to extract multimodal pattern features and factual features in the natural language content and visual content in the original tweet.

[0013] 4) The false news detection module combines a hybrid expert network including a gating network to aggregate tweet features to achieve the classification of false news. That is, in the false news verification module, the present invention proposes a multi-expert network false news detection method, which includes a hybrid expert network including a gating network. The adaptive tweet processing module obtains the tag information of the tweet and the gating network can flexibly process all types of tweets, thereby achieving the ultimate goal of classifying tweets into true messages and false messages. A classifier is added after the hybrid expert network to realize the authenticity verification of the tweets. The global loss is set to the loss of the classifier to return some constraints in the gradient backpropagation stage to coordinate the representation of each expert.

[0014] In order to further better realize the adaptive multimodal false news detection method based on dual features described in the present invention, the present invention proposes an adaptive false news processing method. The organization forms of tweets in social platforms are different. This method adaptively extracts natural language content and visual content according to the different organizational forms, so that subsequent modules can adopt different detection strategies according to the organization type of tweets, effectively improving the accuracy of the method in modality missing scenarios. At the same time, the noise content is removed by the semantic similarity algorithm to maintain the integrity of the image semantics, thereby improving the detection effect. In particular, the following setting method is adopted: the step 1) includes the following steps:

[0015] 1.1) The adaptive tweet processing module collects tweets from the network platform, where tweets posted by authoritative platforms are marked as true, and tweets posted by rumor-debunking websites are marked as false. The tweets marked as true and false are used as the training data set, and tweets posted by other websites are used as the test data set to be tested;

[0016] 1.2) For pictures with natural language information in tweets, the natural language information in the pictures will be extracted and combined with the text in the tweet as the natural language information content of the tweet; the extracted pictures and pure pictures in the tweets will be used as the visual content of the tweet;

[0017] 1.3) Using an adaptive algorithm to calculate the semantic similarity of the obtained natural language information content and visual content, filtering out the redundant and noisy parts in the natural language content and visual content;

[0018] 1.4) After step 1.3), the filtered tweets are classified and marked as those with only natural language content, only visual content, or both natural language and visual content;

[0019] 1.5) Save the natural language content, visual content, and tag information of the tweet.

[0020] To further implement the adaptive multimodal false news detection method based on dual features described in the present invention, the present invention proposes a false news data enhancement method based on a large language model, which uses a large language model to transform the style of tweets, enrich the content of a single training set, and enhance the model's detection ability for tweets of different styles. In particular, the following setting is adopted: Step 2) includes the following steps:

[0021] 2.1) In the training phase of the adaptive multimodal fake news detection model based on dual features, the natural language content of the tweets marked as true in the training dataset is obtained, and the style transfer hint and the natural language content in the tweet are input into the large language model to obtain the natural language content rewritten by style transfer; the visual content and tag information of the tweet are combined with the natural language content rewritten by style transfer to form a new tweet, which is added to the training dataset;

[0022] 2.2) In the training phase of the adaptive multimodal fake news detection model based on dual features, for tweets marked as false in the training dataset, the natural language content of the tweet is obtained, and the style formalization prompts and the natural language content in the tweet are input into the large language model together to obtain the natural language content rewritten by style formalization; the visual content and tag information of the tweet are combined with the natural language content rewritten by style transfer to form a new tweet, which is added to the training dataset.

[0023] In order to further better realize the adaptive multimodal false news detection method based on dual features described in the present invention, the present invention proposes a dual false news feature extraction method to extract multimodal pattern features and multimodal fact features in false messages. For multimodal pattern features, the present invention integrates unimodal representations at different levels. For multimodal fact features, the present invention first uses a large language model to generate a text summary, then uses the text summary to guide the visual language model to generate an accurate picture description, and finally integrates the text summary and the picture description as a multimodal fact feature. The method effectively extracts multimodal false news features from the pattern and fact levels, further improving the prediction performance and generalization ability. In particular, the following setting is adopted: the step 3) includes the following steps:

[0024] 3.1) Constructing a pattern feature extraction network: This paper proposes a pattern fusion method to extract multimodal pattern features. For natural language content, the BERT model is used to obtain the output of each level; for visual content, the ViT model is used to obtain the output of each level;

[0025] Then, for tweets with both natural language and visual content, the present invention uses a learnable bridge layer to connect the various levels of output of natural language content and the various levels of output of visual content to obtain fused multimodal pattern features;

[0026] For tweets with only natural language content or only visual content, the present invention does not fuse features and directly uses the output of each level as the pattern feature;

[0027] 3.2) Fact Extraction:

[0028] For tweets with both natural language and visual content, the multimodal feature construction module first uses the large language model to obtain the text summary of the natural language content. Then, the similarity module of the large visual language model BLIP is used to calculate the similarity between the text summary and the visual content. GradCAM is used to locate the image areas related to the visual content and the text summary. Finally, the large visual language model BLIP is used to generate image descriptions of these image areas.

[0029] For tweets with labeled information that only contains natural language content, the multimodal feature construction module only extracts text summaries;

[0030] For tweets with only visual content, the multimodal feature construction module will directly use the visual language model BLIP to obtain the image description;

[0031] 3.3) Construct fact feature extraction network:

[0032] The present invention proposes a fact fusion method for extracting multimodal fact features. For text summaries obtained from tweets with both natural language and visual content, the multimodal feature construction module uses the BERT model to obtain outputs at all levels. For picture descriptions obtained from tweets with both natural language and visual content, the multimodal feature construction module uses the BERT model to obtain outputs at all levels. The present invention uses a learnable bridge layer to connect the outputs of the text summaries and picture descriptions at all levels to obtain fused multimodal fact features.

[0033] For tweets with labeled information that only contain natural language content, the dual-feature-based adaptive multimodal false news detection model directly uses the output of each level of the text summary as factual features;

[0034] For tweets whose labeled information only contains visual content, the dual-feature based adaptive multimodal false news detection model directly uses the output of each level of the image description as factual features.

[0035] To further implement the adaptive multimodal false news detection method based on dual features described in the present invention, the present invention proposes a multi-expert network false news detection method, which uses a gated network to guide the multi-expert network to learn corresponding representations for different types of false news content, thereby improving the model's ability to detect false news of different organizational types. In particular, the following setting is adopted: Step 4) includes the following specific steps:

[0036] 4.1) Constructing a hybrid expert network: First, concatenate pattern features and fact features to obtain multimodal joint features, and then input the multimodal joint features into multiple expert networks composed of Bi-LSTM networks to obtain multiple sequence fusion features; at the same time, the multimodal joint features are input into the gating network, which first average-pools the multimodal joint features and concatenates them with the vector constructed with the tag information of the tweet. After passing through a fully connected layer, the output is normalized using the softmax function to finally obtain the weight ratio vector output by each Bi-LSTM network; multiple sequence fusion features are weightedly added with the weight ratio to obtain the final tweet representation, and finally a fully connected layer is added as a classifier to complete the construction of the hybrid expert network.

[0037] 4.2) Model training and detection: If the model is currently in the training stage, the output of the fully connected layer of the classifier is first processed by the sigmoid activation function to obtain a prediction result, and the BCE loss is calculated for the prediction result; if the model is currently in the inference stage, the prediction result will be used as the inference result of the adaptive multimodal false news detection model based on dual features.

[0038] An adaptive multimodal false news detection model based on dual features, including

[0039] The adaptive tweet processing module collects tweets from three types of network platforms: authoritative platforms, rumor-debunking websites, and other websites. Tweets posted by authoritative platforms are marked as true, and tweets posted by rumor-debunking websites are marked as false. The tweets marked as true and false are used as training data sets, and tweets posted by other websites are used as test data sets to be tested. The information in different carriers of tweets is extracted to obtain natural language content and visual content, and the content is denoised and classified using an adaptive algorithm based on similarity, obtaining three types of tweets with marked information: tweets with only natural language content, tweets with only visual content, and tweets with both natural language and visual content.

[0040] The data augmentation module takes the natural language content of tweets in the training dataset as raw input, uses a large language model to rewrite and reconstruct the text using two types of true and false tweet prompts, changes the text style while maintaining content consistency, and then adds the new tweets after data augmentation to the training dataset;

[0041] The multimodal feature construction module includes a pattern feature module and a fact feature module, which are used to obtain tweets and extract two types of features: pattern and fact;

[0042] The fake news detection module adopts a fake news detection method based on a hybrid expert network that contains a gating network to aggregate tweet features to achieve the classification of fake news.

[0043] In order to better realize the adaptive multimodal false information detection model based on dual features described in the present invention, the following setting is particularly adopted: the pattern feature module extracts the features of each level of the single modality through the pre-trained model, and uses the weighted full cross-modal bridging layer (learnable bridging layer) to fuse the single modalities of each level to obtain the pattern feature; the fact feature module first uses the large picture and text model to extract the fact content in the natural language content to obtain the fact content summary, and then uses the fact content summary to guide the large picture and text model to generate visual content description, and finally uses the pattern feature method to obtain the fact feature.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] (1) Existing multimodal false information detection methods cannot handle the diverse organizational structure of tweets and are therefore not well suited for application in social media environments. The present invention addresses these deficiencies by proposing an adaptive tweet processing method. The proposed method includes extracting natural language and visual content from tweets, thereby standardizing various tweet formats into a unified structure. Subsequently, a similarity-based algorithm is used to address issues related to noisy content and situations where natural language or visual content may be missing, thereby improving the overall effectiveness of the detection model. In addition, the present invention also proposes a multi-expert network that helps the expert network learn effective features through labels, thereby improving the detection rate of various types of false information.

[0046] (2) Considering that current false information detection methods prioritize deep semantic features over shallow features, resulting in inaccurate representation of multimodal tweets, the present invention designs a weighted bridging layer (i.e., a learnable bridging layer, also a weighted full cross-modal bridging layer) based on a common attention mechanism to fuse different levels of features extracted by pre-trained image and text models. By introducing multiple learnable bridging layers between the text and visual models, different levels of the unimodal model can be connected to the cross-modal module. This achieves effective cross-modal alignment and fusion between different levels of visual and text representations. Combined with learnable weights, this module improves the modal representation of multimodal false information and enhances the false message detection capability.

[0047] (3) In view of the problem that current methods ignore factual features in multimodal false information, the present invention proposes a zero-shot multimodal factual feature extraction method (a fact fusion method) based on a large visual language model. First, the factual content in the text is extracted using the understanding ability of the large visual language model to obtain a text summary; then the text summary is used as a prompt word to guide the large visual language model to generate a title for the key part of the picture; finally, the text summary and the picture title are combined to obtain factual features, and the factual features are used to enhance the false information detection capability.

[0048] (4) In order to solve the problem that the current dataset lacks fake information generated by large language models, resulting in a decrease in detection accuracy, this paper proposes a multimodal fake information data enhancement scheme based on large language models to obtain a dataset similar to the real-world scenario. The scheme uses a large language model to imitate the writing style of real tweets and integrates image descriptions to make fake tweets more complex.

[0049] (5) The method described in the present invention divides the false message detection task into four modules: adaptive tweet processing task, data enhancement task, multimodal feature construction task and false message detection classification task. The method adaptively processes tweets appearing on social platforms to avoid the impact of modal missing or noise content on false message detection. At the same time, the detection ability of the method for messages with diverse styles is enhanced through data enhancement, and the pattern and factual characteristics of false messages can be accurately captured to achieve accurate false message detection. The model divides the false message detection task into four modules: adaptive tweet processing module, data enhancement module, multimodal feature construction module and false message detection module. The model adaptively processes tweets appearing on social platforms to avoid the impact of modal missing or noise content on false message detection. At the same time, the detection ability of the method for messages with diverse styles is enhanced through data enhancement, and the pattern and factual characteristics of false messages can be accurately captured to achieve accurate false message detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The overall flow chart of fake news detection on social networks.

[0051] Figure 2 Processing flow chart for building modules for multimodal features.

[0052] Figure 3 Diagram of the modality fusion structure in the multimodal feature building module.

[0053] Figure 4 Flowchart of fact extraction in the building module for multimodal features.

[0054] Figure 5 Diagram of the fact fusion structure in the multimodal feature building module.

[0055] Figure 6 This is the structural diagram of the false information detection module.

[0056] In the figure, Bridge is the bridge layer; Query is the query matrix in the attention mechanism; Key is the key matrix in the attention mechanism; Value is the value matrix in the attention mechanism; Gate is the gated network; Expert is the expert network; MSA is the multi-head self-attention layer; FNN is the feedforward network layer. DETAILED DESCRIPTION

[0057] The present invention is further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0060] Glossary:

[0061] BERT model: Bidirectional Encoder Representations from Transformers (BERT), a pre-trained model for natural language processing (NLP)

[0062] ViT model: Visual Transformer (ViT), a pre-trained model for the visual field

[0063] BLIP: Bootstrapping language-image pre-training, is a technology used for training large text vision models.

[0064] GradCAM: Grad-CAM (Gradient-weighted Class Activation Mapping) is a method for explaining the predictions of deep learning models.

[0065] Bi-LSTM network: Bidirectional Long Short-Term Memory network (Bi-LSTM) is a deep learning model and a variant of recurrent neural network (RNN).

[0066] BCE: BCE loss function is the abbreviation of Binary Cross-Entropy loss function. In binary classification problems, BCE loss function is usually used to measure the difference between model output and actual label.

[0067] OCR: OCR is the abbreviation of Optical Character Recognition. OCR is a technology that can convert printed or handwritten text (such as books, documents, pictures, etc.) into editable text data. OCR technology recognizes characters, numbers and symbols in images and converts them into text format that can be recognized by computers.

[0068] AdamW: Adam (Adaptive Moment Estimation) is a gradient descent optimization algorithm. AdamW is a variant of the Adam optimization algorithm. It improves the problem of learning rate attenuation in the Adam algorithm.

[0069] Embodiment 1:

[0070] The present invention designs an adaptive multimodal false information detection model based on dual features, combining Figure 1 Shown, including

[0071] The adaptive tweet processing module (i.e., adaptive message processing module) collects tweets from three types of network platforms: authoritative platforms, rumor-debunking websites, and other websites. Tweets posted by authoritative platforms are marked as true, and tweets posted by rumor-debunking websites are marked as false. The tweets marked as true and false are used as training data sets, and the tweets posted by other websites are used as test data sets to be tested. The information in different carriers of tweets is extracted to obtain natural language content and visual content, and the content is denoised and classified using an adaptive algorithm based on similarity, obtaining three types of tweets with marked information: tweets with only natural language content, tweets with only visual content, and tweets with both natural language and visual content.

[0072] The data augmentation module augments the tweets in the database to make them fit the real-world scenarios. The natural language content of the tweets in the training dataset is used as the original input. The large language model is used to rewrite and reconstruct the text using two types of true and false tweets. The text style is changed while maintaining content consistency. The new tweets after data augmentation are then added to the training dataset.

[0073] The multimodal feature construction module is provided with a pattern feature module and a fact feature module, which are used to obtain tweets and extract two types of features: pattern and fact. The pattern feature module extracts the features of each level of the single modality through a pre-trained model, and fuses the single modalities of each level using a weighted full cross-modal bridge layer (learnable bridge layer) to obtain the pattern feature. The fact feature module first uses the large picture and text model to extract the fact content in the natural language content to obtain the fact content summary, and then uses the fact content summary to guide the large picture and text model to generate visual content description, and finally uses the pattern feature method to obtain the fact feature.

[0074] The fake news detection module adopts a fake news detection method based on a hybrid expert network that includes a multi-expert network and a gated network to aggregate tweet features to achieve the classification of fake news (distinguishing between real news and fake news).

[0075] Embodiment 2:

[0076] An adaptive multimodal false news detection method based on dual features, combined with Figure 1 to Figure 6 As shown, the dual-feature-based adaptive multimodal false news detection model composed of an adaptive tweet processing module, a data enhancement module, a multimodal feature construction module and a false news detection module is implemented, including the following steps:

[0077] 1) The adaptive tweet processing module collects tweets from three types of network platforms, namely, authoritative platforms, rumor-debunking websites and other websites, and forms a training data set with tweets that have been marked for authenticity, and forms a test data set to be tested with tweets that have not been marked for authenticity. The collected tweets are processed to obtain the natural language content, visual content and marking information of the tweets. That is, in the adaptive tweet processing module, the present invention first extracts the natural language content and visual content in the tweets; secondly, uses an adaptive algorithm to remove the redundant and noisy parts in the natural language content and visual content; finally, the modal type of the tweet (only natural language content, only visual content and coexistence of natural language and visual content) is marked to obtain the natural language content, visual content and marking information of the tweet.

[0078] 2) The data enhancement module performs data enhancement on the training data set: the natural language content of the tweets in the training data set is obtained, the natural language content of the tweets is input into the large language model of the data enhancement module to rewrite the text, and a new tweet is formed with the visual content and tag information of the tweet to expand the training data set; that is, in the data enhancement module, the present invention first obtains the natural language content of the tweets in the training data set, inputs it into the large language model of the data enhancement module, rewrites the text, and changes the text style while maintaining content consistency; to reflect reality, the data enhancement module uses two different prompts for true and false tweets.

[0079] 3) The multimodal feature construction module extracts multimodal pattern features in natural language content and visual content, as well as factual features in natural language content and visual content through a multimodal dual feature extraction method; that is, in the multimodal feature construction module, the present invention proposes a multimodal dual feature extraction method, which includes two parts: pattern feature extraction and factual feature extraction, to extract multimodal pattern features and factual features in the natural language content and visual content in the original tweet.

[0080] 4) The false news detection module combines a hybrid expert network including a gating network to aggregate tweet features to achieve the classification of false news. That is, in the false news verification module, the present invention proposes a multi-expert network false news detection method, which includes a hybrid expert network including a gating network. The adaptive tweet processing module obtains the tag information of the tweet and the gating network can flexibly process all types of tweets, thereby achieving the ultimate goal of classifying tweets into true messages and false messages. A classifier is added after the hybrid expert network to realize the authenticity verification of the tweets. The global loss is set to the loss of the classifier to return some constraints in the gradient backpropagation stage to coordinate the representation of each expert.

[0081] Embodiment 3:

[0082] This embodiment is further optimized on the basis of the above embodiment, and the same points as the above technical solution are not repeated here. Figure 1 to Figure 6 As shown, the adaptive tweet processing module includes the following steps when processing the received tweets:

[0083] S1: The adaptive tweet processing module collects tweets from the network platform, among which tweets released by authoritative platforms are marked as true, and tweets released by rumor-debunking websites are marked as false. The tweets marked as true and false are used as training data sets, and tweets released by other websites are used as test data sets to be tested.

[0084] S2: For the pictures in the tweets, extract the natural language information and combine it with the text in the tweets as the natural language content of the tweets. For the pictures in the tweets, use them as the visual content of the tweets. Specifically, for a tweet m = [T, I1, ..., I n ], n≥0, where T is the text, I i represents one of the pictures, then the natural language content C l =[T], visual content C v =[I1, ..., I n ]. By observing the natural language content embedded in the image (such as subtitles, text screenshots, etc.), it contains important clues to judge the authenticity of the tweet. Therefore, the adaptive tweet processing module uses OCR to extract the natural language T in the image. I , get the natural language content C l =[T,T I ].

[0085] S3: Calculate the semantic similarity of the natural language content and visual content of the tweet, and filter out irrelevant visual content. l and C v , if C l If is empty, it means that there is no natural language content in the tweet, and the content type θ of the tweet is marked as only the visual content type θ v If C l Not empty for C v Each picture in I i , use the BLIP model to calculate the image-text similarity sim(C l , I i ). If the image-text similarity is greater than the threshold 0, then image I i Add to the candidate list P. If the final candidate list P is empty, that is, the visual modalities in the tweet are all noise, then the content type of the original tweet is recorded as only the natural language content type θ l If the set P is not empty, that is, the tweet contains both visual content and natural language content, then the modality type of the tweet is marked as the natural language and visual content coexistence type θ f , and C v Update to P to eliminate the impact of noise on subsequent tasks. Finally, C v The images in the image are stitched vertically into one large image.

[0086] Embodiment 4:

[0087] This embodiment is further optimized on the basis of embodiment 2 or 3, and the same points as the above technical solutions are not repeated here. Figure 1 to Figure 6As shown, the data enhancement module includes the following steps when processing:

[0088] S1: In the training phase of the adaptive multimodal fake news detection model based on dual features, the natural language content of the tweets marked as true in the training dataset is obtained, and the style transfer prompt "Rewriting task, please use your ability to rewrite the text to be rewritten with high quality. Make sure to retain the core information of the original text. Use different vocabulary and sentence structures so that the new version is not too similar to the original. Eliminate traces of direct copying and adapt to different audiences." The natural language content and prompts are input into the large language model to obtain the output, which is added to the training dataset as the tweet after style transformation.

[0089] S2: In the training phase of the adaptive multimodal false news detection model based on dual features, the natural language content of the tweets marked as false in the training dataset is obtained, and the style rewriting prompt is used to "rewrite and package a piece of false information text, so as to make it more concealed and more credible by reconstructing the text and transforming the expression while retaining the false information in the text, so that readers will find it difficult to identify the false information in the original text when reading. The original information in the text (time, place, people, identity, events, positions, opinions, emotions) is retained. Avoid colloquialism and use written language. Use the skills of the language model to enrich the details of the event." The natural language content and prompts are input into the large language model to obtain the output, which is added into the training dataset as the tweet after style transformation.

[0090] Embodiment 5:

[0091] This embodiment is further optimized on the basis of Embodiment 2, 3 or 4, and the same points as the above-mentioned technical solutions are not repeated here. Figure 1 to Figure 6 As shown, the multimodal feature construction module includes the following steps when processing:

[0092] S1: Extracting pattern features from natural language content. The present invention uses natural language content with an input length of L as T0 to pre-train the BERT model. In the 12-layer BERT model, each layer consists of a multi-head self-attention layer (MSA) and a feed-forward network layer (FFN). The i-th layer of the BERT model is simplified as Then the output feature of the i-th layer can be expressed as T i is a 768-dimensional sequence of length L. For content type θ l For the message, the present invention takes T 12 As its mode feature r p .

[0093] S2: Extracting pattern features from visual content. The present invention inputs visual content into the pre-trained ViT model. V ∈R H×W×C , where (H, W) is the resolution of the input image and C is the number of channels, reshaped into a two-dimensional sequence of blocks As input V0, where (Q, Q) is the image block resolution, N = HW / P 2 is the number of blocks. The i-th layer of the ViT model is simplified as The output features of the i-th layer can be expressed as V i is a 768-dimensional sequence of length N. For content type θ v For the message, the present invention takes V 12 As its mode feature r p .

[0094] S3: Obtain multimodal pattern features. The present invention adopts a common attention mechanism to fuse the pattern features of visual content and the pattern features of natural language content, and uses a bridging method to align the single-modal features of each layer during cross-modal fusion to achieve cross-modal feature fusion. At the same time, in order to prevent the weight of a single modality from being too large, the present invention introduces learnable weights to learn the proportion of each modality in the fusion feature. The present invention defines the i-th layer of the cross-modal fusion model as It consists of a visual module and a text module. Each part consists of an MSA, a multi-head cross attention layer (MCA), and a FFN, such as Figure 3 Calculate f i v is the visual output representation of the i-th layer, f i t is the natural language output representation of the i-th layer,

[0095]

[0096] in The intermediate variables in Calculated as:

[0097]

[0098] where i represents the index in the natural language or vision model. It will be used as the input of the text module MSA and the K and V matrices of the visual module MCA, and vice versa.

[0099] Among them, Bridge(x, y)=LayerNorm(x+y);

[0100] Finally, calculate the fusion features for:

[0101]

[0102] Where α is the learnable weight, satisfying the constraints For content type θ f For the message, the present invention takes As its mode characteristics rp .

[0103] S4: Extract factual features from natural language content. The present invention obtains the natural language content of the tweet, uses style transfer to prompt "Summary, please use summarization skills to summarize the provided text in high quality. Ensure that the key content of the original text (who, what, where, when, why and how) remains unchanged during the summarization process. Eliminate irrelevant language and focus on key content." The natural language content and prompt are input into the large language model to obtain the output as a text summary C s . The text summary C s As S0 inputs the pre-trained BERT model, the i-th layer of the BERT model is simplified as Then the output feature of the i-th layer can be expressed as S i is a 768-dimensional sequence. For content type θ l For messages of 12 As a fact feature f .

[0104] S5: Extract factual features from visual content. v , using the BLIP model to obtain a two-dimensional block sequence Where N is the number of blocks, D v is the visual feature dimension. s The characteristic representation of Where M is C s Length, D t is the text feature dimension. For each cross-attention head, there is a parameter matrix and Calculate the cross attention A∈R M×K :

[0105]

[0106] The jth row of A indicates the attention amount assigned to all 2D patches by the jth text tag. The image-text similarity sim(C V , C s), and find the derivative with respect to A And multiply the gradient matrix element by element by A. Then, the correlation rel(i) of the i-th two-dimensional block is:

[0107]

[0108] Where H is the number of attention heads, and (H) represents the index of the attention head. The present invention uses a random top-K strategy to sample the two-dimensional block N times according to the correlation, and uses the BLIP model to obtain N picture descriptions C c For content type θ v For the message, the present invention directly uses the BLIP model to obtain the picture description. c As C0 input pre-trained BERT model, the i-th layer of BERT model is simplified as Then the output feature of the i-th layer can be expressed as C i is a 768-dimensional sequence. For content type θ v For messages of C 12 The fact feature r f .

[0109] S6: Obtain multimodal fact features. The present invention adopts a common attention mechanism to fuse visual content fact features and natural language content fact features. The present invention defines the i-th layer of the cross-modal fusion model as It consists of a visual module and a text module. Each part consists of an MSA, a multi-head cross attention layer (MCA), and a FFN, such as Figure 5 Calculate f i C is the visual output representation of the i-th layer, f i S is the natural language output representation of the i-th layer,

[0110]

[0111] in The intermediate variable Calculated as:

[0112]

[0113] where i represents the index in the natural language or vision model. It will be used as the input of the text module MSA and the K and V matrices of the visual module MCA, and vice versa.

[0114] Where Bridge(x,y)=LayerNorm(x+y);

[0115] Finally, calculate the fusion fact features for:

[0116]

[0117] Where α is the learnable weight, satisfying the constraints For content type θ f For the message, the present invention takes Its factual characteristics f .

[0118] Embodiment 6:

[0119] This embodiment is further optimized on the basis of Embodiment 2, 3, 4 or 5, and the same points as the above technical solutions are not repeated here. Figure 1 to Figure 6 As shown, false information detection includes the following steps when processing:

[0120] S1: Building a hybrid expert network. The present invention first obtains the pattern feature r p and factual features r f Concatenate to obtain the joint representation of multimodal sequence features r c Then multiple expert networks are used to extract representations of fake news. A single expert network can be used with Z i (1≤i≤N) represents, N is the number of expert networks. Through this expert network, we can get the representation r i :r i =Z i (r c );

[0121] The present invention uses a Bi-LSTM network as an expert network. Then, a gating network is constructed to obtain the output weights of each expert network. First, a learnable vector e is constructed. g Used to indicate different message types. c The joint representation e obtained by average pooling f , and e g After concatenation, the fully connected layer G(·) is passed through and the output is normalized using the softmax function to finally obtain the weight ratio vector a of each expert network output:

[0122] Then, the final message characteristics can be obtained:

[0123] S2: If the model is currently in the training stage, the prediction result of the adaptive multimodal false news detection model based on dual features is Where P() is a fully connected layer (MLP layer) with a sigmoid activation function. The final loss function of the adaptive multimodal false news detection model based on dual features can be expressed as:

[0124] Among them, Loss() is the BCE loss function. Finally, the gradient descent method of AdamW is used to optimize all learnable weights and parameters in the network. After the training of the adaptive multimodal false news detection model based on dual features is completed, it will be placed in Figure 1 In the fake news detection module in , the purpose of detection is achieved. The classification result of the classifier is used as the final classification result of the adaptive multimodal fake news detection model based on dual features.

[0125] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. An adaptive multimodal false news detection method based on dual features, characterized by: The implementation of the dual-feature based adaptive multimodal false news detection model, which is composed of an adaptive tweet processing module, a data enhancement module, a multimodal feature construction module and a false news detection module, includes the following steps: 1) The adaptive tweet processing module collects tweets from three types of online platforms: authoritative platforms, rumor-debunking websites, and other websites. The tweets with authenticity marks form a training data set, and the tweets without authenticity marks form a test data set to be tested. The collected tweets are processed to obtain the natural language content, visual content, and tag information of the tweets; 2) The data enhancement module performs data enhancement on the training dataset: the natural language content of the tweets in the training dataset is obtained, and the natural language content of the tweets is input into the large language model of the data enhancement module to rewrite the text, and the new tweets are formed with the visual content and tag information of the tweets to expand the training dataset; 3) The multimodal feature construction module extracts multimodal pattern features in natural language content and visual content, as well as factual features in natural language content and visual content through a multimodal dual feature extraction method; 4) The fake news detection module combines a hybrid expert network including a gating network to aggregate tweet features to achieve fake news classification.

2. The method for detecting fake news based on dual features according to claim 1 is characterized in that: The step 1) comprises the following steps: 1.1) The adaptive tweet processing module collects tweets from the network platform, among which tweets published by authoritative platforms are marked as true, and tweets published by rumor-debunking websites are marked as false. The tweets marked as true and false are used as the training data set, and tweets published by other websites are used as the test data set to be tested; 1.2) For pictures with natural language information in tweets, the natural language information in the pictures will be extracted and combined with the text in the tweet as the natural language information content of the tweet; For the extracted images and pure images in the tweets as the visual content of the tweets; 1.3) Adopting an adaptive algorithm to calculate the semantic similarity of the obtained natural language information content and visual content, and filtering out the redundant and noisy parts in the natural language content and visual content; 1.4) After step 1.3), the filtered tweets are classified and labeled as those with only natural language content, only visual content, or both natural language and visual content; 1.5) Save the natural language content, visual content, and tag information of the tweet.

3. The method for detecting fake news based on dual features according to claim 1 is characterized in that: The step 2) comprises the following steps: 2.1) In the training phase of the adaptive multimodal fake news detection model based on dual features, the natural language content of the tweets marked as true in the training dataset is obtained, and the style transfer hint and the natural language content in the tweet are input into the large language model to obtain the natural language content rewritten by style transfer; Combine the visual content and tag information of the tweet with the natural language content rewritten by style transfer to form a new tweet and add it to the training dataset; 2.2) In the training phase of the adaptive multimodal fake news detection model based on dual features, for tweets marked as fake in the training dataset, the natural language content of the tweet is obtained, and the style formalization hint and the natural language content in the tweet are input into the large language model to obtain the natural language content after the style formalization rewrite; The visual content and tag information of the tweet are combined with the natural language content rewritten by style transfer to form a new tweet and added to the training dataset.

4. The method for detecting fake news based on dual features according to claim 1 is characterized in that: The step 3) comprises the following steps: 3.1) Constructing a pattern feature extraction network: For natural language content, the BERT model is used to obtain the output of each level; for visual content, the ViT model is used to obtain the output of each level; Then, for tweets with both natural language and visual content, a learnable bridge layer is used to connect the various levels of output of natural language content and the various levels of output of visual content to obtain fused multimodal pattern features. For tweets with labeled information that only contains natural language content or only contains visual content, the output of each level is directly used as the pattern feature; 3.2) Fact Extraction: For tweets with both natural language and visual content, the multimodal feature construction module first uses the large language model to obtain the text summary of the natural language content. Then, the similarity module of the large visual language model BLIP is used to calculate the similarity between the text summary and the visual content. GradCAM is used to locate the image areas related to the visual content and the text summary. Finally, the large visual language model BLIP is used to generate image descriptions of these image areas. For tweets with labeled information that only contains natural language content, the multimodal feature construction module only extracts text summaries; For tweets with only visual content, the multimodal feature construction module will directly use the visual language model BLIP to obtain the image description; 3.3) Construct fact feature extraction network: For the text summaries obtained from tweets with both natural language and visual content, the multimodal feature construction module uses the BERT model to obtain outputs at all levels. For the image descriptions obtained from tweets with both natural language and visual content, the multimodal feature construction module uses the BERT model to obtain outputs at all levels. A learnable bridge layer is used to connect the outputs of the text summaries and image descriptions at all levels to obtain fused multimodal fact features. For tweets with labeled information that only contain natural language content, the dual-feature-based adaptive multimodal false news detection model directly uses the output of each level of the text summary as factual features; For tweets whose labeled information only contains visual content, the dual-feature based adaptive multimodal false news detection model directly uses the output of each level of the image description as factual features.

5. The method for detecting fake news based on dual features according to claim 1, 2 or 3 is characterized in that: The step 4) includes the following specific steps: 4.1) Constructing a hybrid expert network: First, concatenate the pattern features and fact features to obtain multimodal joint features, then input the multimodal joint features into multiple expert networks composed of Bi-LSTM networks to obtain multiple sequence fusion features; at the same time, the multimodal joint features are input into the gating network, which first average-pools the multimodal joint features and concatenates them with the vector constructed from the tag information of the tweets. After passing through a fully connected layer, the output is normalized using the softmax function to finally obtain the weight ratio vector output by each Bi-LSTM network; multiple sequence fusion features and weight ratios are weightedly added to obtain the final tweet representation, and finally a fully connected layer is added as a classifier to complete the construction of the hybrid expert network; 4.2) Model training and detection: If the model is currently in the training stage, the output of the fully connected layer of the classifier is first processed by the sigmoid activation function to obtain a prediction result, and the BCE loss is calculated for the prediction result; if the model is currently in the inference stage, the prediction result will be used as the inference result of the adaptive multimodal false news detection model based on dual features.

6. An adaptive multimodal false news detection model based on dual features, characterized by: include The adaptive tweet processing module collects tweets from three types of network platforms: authoritative platforms, rumor-debunking websites, and other websites. Tweets posted by authoritative platforms are marked as true, and tweets posted by rumor-debunking websites are marked as false. The tweets marked as true and false are used as training data sets, and tweets posted by other websites are used as test data sets to be tested. The information in different carriers of tweets is extracted to obtain natural language content and visual content, and the content is denoised and classified using a similarity-based adaptive algorithm to obtain three types of tweets with labeled information: tweets with only natural language content, tweets with only visual content, and tweets with both natural language and visual content. The data augmentation module takes the natural language content of tweets in the training dataset as raw input, uses a large language model to rewrite and reconstruct the text using two types of true and false tweet prompts, changes the text style while maintaining content consistency, and then adds the new tweets after data augmentation to the training dataset; The multimodal feature construction module includes a pattern feature module and a fact feature module, which are used to obtain tweets and extract two types of features: pattern and fact; The fake news detection module adopts a fake news detection method based on a hybrid expert network that contains a gating network to aggregate tweet features to achieve the classification of fake news.

7. The dual-feature-based adaptive multimodal false information detection model according to claim 6, characterized in that: The pattern feature module extracts features of each level of single modality through a pre-trained model, and uses a weighted full cross-modal bridging layer to fuse single modalities at each level to obtain pattern features; the fact feature module first uses a large picture and text model to extract factual content from natural language content to obtain a factual content summary, then uses the factual content summary to guide the large picture and text model to generate a visual content description, and finally uses the pattern feature method to obtain factual features.

Images