A multi-modal fake news detection method and system

Abstract

The present application relates to a kind of multi-modal fake news detection method and system, belong to fake news detection technical field, based on resistance network thought, training stage is added in multi-modal feature extractor and fake news detector in event classifier to predict event auxiliary label, and the corresponding loss is used to estimate the difference between the feature representation of different events, the greater the loss, the lower the difference, while multi-modal feature extractor attempts to deceive event classifier to learn the common features of different events, enrich the extracted feature types, so as to complete the detection and identification of fake news by the trained multi-modal feature extractor and fake news detector cooperation, improve the accuracy of fake news detection.

Description

Technical Field

[0001] This invention relates to the field of fake news detection technology, and in particular to a multimodal fake news detection method and system. Background Technology

[0002] Social media platforms provide users with convenient channels to create, access, and share various types of information. Today, more and more people use social media to find and receive timely news, which can provide timely and comprehensive multimedia information about events happening around the world. Compared to traditional news containing only text, news with images and videos can make the content more vivid and attract more readers. However, these features also become advantages for fake news, misleading readers and spreading rapidly. The widespread dissemination of fake news can cause large-scale negative impacts, sometimes even becoming public events with severe repercussions. Furthermore, rampant "online" fake news can also lead to "offline" social events. Therefore, there is an urgent need for an automated detector to detect and identify fake news, reduce its spread in the media, and thus mitigate the serious negative impacts of fake news.

[0003] To date, various fake news detection methods have been used to identify fake news, including traditional learning methods and deep learning-based models. Through thorough validation on different events, existing deep learning models have shown improved performance compared to traditional models. However, detecting fake news on social media still faces unique challenges, namely detecting and identifying fake news on newly emerging events. Due to the lack of relevant prior knowledge, it is difficult to obtain posts that can validate such events in a timely manner, leading to unsatisfactory performance of existing models. In fact, existing models tend to capture event-specific characteristics during training and use these characteristics to predict different events during the prediction phase. While these event-specific characteristics can accurately classify posts of validated events, they do not perform well when detecting newly emerging events.

[0004] Against this backdrop, engineers may need to learn common attributes or features across different events to detect fake news from unverified posts. Posts representing different events typically possess unique or non-shareable specific features. These features can be detected by comparing the differences between corresponding posts across different events. Therefore, identifying event-specific features is equivalent to measuring the differences between features across different events. However, this is a technically challenging problem. First, fake news content is characterized by incomplete and large-scale data processing. Second, after data processing, because learning the feature representations of posts is quite complex, loss estimation methods such as squared error may fail to estimate the differences in these complex feature representations. Third, the features of fake news change continuously during training, requiring the proposed model to capture these changes and consistently provide accurate measurements. Therefore, effectively estimating the differences between features learned from different events is a prerequisite for removing event-specific features, and is particularly important for fake news detection. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal fake news detection method and system to improve the accuracy of fake news detection.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A multimodal fake news detection method includes:

[0008] Construct a multimodal feature extractor for extracting multimodal features from news posts;

[0009] Build an event classifier for categorizing news post events;

[0010] Establish a fake news detector to predict the authenticity of news based on the multimodal features extracted from news posts;

[0011] An ensemble model is obtained by integrating a multimodal feature extractor, an event classifier, and a fake news detector.

[0012] Train the ensemble model;

[0013] The news post to be detected is input into the trained ensemble model, which outputs the prediction result of the news; the prediction result is either true or false.

[0014] Optionally, the multimodal feature extractor includes: a text feature extractor, a visual feature extractor, and an output layer;

[0015] The text feature extractor includes a word2vec embedding layer, a text-CNN convolutional layer, a MaxPooling extraction layer, and a first fully connected layer connected in sequence.

[0016] The word2vec embedding layer is used to initialize each word in the text of a news post as a word embedding vector using pre-trained word2vec;

[0017] The text-CNN convolutional layer is used to obtain the n-gram feature vector representation of each sentence in the text through one-dimensional convolution based on the word embedding vector;

[0018] The Max Pooling extraction layer is used to extract text features with different granularities from the n-gram feature vector representation of each sentence using filters of different window sizes;

[0019] The first fully connected layer is used to represent all text features with different granularities as text features of a preset dimension;

[0020] The visual feature extractor includes: a ResNet50 neural network extraction layer and a second fully connected layer; the ResNet50 neural network extraction layer is connected to the second fully connected layer.

[0021] The ResNet50 neural network extraction layer is used to extract visual features from images in news posts;

[0022] The second fully connected layer is used to represent the extracted visual features as visual features of the preset dimension;

[0023] The output layer is connected to the first fully connected layer and the second fully connected layer respectively. The output layer is used to connect text features of a preset dimension and visual features of a preset dimension to form multimodal features and output the multimodal features.

[0024] Optionally, the event classifier is represented as G. e (R F ;θ e ); where R F Represents multimodal features, θ e Indicates the parameters of the event classifier;

[0025] The loss function of the event classifier is: Among them, L e (θ f θ e ) represents the loss of the event classifier, θ f This represents the parameters that the multimodal feature extractor needs to learn, where M represents a set of text and visual articles. The cross-entropy loss function represents the loss function that categorizes a set of text and images into event labels after input. [k＝y] This means that all k fall within Y. eThe probability of being in the middle is 1, m represents a set of text and visual posts, y represents the event tag, k represents the variable that can be divided into events k, K represents the K events that are divided, Y e G represents a collection of event labels. f (m, θ) f ) represents a multimodal feature extractor.

[0026] Optionally, the fake news detector includes: a fully connected layer with softmax; the fully connected layer is used to output a prediction result of the news post based on the multimodal features extracted from the news post;

[0027] The loss function of the fake news detector is: Among them, L d (θ f θ d This indicates the loss of the fake news detector. Y represents the cross-entropy loss function that categorizes a set of text and images into news tags. d P represents a set of tags for news. θ (m) represents the probability of a fake news post, θ d This represents the parameters of the fake news detector.

[0028] Optionally, the integrated multimodal feature extractor, event classifier, and fake news detector specifically include:

[0029] Connect the multimodal feature extractor and the event classifier, and the multimodal feature extractor and the fake news detector respectively;

[0030] Using formula L sum (θ f ,θ d ,θ e ) = L d (θ f ,θ d )-λL e (θ f ,θ e The loss of the event classifier and the loss of the fake news detector are integrated into the final loss; where L sum (θ f θ d θ e ) represents the final loss, and λ represents the weights of the loss function of the event classifier.

[0031] Optionally, training the ensemble model specifically includes:

[0032] The saddle point of the final objective function of the ensemble model is determined as follows: and in, This represents the optimal parameters that the multimodal feature extractor needs to learn. The optimal parameters for a fake news detector are represented. This represents the optimal parameters for the event classifier. This represents the optimal parameters under the interaction of the event classifier and the fake news detector. The corresponding final loss function, This indicates the optimal parameters under the interaction between the event classifier and the fake news detector. The corresponding final loss function;

[0033] The preset decay learning rate is Where η′ represents the decaying learning rate, η represents the initial learning rate, and α and β represent the first and second hyperparameters, respectively;

[0034] The learning rate is decayed, and the learning set is trained using news posts. Stochastic gradient descent is then employed to find the optimal learning rate.

[0035] A multimodal fake news detection system includes:

[0036] The Multimodal Feature Extractor Building Module is used to build a multimodal feature extractor for extracting multimodal features from news posts.

[0037] The event classifier building module is used to build event classifiers for categorizing news post events;

[0038] The fake news detector building module is used to build a fake news detector that predicts the authenticity of news based on the multimodal features of extracted news posts.

[0039] An integration module is used to integrate a multimodal feature extractor, an event classifier, and a fake news detector to obtain an integrated model;

[0040] The training module is used to train the ensemble model;

[0041] The prediction module is used to input the news post to be detected into the trained ensemble model and output the prediction result of the news; the prediction result is either true or false.

[0042] Optionally, the multimodal feature extractor includes: a text feature extractor, a visual feature extractor, and an output layer;

[0043] The text feature extractor includes a word2vec embedding layer, a text-CNN convolutional layer, a MaxPooling extraction layer, and a first fully connected layer connected in sequence.

[0044] The word2vec embedding layer is used to initialize each word in the text of a news post as a word embedding vector using pre-trained word2vec;

[0045] The text-CNN convolutional layer is used to obtain the n-gram feature vector representation of each sentence in the text through one-dimensional convolution based on the word embedding vector;

[0046] The Max Pooling extraction layer is used to extract text features with different granularities from the n-gram feature vector representation of each sentence using filters of different window sizes;

[0047] The first fully connected layer is used to represent all text features with different granularities as text features of a preset dimension;

[0048] The visual feature extractor includes: a ResNet50 neural network extraction layer and a second fully connected layer; the ResNet50 neural network extraction layer is connected to the second fully connected layer.

[0049] The ResNet50 neural network extraction layer is used to extract visual features from images in news posts;

[0050] The second fully connected layer is used to represent the extracted visual features as visual features of the preset dimension;

[0051] The output layer is connected to the first fully connected layer and the second fully connected layer respectively. The output layer is used to connect text features of a preset dimension and visual features of a preset dimension to form multimodal features and output the multimodal features.

[0052] Optionally, the event classifier is represented as G. e (R F ;θ e ); where R F Represents multimodal features, θ e Indicates the parameters of the event classifier;

[0053] The loss function of the event classifier is: Among them, L e (θ f θ e ) represents the loss of the event classifier, θ f This represents the parameters that the multimodal feature extractor needs to learn, where M represents a set of text and visual articles. The cross-entropy loss function represents the process of categorizing a set of text and images into event labels. [k＝y] This means that all k fall within Y. e The probability of being in the middle is 1, m represents a set of text and visual posts, y represents the event tag, k represents the variable that can be divided into events k, K represents the K events that are divided, Y e G represents a collection of event labels. f (m, θ) f) represents a multimodal feature extractor.

[0054] Optionally, the fake news detector includes: a fully connected layer with softmax; the fully connected layer is used to output a prediction result of the news post based on the multimodal features extracted from the news post;

[0055] The loss function of the fake news detector is: Among them, L d (θ f θ d This indicates the loss of the fake news detector. Y represents the cross-entropy loss function that categorizes a set of text and images into news tags. d P represents a set of tags for news. θ (m) represents the probability of a fake news post, θ d This represents the parameters of the fake news detector.

[0056] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0057] This invention discloses a multimodal fake news detection method and system. Based on the concept of resistant networks, an event classifier is added to the multimodal feature extractor and fake news detector during the training phase to predict auxiliary labels for events. The corresponding loss is used to estimate the differences in feature representations between different events. The greater the loss, the lower the difference. At the same time, the multimodal feature extractor attempts to deceive the event classifier to learn the common features of different events, thus enriching the types of extracted features. In this way, the trained multimodal feature extractor and the fake news detector work together to complete the detection and identification of fake news, thereby improving the accuracy of fake news detection. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart illustrating a multimodal fake news detection method provided in this embodiment of the invention;

[0060] Figure 2 A simplified schematic diagram of a multimodal fake news detection method provided in an embodiment of the present invention;

[0061] Figure 3 An integrated model structure diagram provided for embodiments of the present invention;

[0062] Figure 4 This is a comparison chart of the accuracy of fake news detection provided in an embodiment of the present invention;

[0063] Figure 5 A comparison chart of F1 scores for fake news detection provided in an embodiment of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] The purpose of this invention is to provide a multimodal fake news detection method and system to improve the accuracy of fake news detection.

[0066] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] This invention provides a multimodal fake news detection method, such as... Figure 1-2 As shown, it includes:

[0068] Step S1: Construct a multimodal feature extractor for extracting multimodal features from news posts.

[0069] Step 1.1 utilizes a variant of convolutional neural networks, text-CNN, as the core module of the text extractor. Text-CNN obtains the n-gram feature representation of a sentence through one-dimensional convolution. For the detailed steps of the text feature extractor, the embedding layer is initialized using a pre-trained word2vec. For words not present in the pre-trained word2vec, initialization is performed randomly. Then, the entire network is fixed, and each word in the text is represented as a word embedding vector. For the k-th word in the sentence, the corresponding j-dimensional word embedding vector is represented as A. k ∈R j Therefore, a sentence with n words can be represented as:

[0070]

[0071] in, This is the concatenation operator. A convolutional filter with a window size of h takes a continuous sequence of h words in a sentence as input and outputs a feature. Taking a continuous sequence of h words starting with the i-th word as an example, the filter operation can be represented as:

[0072] ai =σ(W c ·A i:i+h-1 (2)

[0073] Where σ(·) is the ReLU activation function, W c This represents the filter weights. Filters of different sizes can generate feature vectors of different sizes. A filter of the same size can also be applied to other words, resulting in a feature vector for the sentence: a = [a1, a2, ..., a...]. n-h+1 ].

[0074] For each feature vector, K-Max Pooling is used to extract the top-k scores from all feature values, preserving their original order to extract the most important information and obtain the corresponding features for a specific filter. This process is repeated until the features for all filters are obtained. To extract text features with different granularities, different window sizes are applied. For a specific window size, we have n h There are c different filters. Therefore, assuming there are c possible window sizes, we have a total of c·n. h A filter. The text properties after the Max Pooling operation are written as... Following the MaxPooling operation, a fully connected layer is used to ensure the final text feature representation (denoted as R). A ∈R p It has the same dimension as the visual feature representation (denoted as p):

[0075]

[0076] Among them, W af This is the weight matrix of the fully connected layer.

[0077] Step 1.2 For the image information attached to the news post, a pre-trained ResNet50 neural network is used to extract visual features. The p-dimensional visual features are represented as R. v ∈R p The operation of the last layer in the visual feature extractor can be represented as:

[0078]

[0079] in, W represents the visual features obtained from the pre-trained ResNet50. vf This is the weight matrix of the fully connected layer in the visual feature extractor.

[0080] Text feature representation R A and visual feature representation RA Connections form multimodal feature representations This is the output of the multimodal feature extractor. We denote the multimodal feature extractor as G. f (M;θ f ), where M is typically a set of text and visual articles, which is the input to the multimodal feature extractor, and θ f This indicates the parameters that need to be learned.

[0081] Step S2: Construct an event classifier for categorizing news post events.

[0082] The event classifier is represented as G. e (R F ;θ e ),in θ e This represents its parameters. Similarly, cross-entropy is used to define the loss of the event classifier, and Y is used... e To represent a collection of event labels:

[0083]

[0084] Event classifier minimum loss L e (θ f ,θ e The parameter of ) is written as:

[0085]

[0086] The above losses It can be used to estimate the differences in the distributions of different events. A larger loss means that the distributions representing different events are more similar, and the learned features are the general common features of the events. Therefore, to eliminate the uniqueness of each event's features, we need to find the optimal parameter θ. f To maximize the identification loss The design philosophy of the above model inspires a minimax game between the multimodal feature extractor and the event classifier. On the one hand, the event classifier aims to discover event-specific feature information contained in the feature representation to identify and classify events. On the other hand, the multimodal feature extractor attempts to deceive the event classifier to maximize the recognition loss in order to obtain common features of different events.

[0087] Step S3: Establish a fake news detector to predict the authenticity of news based on the multimodal features of extracted news posts.

[0088] We utilize fully connected layers with softmax to predict the authenticity of these posts. Softmax is a generalization of the binary classification function sigmoid to multi-class classification, aiming to represent the results of multi-class classification as probabilities. Furthermore, we use a dropout operation in the prediction to prevent overfitting. The fake news detector in this model is built upon a multimodal feature extractor, thus representing R with multimodal features. F As input, we denote the fake news detector as G. d (R F ;θ d ), where θ d This represents all parameters included. The output of the fake news detector for the i-th multimedia post is denoted as m. i What is the probability that this post is fake?

[0089] P θ (m i ) = G d (G f (m i ;θ f );θ d (7)

[0090] Then, cross-entropy is used to calculate the detection loss:

[0091]

[0092] Among them, Y d A set of tags used to represent news.

[0093] By finding the optimal parameters and To minimize the detection loss function L d (θ f ,θ d This process can be represented as:

[0094]

[0095] Therefore, in order to capture the common features among all news events, the model needs to be able to learn feature representations of more general events. To achieve this goal, the specific features of each event are removed, and the differences in feature representations across different events are measured and removed to obtain a common feature representation that is invariant to the events.

[0096] Step S4: Integrate the multimodal feature extractor, event classifier, and fake news detector to obtain the integrated model.

[0097] Social media posts and news content typically contain various types of information. The role of a multimodal feature extractor is to handle these different types of input. After learning the latent feature representations of text and images, it concatenates them to form a multimodal feature representation, presented in matrix form. An event classifier then identifies the type of event represented by the content based on the output value of the multimodal feature extractor. Building upon this, a fake news detector and an event classifier detect fake news and classify events. The fake news detector takes the learned feature representation matrix as input and predicts whether the news is true or false.

[0098] Integrated model structure such as Figure 3 As shown, Figure 3 The words in the top left corner box are words from the news post. text-fc represents the fully connected layer (first connected layer) of the text, vis-fc represents the fully connected layer (second connected layer) of the image, reversal represents the gradient reversal layer, adv-fc1 represents the first fully connected layer of the event classifier, and adv-fc2 represents the second fully connected layer of the event classifier.

[0099] Step S5: Train the ensemble model.

[0100] The multimodal feature extractor, event classifier, and fake news detector are integrated and trained to improve the multimodal feature extractor G. f (M;θ f ) Requires the use of the fake news detector G d (R F ;θ d (To cooperate and minimize detection losses) d (θ f ,θ d This improves the performance of fake news detection tasks. Meanwhile, the multimodal feature extractor G... f (M;θ f Attempting to maximize event identification loss L e (θ f ,θ e To deceive the event classifier To obtain a common representation of events. Event classifier G e (R F ;θ e This attempt aims to identify each event by minimizing the event classification loss based on multimodal feature representations. The final loss is defined as:

[0101] L sum (θ f ,θ d ,θ e ) = L d (θ f ,θd )-λL e (θ f ,θ e (10)

[0102] Here, λ controls the weights of the loss functions for the fake news detector and event classifier modules. For ease of design, λ is simply set to 1.

[0103] For the minima strategy, the set of optimal parameters sought is the saddle point of the final objective function:

[0104]

[0105]

[0106] Next, we use stochastic gradient descent to find the optimal parameter set. Since the gradient inversion layer performs the identity transformation during forward propagation, it is used. During backward propagation, the gradient direction is automatically inverted, as shown below:

[0107]

[0108] R λ (x)=x (14)

[0109]

[0110] To stabilize the training process, the learning rate η is decayed using the following formula:

[0111]

[0112] Where η is the initial learning rate, with a value of 0.01, α and β are hyperparameters, set to α = 10 and β = 0.75, and p represents the relative value of the iteration process, that is, the ratio of the current iteration number to the total number of iterations, which changes linearly from 0 to 1.

[0113] Step S6: Input the news post to be detected into the trained ensemble model and output the prediction result of the news; the prediction result is true or false.

[0114] This invention further verifies the effectiveness of the Event Categorizer Neural Network (ECNN) model by completing a fake news detection task and analyzing its detection performance. Experiments are presented to evaluate the effectiveness of the ECNN model (ensemble model), using two large social media datasets for fake news detection. The performance of the fake news detection algorithm is evaluated through accuracy, precision, recall, and the F1 score.

[0115] The dataset used for fake news detection in this invention is shown in Table 1:

[0116] Table 1 Statistics of Real-World Datasets

[0117] platform Twitter Weibo #oftruenews 5723 4849 #offakenews 6232 4865 #ofimages 682 9714

[0118] The performance comparison chart of the fake news detection of this invention is shown below. Figure 4 and Figure 5 As shown in Table 1 and the results graph, the accuracy and F1 score of text-CNN+ and Visual+ are significantly higher than those of text-CNN and Visual+. Therefore, including an event classifier module in the fake news detection task is necessary to effectively capture transferable features between different events. Furthermore, the event classifier neural network model framework of this invention enables semi-automatic construction of the fake news detector, assisting engineers in performing fake news detection faster and more accurately.

[0119] The performance comparison results of this invention in detecting fake news are shown in Table 2:

[0120] Table 2 Comparison of Fake News Detection Performance Indicators

[0121]

[0122]

[0123] As shown in Table 2 above, the neural network model based on the event classifier outperforms the baseline in terms of accuracy, precision, and F1 score. On the Twitter dataset, the number of tweets related to different events is unevenly distributed, with over 70% of tweets relating to a single event. This results in the learned text features being primarily concentrated on specific events, making it difficult for the model to learn common features across different events. Compared to the visual modality, the text modality contains more obvious event-specific features, allowing for the extraction of event features through keywords or thematic terms in the text, which is difficult to achieve by simply extracting image features. This situation severely hinders the text model from extracting transferable features from different events, making transfer learning difficult. Therefore, the performance of the text-only modality model is the lowest among all methods. Compared to the visual single-modality baseline Visual, its performance is better than the text-based model because it is difficult to classify images into specific events based solely on image information. Therefore, the features learned by the model are often more transferable, which is beneficial for transfer learning. However, a problem exists: the dataset contains relatively few accompanying image information for news, resulting in less than ideal model performance. Although the visual modality is effective for feature extraction in fake news detection, Visual's performance is still lower than that of multimodal methods. In multimodal models, att-RNN outperforms NeuralTalk and TCNN-URG, indicating that applying attention mechanisms can help models extract key features more effectively and improve the performance of prediction models.

[0124] The given variant of ECNN removes the event classifier module, thus tending to capture event-specific features. This results in insufficient transferable features being learned between events. In contrast, with the help of the event classifier, the full model improves accuracy by 1.4% and precision by 1.2%, while maintaining performance close to the best baseline in F1 score, demonstrating the effectiveness of the event classifier in improving performance.

[0125] On the Weibo dataset, we can observe that for single-modal methods, relying solely on text for fake news detection significantly outperforms relying solely on images, contrary to previous experimental results. This is due to two main reasons: firstly, the Weibo dataset does not suffer from data imbalance compared to the Twitter dataset; the former possesses diverse textual data from which useful language patterns can be extracted for fake news detection. Secondly, the images in the Weibo dataset are semantically much more complex than those in the Twitter dataset. For such a challenging image dataset, the baseline Visual model fails to learn meaningful features effectively. It can be seen that the event classifier-based neural network model outperforms all multimodal methods on this dataset. Component analysis of the model shows that combining multimodal data helps improve the model's fake news detection performance; furthermore, modeling both textual content and visual images is necessary because they contain complementary information and can also address the data imbalance problem.

[0126] The method of this invention mainly comprises three components: a multimodal feature extractor, an event classifier, and a fake news detector. The multimodal feature extractor and the fake news detector work together to detect and identify fake news. Simultaneously, the multimodal feature extractor attempts to deceive the event classifier to learn common features of different events, enriching the types of extracted features. Based on the concept of resistive networks, an event classifier is added during the training phase to predict auxiliary event labels, and the corresponding loss is used to estimate the differences in feature representations between different events. The greater the loss, the lower the difference, thereby improving the performance of fake news detection. This invention proposes a minimax game framework between the multimodal feature extractor and the event classifier based on a neural network model of the event classifier. Specifically, the multimodal feature extractor deceives the event classifier to learn common features that can represent different events. In this way, it effectively eliminates the tight dependence of data on specific events and exhibits better transferability to newly emerging events.

[0127] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0128] 1. This invention provides a multimodal fake news detection method based on an event-triggered neural network. The method mainly comprises three components: a multimodal feature extractor, an event classifier, and a fake news detector. The multimodal feature extractor and the fake news detector work together to detect and identify fake news. Simultaneously, the multimodal feature extractor attempts to deceive the event classifier into learning the common features of different events, thus enriching the types of extracted features.

[0129] 2. Based on the concept of resistance networks, this invention incorporates an event classifier during the training phase to predict auxiliary event labels and uses the corresponding loss to estimate the differences in feature representations between different events. The greater the loss, the lower the difference, thereby improving the performance of fake news detection.

[0130] 3. The present invention proposes a minimax game framework between a multimodal feature extractor and an event classifier based on a neural network model. Specifically, the multimodal feature extractor tricks the event classifier into learning common features that represent different events. In this way, it effectively eliminates the tight dependence of data on specific events and exhibits better transferability to newly emerging events.

[0131] This invention also provides a multimodal fake news detection system, comprising:

[0132] The Multimodal Feature Extractor Building Module is used to build a multimodal feature extractor for extracting multimodal features from news posts.

[0133] The event classifier building module is used to build event classifiers for categorizing news post events;

[0134] The fake news detector building module is used to build a fake news detector that predicts the authenticity of news based on the multimodal features of extracted news posts.

[0135] An integration module is used to integrate a multimodal feature extractor, an event classifier, and a fake news detector to obtain an integrated model;

[0136] The training module is used to train the ensemble model;

[0137] The prediction module is used to input the news post to be detected into the trained ensemble model and output the prediction result of the news; the prediction result is true or false.

[0138] The multimodal feature extractor includes: a text feature extractor, a visual feature extractor, and an output layer;

[0139] The text feature extractor consists of a word2vec embedding layer, a text-CNN convolutional layer, a Max Pooling extraction layer, and a first fully connected layer connected in sequence.

[0140] The word2vec embedding layer is used to initialize each word in the text of a news post as a word embedding vector using a pre-trained word2vec.

[0141] The text-CNN convolutional layer is used to obtain the n-gram feature vector representation of each sentence in the text through one-dimensional convolution based on the word embedding vectors;

[0142] The Max Pooling extraction layer is used to extract text features with different granularities from the n-gram feature vector representation of each sentence using filters of different window sizes;

[0143] The first fully connected layer is used to represent all text features with different granularities as text features of a preset dimension;

[0144] The visual feature extractor includes: a ResNet50 neural network extraction layer and a second fully connected layer; the ResNet50 neural network extraction layer is connected to the second fully connected layer.

[0145] The resnet50 neural network extraction layer is used to extract visual features from images in news posts;

[0146] The second fully connected layer is used to represent the extracted visual features as visual features of a preset dimension;

[0147] The output layer is connected to the first fully connected layer and the second fully connected layer respectively. The output layer is used to connect the text features of the preset dimension and the visual features of the preset dimension to form multimodal features and output the multimodal features.

[0148] The event classifier is represented as G. e (R F ;θ e ); where R F Represents multimodal features, θ e Indicates the parameters of the event classifier;

[0149] The loss function of the event classifier is: Among them, L e (θ f θ e ) represents the loss of the event classifier, θ f This represents the parameters that the multimodal feature extractor needs to learn, where M represents a set of text and visual articles. The cross-entropy loss function represents the process of categorizing a set of text and images into event labels. [k＝y] This means that all k fall within Y. e The probability of being in the middle is 1, m represents a set of text and visual posts, y represents the event tag, k represents the variable that can be divided into events k, K represents the K events that are divided, Y e G represents a collection of event labels. f (m, θ) f ) represents a multimodal feature extractor, and Ge() represents an event classifier representation function.

[0150] The fake news detector includes: a fully connected layer with softmax; the fully connected layer is used to output a prediction of the news post based on the multimodal features extracted from the news post;

[0151] The loss function of the fake news detector is: Among them, L d (θ f θ d This indicates the loss of the fake news detector. Y represents the cross-entropy loss function that categorizes a set of text and images into news tags. d P represents a set of tags for news. θ (m) represents the probability of a fake news post, θ d This represents the parameters of the fake news detector.

[0152] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0153] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal fake news detection method, characterized in that, include: Construct a multimodal feature extractor for extracting multimodal features from news posts; Construct an event classifier for categorizing news post events; the event classifier is represented as... ;in, Represents multimodal features, These represent the parameters of the event classifier; the loss function of the event classifier is: ;in, This represents the loss of the event classifier. This represents the parameters that the multimodal feature extractor needs to learn. This represents a collection of posts containing a set of text and images. This represents the cross-entropy loss function used to categorize posts containing a set of text and images into event tags. Indicates in equal Under this condition, all All fell The probability of winning is 1. This indicates a post that contains a set of text and images. Labels representing events, This indicates that events can be divided into categories. variables, This indicates that news posts are divided into categories. One event, A collection representing event labels. This represents a multimodal feature extractor; Establish a fake news detector to predict the authenticity of news based on the multimodal features extracted from news posts; An ensemble model is obtained by integrating a multimodal feature extractor, an event classifier, and a fake news detector. Train the ensemble model; The news post to be detected is input into the trained ensemble model, which outputs the prediction result of the news; the prediction result is either true or false.

2. The multimodal fake news detection method according to claim 1, characterized in that, The multimodal feature extractor includes: a text feature extractor, a visual feature extractor, and an output layer; The text feature extractor includes a word2vec embedding layer, a text-CNN convolutional layer, a Max Pooling extraction layer, and a first fully connected layer connected in sequence. The word2vec embedding layer is used to initialize each word in the text of a news post as a word embedding vector using pre-trained word2vec; The text-CNN convolutional layer is used to obtain the n-gram feature vector representation of each sentence in the text through one-dimensional convolution based on the word embedding vector; The Max Pooling extraction layer is used to extract text features with different granularities from the n-gram feature vector representation of each sentence using filters of different window sizes; The first fully connected layer is used to represent all text features with different granularities as text features of a preset dimension; The visual feature extractor includes: a ResNet-50 model and a second fully connected layer; the ResNet-50 model is connected to the second fully connected layer. The ResNet-50 model is used to extract visual features from images in news posts; The second fully connected layer is used to represent the extracted visual features as visual features of the preset dimension; The output layer is connected to the first fully connected layer and the second fully connected layer respectively. The output layer is used to connect text features of a preset dimension and visual features of a preset dimension to form multimodal features and output the multimodal features.

3. The multimodal fake news detection method according to claim 2, characterized in that, The fake news detector includes: a fully connected layer with softmax; the fully connected layer is used to output a prediction result of the news post based on the multimodal features extracted from the news post; The loss function of the fake news detector is: ;in, This indicates the loss of the fake news detector. The cross-entropy loss function represents the loss function used to categorize posts containing a set of text and images into news tags. A set of tags representing news. This indicates the probability of a fake news post. This represents the parameters of the fake news detector.

4. The multimodal fake news detection method according to claim 3, characterized in that, The integrated multimodal feature extractor, event classifier, and fake news detector specifically include: Connect the multimodal feature extractor and the event classifier, and the multimodal feature extractor and the fake news detector, respectively; Using formula The loss from the event classifier and the loss from the fake news detector are integrated into the final loss; whereby, Indicates the final loss. The weights represent the loss function of the event classifier.

5. The multimodal fake news detection method according to claim 4, characterized in that, Training the ensemble model specifically includes: The saddle point of the final objective function of the ensemble model is determined as follows: and ;in, This represents the optimal parameters that the multimodal feature extractor needs to learn. The optimal parameters for a fake news detector are represented. This represents the optimal parameters for the event classifier. This indicates the optimal parameters under the interaction of the event classifier and the fake news detector. The corresponding final loss function, This indicates the optimal parameters under the interaction between the event classifier and the fake news detector. The corresponding final loss functions are designed to minimize the loss in the former case and maximize the loss in the latter case, resulting in a game-like interaction. The preset decay learning rate is ;in, This represents the decaying learning rate. This represents the initial learning rate. and These represent the first and second hyperparameters, respectively. p Represents the relative value of the iteration process; The learning rate is decayed, and the learning set is trained using news posts. Stochastic gradient descent is then employed to find the optimal learning rate.

6. A multimodal fake news detection system, characterized in that, include: The Multimodal Feature Extractor Building Module is used to build a multimodal feature extractor for extracting multimodal features from news posts. The event classifier building module is used to construct an event classifier for categorizing news post events; the event classifier is represented as... ;in, Represents multimodal features, These represent the parameters of the event classifier; the loss function of the event classifier is: ;in, This represents the loss of the event classifier. This represents the parameters that the multimodal feature extractor needs to learn. This represents a collection of posts containing a set of text and images. This represents the cross-entropy loss function used to categorize posts containing a set of text and images into event tags. Indicates in equal Under this condition, all All fell The probability of winning is 1. This indicates a post that contains a set of text and images. Labels representing events, This indicates that events can be divided into categories. variables, This indicates that news posts are divided into categories. One event, A collection representing event labels. This represents a multimodal feature extractor; The fake news detector building module is used to build a fake news detector that predicts the authenticity of news based on the multimodal features of extracted news posts. An integration module is used to integrate a multimodal feature extractor, an event classifier, and a fake news detector to obtain an integrated model; The training module is used to train the ensemble model; The prediction module is used to input the news post to be detected into the trained ensemble model and output the prediction result of the news; the prediction result is either true or false.

7. The multimodal fake news detection system according to claim 6, characterized in that, The multimodal feature extractor includes: a text feature extractor, a visual feature extractor, and an output layer; The text feature extractor includes a word2vec embedding layer, a text-CNN convolutional layer, a Max Pooling extraction layer, and a first fully connected layer connected in sequence. The word2vec embedding layer is used to initialize each word in the text of a news post as a word embedding vector using pre-trained word2vec; The text-CNN convolutional layer is used to obtain the n-gram feature vector representation of each sentence in the text through one-dimensional convolution based on the word embedding vector; The Max Pooling extraction layer is used to extract text features with different granularities from the n-gram feature vector representation of each sentence using filters of different window sizes; The first fully connected layer is used to represent all text features with different granularities as text features of a preset dimension; The visual feature extractor includes: a ResNet-50 model and a second fully connected layer; the ResNet-50 model is connected to the second fully connected layer. The ResNet-50 model is used to extract visual features from images in news posts; The second fully connected layer is used to represent the extracted visual features as visual features of the preset dimension; The output layer is connected to the first fully connected layer and the second fully connected layer respectively. The output layer is used to connect text features of a preset dimension and visual features of a preset dimension to form multimodal features and output the multimodal features.

8. The multimodal fake news detection system according to claim 7, characterized in that, The fake news detector includes: a fully connected layer with softmax; the fully connected layer is used to output a prediction result of the news post based on the multimodal features extracted from the news post; The loss function of the fake news detector is: ;in, This indicates the loss of the fake news detector. The cross-entropy loss function represents the loss function used to categorize posts containing a set of text and images into news tags. A set of tags representing news. This indicates the probability of a fake news post. This represents the parameters of the fake news detector.

Images