A false information detection model training method and false information detection method

Abstract

The present invention discloses a false information detection model training method and a false information detection method, wherein the false information detection model training method comprises: obtaining training information, the training information comprises text and image; inputting the text and image into respective encoders respectively to obtain text representation and image representation; interacting the text representation and image representation to determine text-oriented multimodal features and vision-oriented multimodal features; inputting the text-oriented multimodal features and vision-oriented multimodal features into respective classifiers respectively to obtain text-oriented loss values ​​and vision-oriented loss values; determining the objective function of the false information detection model according to the text-oriented loss values ​​and vision-oriented loss values, and training the false information detection model according to the objective function. The present invention improves the accuracy and efficiency of false information detection.

Description

Technical Field

[0001] The invention discloses a false information detection model training method and a false information detection method, belonging to the technical field of false information detection. Background Art

[0002] Online social media has become an indispensable platform for people to share and obtain information in their daily lives. Due to the loose restrictions on user-generated content on online social media, in addition to real news, there may also be a large amount of false news that distorts and fabricates facts on online social media. The widespread dissemination of false news may mislead readers and even cause serious social consequences. Therefore, it is particularly urgent and important to automatically detect false news on social media.

[0003] Early research on fake news detection focused on the text modality of user posts. Although it can identify false information, it ignores the fact that a large amount of news involves images. On the one hand, images may contain key clues for fake news detection. Figure 1 As shown in (a) of Figure 1, it is difficult to determine the authenticity of the news based on the text alone, but by considering the visual image of the tweet, it can be confirmed that the news is false. On the other hand, even if the image itself does not contain irregular clues, it can enhance the judgment of the authenticity of the news. Figure 1 As shown in (b), the text suggests that it might be fake, while the image seems normal. However, if the text and image are considered jointly, it can be easily inferred that it is fake news.

[0004] Therefore, in recent years, people have made a lot of efforts in multimodal fake news detection. The existing technology of multimodal fake news detection mainly focuses on designing more advanced feature extraction procedures for multimodality, and realizes it through serial operation when multimodal fusion is performed according to the extracted features. However, the existing technology fails to model the interaction between the two modalities, resulting in low detection accuracy and poor detection efficiency when using it for fake news detection. Summary of the invention

[0005] The purpose of this application is to provide a false information detection model training method and a false information detection method to solve the technical problems of low detection accuracy and poor detection efficiency in existing false information detection methods.

[0006] A first aspect of the present invention provides a false information detection model training method, comprising:

[0007] Acquire training information, wherein the training information includes text and images related to the text;

[0008] Input text and image into their respective encoders to obtain text representation and image representation;

[0009] Interact text representation with image representation to determine text-oriented multimodal features and vision-oriented multimodal features;

[0010] Input the text-oriented multimodal features and the vision-oriented multimodal features into their respective classifiers to obtain the text-oriented loss value and the vision-oriented loss value;

[0011] An objective function of a false information detection model is determined according to a text-oriented loss value and a vision-oriented loss value, and the false information detection model is trained according to the objective function.

[0012] Preferably, the text and the image are input into respective encoders to obtain text representation and image representation, which specifically includes:

[0013] Obtaining text embedding information, and inputting the text embedding information into a text encoder to obtain text representation;

[0014] Obtaining embedded information of an image, and inputting the embedded information of the image into an image encoder to obtain an image representation;

[0015] The text encoder and the image encoder are both Transformer encoders;

[0016] The text representation and the image representation both include a query vector, a key vector and a value vector.

[0017] Preferably, the text representation and the image representation are interacted to determine the text-oriented multimodal features and the vision-oriented multimodal features, which specifically include:

[0018] Inputting the query vector in the text representation and the key vector and the value vector in the image representation into a text-oriented multimodal fusion block to obtain text-oriented multimodal features;

[0019] The query vector in the image representation and the key vector and the value vector in the text representation are input into a vision-oriented multimodal fusion block to obtain vision-oriented multimodal features.

[0020] Preferably, the text-oriented multimodal fusion block includes a first modal interaction unit and a first key information selection unit;

[0021] The query vector in the text representation and the key vector and value vector in the image representation are input into a text-oriented multimodal fusion block to obtain text-oriented multimodal features, specifically including:

[0022] Inputting the query vector in the text representation and the key vector and the value vector in the image representation into a first modality interaction unit to obtain preliminary multimodal fusion features;

[0023] Inputting the preliminary multimodal fusion features into the first key information selection unit to obtain text-oriented multimodal features;

[0024] The mechanism used in the first modality interaction unit is a collaborative attention mechanism;

[0025] The mechanism used by the first key information selection unit is the self-attention mechanism.

[0026] Preferably, the vision-oriented multimodal fusion block includes a second modality interaction unit and a second key information selection unit;

[0027] Inputting the query vector in the image representation and the key vector and value vector in the text representation into a vision-oriented multimodal fusion block to obtain vision-oriented multimodal features, specifically including:

[0028] Inputting the query vector in the image representation and the key vector and the value vector in the text representation into a second modality interaction unit to obtain preliminary multimodal fusion features;

[0029] Input the preliminary multimodal fusion features into the second key information selection unit to obtain vision-oriented multimodal features;

[0030] The mechanism used in the second modality interaction unit is a collaborative attention mechanism;

[0031] The mechanism used by the second key information selection unit is the self-attention mechanism.

[0032] Preferably, the text-oriented multimodal features and the vision-oriented multimodal features are respectively input into respective classifiers to obtain text-oriented loss values ​​and vision-oriented loss values, which specifically include:

[0033] Inputting the text-oriented multimodal features into a text-oriented classifier to obtain a first prediction probability of the training information, and determining a text-oriented loss value according to the first prediction probability and the true value of the training information;

[0034] The vision-oriented multimodal features are input into a vision-oriented classifier to obtain a second prediction probability of the training information, and the vision-oriented loss value is determined according to the second prediction probability and the true value of the training information.

[0035] Preferably, the objective function of the false information detection model is determined according to the text-oriented loss value and the vision-oriented loss value, specifically including:

[0036] Determine, according to the first prediction probability and the second prediction probability, a first interaction loss value from the text-oriented classifier to the vision-oriented classifier and a second interaction loss value from the vision-oriented classifier to the text-oriented classifier;

[0037] The objective function of the false information detection model is determined according to the text-oriented loss value, the vision-oriented loss value, the first interaction loss value and the second interaction loss value.

[0038] Preferably, determining a first interaction loss value from the text-oriented classifier to the vision-oriented classifier and a second interaction loss value from the vision-oriented classifier to the text-oriented classifier according to the first prediction probability and the second prediction probability specifically includes:

[0039] According to the first prediction probability and the second prediction probability, a first interaction loss value from the text-oriented classifier to the vision-oriented classifier and a second interaction loss value from the vision-oriented classifier to the text-oriented classifier are determined using information divergence.

[0040] Preferably, the objective function of the false information detection model is determined according to the text-oriented loss value, the vision-oriented loss value, the first interaction loss value, and the second interaction loss value, specifically including:

[0041] The objective function of the false information detection model is determined using the first formula, where the first formula is:

[0042]

[0043] In the formula, is the text-oriented loss value, is the visual loss value, is the first interaction loss value, is the second interaction loss value, λ KL is the weight of mutual learning loss.

[0044] A second aspect of the present invention provides a false information detection method, comprising:

[0045] Obtain the information to be tested;

[0046] The information to be detected is input into the false information detection model trained by the false information detection model training method to obtain a false information detection result.

[0047] Compared with the prior art, the false information detection model training method and false information detection method of the present invention have the following beneficial effects:

[0048] The present invention is a multimodal fusion method and system based on a mutual learning network, wherein the multimodal fusion block can jointly capture the interaction between multimodal features and extract key information for false news detection. On this basis, the mutual learning strategy is further combined to enable information transmission between text-oriented classifiers and vision-oriented classifiers, thereby promoting multimodal fusion to enhance the efficiency and accuracy of false news detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 (a) and (b) are fake news on online social media;

[0050] Figure 2 This is a schematic flow chart of a false information detection model training method according to an embodiment of the present invention;

[0051] Figure 3 A detailed flow chart of a false information detection model training method according to an embodiment of the present invention;

[0052] Figure 4 is a flow chart of a false information detection method according to an embodiment of the present invention;

[0053] Figure 5 (a) is a diagram of the ablation results of different models on Weibo in an embodiment of the present invention, and (b) is a diagram of the ablation results of different models on Twitter;

[0054] Figure 6 (a) is a visualization result of the attention weight of a news word on an image block in an embodiment of the present invention, and (b) is a visualization result of the attention weight of another news word on an image block;

[0055] Figure 7 are different λ in the embodiments of the present invention KL The impact of the value on the two data sets. DETAILED DESCRIPTION

[0056] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0057] A first aspect of an embodiment of the present invention discloses a false information detection model training method, such as Figure 2 and Figure 3 As shown, including:

[0058] Step 1: Obtain training information, which includes text and images related to the text.

[0059] The training information in the embodiment of the present invention is news information.

[0060] Step 2: Input the text and image into the text encoder and visual encoder respectively to obtain text representation and image representation, which specifically includes:

[0061] Step 21: Obtain the embedded information of the text, and input the embedded information into the text encoder to obtain the text representation.

[0062] In order to accurately model the semantic information of the text T in the training information and avoid word ambiguity, the pre-trained BERT (Bidirectional Encoder Representation from Transformers) is used in the embodiment of the present invention to obtain word embedding. In particular, the text T is divided into m+1 tokens {t0, t1, t2, …, t m}, where t0 is a special token [CLS] inserted at the beginning of the sentence. After the encoding layer of BERT, the corresponding embedding information of the converted text is obtained in is the output word embedded in BERT, d t is the dimension of word embedding.

[0063] In order to capture the intra-modal interactions between words in the text, the embodiment of the present invention adopts a standard Transformer layer, which consists of a multi-head self-attention and a feed-forward network (FFN) to learn the text representation H T , that is, the text encoder of the embodiment of the present invention is a Transformer encoder. Wherein the text represents H T As shown in formula (1):

[0064]

[0065] In the formula is the parameter-free position embedding corresponding to the text representation, Transformer() means to transform E T and Input to the Transformer layer.

[0066] The text representation obtained by the present invention is H T It includes query vector (Q), key vector (K) and value vector (V).

[0067] Step 22: Obtain embedded information of the image, and input the embedded information into an image encoder to obtain an image representation.

[0068] Different from the existing method of stacking multiple convolutional layers to extract visual features, the embodiment of the present invention adopts the visual Transformer model (VIT). Since the standard Transformer uses a one-dimensional symbol embedding sequence as input, the embodiment of the present invention first divides the image into multiple image blocks, obtains the embedding information of the image blocks, and then inputs the embedding information into the image encoder to obtain the image representation.

[0069] The embodiment of the present invention converts the image Reshape into a series of flat image patches Where (H, W) is the resolution of the original image, H is the height of the original image, W is the width of the original image, C is the number of channels, (P, P) is the resolution of each image block, n = HW / P 2 is the number of image blocks.

[0070] Subsequently, the embodiment of the present invention uses a trainable linear projection to flatten the image block and map it to D dimensions, thereby obtaining As the input of the pre-trained Vision Transormer (VIT). In particular, the embodiment of the present invention uses the weights of ViT-B / 16 pre-trained on ImageNet to obtain the embedding information of the image block, as shown in formula (2):

[0071]

[0072] in is the embedding of the special token '[CLS]', is the embedding of the ith image patch, d I is the dimension of the image patch embedding.

[0073] The embodiment of the present invention adopts a standard Transformer encoder to construct the internal interaction of the visual modality, that is, the image encoders in the embodiment of the present invention are all Transformer encoders, and the following image representation H can be obtained. I :

[0074]

[0075] in, is the parameter-free position embedding corresponding to the image representation, Transformer() represents E I and Input to the Transformer layer.

[0076] The image obtained by the present invention represents H I It includes query vector (Q), key vector (K) and value vector (V).

[0077] The order of step 21 and step 22 in the embodiment of the present invention can be changed.

[0078] Step 3: Interact the text representation and the image representation to determine the text-oriented multimodal features and the vision-oriented multimodal features, including:

[0079] Step 31: Input the query vector in the text representation and the key vector and value vector in the image representation into the text-oriented multimodal fusion block to obtain text-oriented multimodal features.

[0080] The text-oriented multimodal fusion block of the embodiment of the present invention includes a first modal interaction unit and a first key information selection unit;

[0081] Then step 31 specifically includes:

[0082] Step 311: input the query vector in the text representation and the key vector and value vector in the image representation into the first modal interaction unit to obtain preliminary multimodal fusion features; wherein the mechanism used in the first modal interaction unit is a collaborative attention mechanism.

[0083] In the embodiment of the present invention, the input of the i-th collaborative attention head (i=1,2,…,M) is the text representation H T And the image represents H I Converted into, as shown in formula (4):

[0084]

[0085] In the formula, is the projection matrix of the query vector in the text representation, is the projection matrix of the key vector in the image representation, is the projection matrix of the value vector in the image representation.

[0086] Then the calculation of the collaborative attention mechanism in the collaborative attention model is as follows:

[0087]

[0088] In the formula, softmax() is the softmax function, Att i is the i-th head of the multi-head attention (consisting of M heads in total), and formula (6) can be obtained from formula (5):

[0089]

[0090] Where MH_Att() is the multi-head attention function, is the weight matrix, Represents a concatenation operation.

[0091] Wrap the residual connection, feed-forward network (FFN) and normalization layer (LayerNorm (LN)) to obtain the preliminary multimodal fusion features as shown in formula (7):

[0092]

[0093] In the formula, LN() means that its parameters are input into the normalization layer (LayerNorm) of the first modal interaction unit, and FFN() means that its parameters are input into the feedforward network of the first modal interaction unit. As shown in formula (8):

[0094]

[0095] The LN() in the formula means that the parameters therein are input into the normalization layer (LayerNorm) of the first modal interaction unit.

[0096] Step 312: input the preliminary multimodal fusion features into the first key information selection unit to obtain text-oriented multimodal features; wherein the mechanism used by the first key information selection unit is a self-attention mechanism.

[0097] The embodiment of the present invention determines the text-oriented multimodal features according to formula (9):

[0098]

[0099] In the formula, LN() indicates that its parameters are input into the normalization layer (LayerNorm) of the first key information selection unit, and FFN() indicates that its parameters are input into the feedforward network of the first key information selection unit. As shown in formula (10):

[0100]

[0101] Wherein, LN() indicates that its parameters are input into the normalization layer (LayerNorm) of the first key information selection unit.

[0102] Step 32: Input the query vector in the image representation and the key vector and value vector in the text representation into the vision-oriented multimodal fusion block to obtain the vision-oriented multimodal features.

[0103] The vision-oriented multimodal fusion block of the embodiment of the present invention includes a second modality interaction unit and a second key information selection unit.

[0104] Then step 32 specifically includes:

[0105] Step 321: Input the query vector in the image representation and the key vector and value vector in the text representation into the second modality interaction unit to obtain preliminary multimodal fusion features, wherein the mechanism used in the second modality interaction unit is a collaborative attention mechanism.

[0106] In the embodiment of the present invention, the input of the i-th collaborative attention head (i=1,2,…,M) is the text representation H T And the image represents H I Converted into, as shown in formula (11):

[0107]

[0108] In the formula, is the projection matrix of the query vector in the image representation, is the projection matrix of the key vector in the text representation, is the projection matrix of the median vector in the text representation.

[0109] Then the calculation of the collaborative attention mechanism in the collaborative attention model is as follows:

[0110]

[0111] In the formula, softmax() is the softmax function, Att i is the i-th head of the multi-head attention (consisting of M heads in total), and formula (13) can be obtained from formula (12):

[0112]

[0113] In the formula, is the weight matrix, Represents a concatenation operation.

[0114] Wrap the residual connection, feedforward network (FFN) and normalization layer (LayerNorm (LN)) to obtain the preliminary multimodal fusion features as shown in formula (14):

[0115]

[0116] In the formula, LN() means that its parameters are input into the normalization layer (LayerNorm) of the second modality interaction unit, and FFN() means that its parameters are input into the feedforward network of the second modality interaction unit. As shown in formula (15):

[0117]

[0118] The LN() in the formula means that the parameters therein are input into the normalization layer (LayerNorm) of the second modal interaction unit.

[0119] Step 322: Input the preliminary multimodal fusion features into the second key information selection unit to obtain vision-oriented multimodal features, wherein the mechanism used by the second key information selection unit is a self-attention mechanism.

[0120] The embodiment of the present invention determines the vision-oriented multimodal features according to formula (16):

[0121]

[0122] In the formula, LN() indicates that its parameters are input into the normalization layer (LayerNorm) of the second key information selection unit, and FFN() indicates that its parameters are input into the feedforward network of the second key information selection unit. As shown in formula (17):

[0123]

[0124] Wherein, LN() indicates that its parameters are input into the normalization layer (LayerNorm) of the second key information selection unit.

[0125] The order of step 31 and step 32 in the embodiment of the present invention can be changed.

[0126] The function of the multimodal fusion block of the present invention is to model multimodal interaction and select key information for false information detection. In view of the fact that people judge false information from a text-oriented perspective and a visual-oriented perspective, the present invention constructs corresponding text-oriented and visual-oriented multimodal fusion blocks. Figure 3 As shown on the right, the two fusion blocks share the same architecture, each of which consists of two units, responsible for inter-modality interaction and key information selection, respectively.

[0127] Step 4: Input the text-oriented multimodal features and the vision-oriented multimodal features into their respective classifiers to obtain the text-oriented loss value and the vision-oriented loss value, which specifically include:

[0128] Step 41: input the text-oriented multimodal features into a text-oriented classifier to obtain a first prediction probability of the authenticity of the training information, and determine the text-oriented loss value based on the first prediction probability and the true value of the training information.

[0129] The embodiment of the present invention utilizes a fully connected layer and then uses a softmax function to predict the authenticity of the training information.

[0130] The first prediction probability is as follows:

[0131] P T =softmax(W T O T +b T ) (18)

[0132] Where P T is the first predicted probability, W T is the weight of the fully connected layer in the text-oriented classifier, O T is a text-oriented multimodal feature, b T is the bias in the text-oriented classifier.

[0133] The cross entropy used in the text-oriented classifier in the embodiment of the present invention is as shown in formula (19):

[0134]

[0135] In the formula, is the text-oriented loss value, N is the number of training information, and are the true value and the first predicted probability of the i-th training information of the text-oriented classifier, respectively.

[0136] Step 42: input the vision-oriented multimodal features into the vision-oriented classifier to obtain a second prediction probability of the authenticity of the training information, and determine the vision-oriented loss value according to the second prediction probability and the true value of the training information.

[0137] The second prediction probability is as follows:

[0138] P I =softmax(W I O I +bI) (20)

[0139] Where P I is the second predicted probability, W I is the weight of the fully connected layer in the vision-oriented classifier, O I is the multimodal feature for vision, b I is the bias for the vision-oriented classifier.

[0140]

[0141] In the formula, is the vision-oriented loss value, N is the number of training information, and are the true value and the second predicted probability of the i-th training information of the vision-oriented classifier, respectively.

[0142] Step 5: Determine the objective function of the false information detection model based on the text-oriented loss value and the visual-oriented loss value, and train the false information detection model based on the objective function, such as Figure 3 Shown in the upper part.

[0143] Inspired by people's dual perspectives in judging false information, the present invention adopts a mutual learning strategy to enable information transfer between text-oriented classifiers and vision-oriented classifiers, thereby promoting multimodal fusion in the process of false news detection.

[0144] For mutual learning networks, the present invention forces the two classifiers to imitate each other in the final predicted probability.

[0145] Step 5 specifically includes:

[0146] Step 51: Determine a first interaction loss value from a text-oriented classifier to a vision-oriented classifier and a second interaction loss value from a vision-oriented classifier to a text-oriented classifier according to the first prediction probability and the second prediction probability.

[0147] The embodiment of the present invention uses information divergence (Kullback Leibler (KL) divergence) to quantify the prediction P of the two networks. T and P I The KL distance calculation from the text-oriented classifier to the vision-oriented classifier is as follows:

[0148]

[0149] In the mutual learning strategy, the first interaction loss value from the text-oriented classifier to the vision-oriented classifier is as follows:

[0150]

[0151] Similarly, the KL distance calculation from the vision-oriented classifier to the text-oriented classifier is as shown in formula (24):

[0152]

[0153] In the mutual learning strategy, the second interaction loss value from the vision-oriented classifier to the text-oriented classifier is as shown in formula (25):

[0154]

[0155] Step 52: determine the objective function of the false information detection model according to the text-oriented loss value, the vision-oriented loss value, the first interaction loss value, and the second interaction loss value, and train the false information detection model according to the objective function.

[0156] The objective function of the false information detection model of the present invention is shown in formula (26):

[0157]

[0158] In the formula, is the text-oriented loss value, is the visual loss value, is the first interaction loss value, is the second interaction loss value, λ KL is the weight of mutual learning loss.

[0159] A second aspect of the present invention discloses a false information detection method, such as Figure 4 As shown, including:

[0160] Step 101: Obtain information to be detected.

[0161] The information to be detected in the embodiment of the present invention includes text and pictures related to the text.

[0162] Step 102: input the information to be detected into the false information detection model trained by the training method of the false information detection model to obtain a false information detection result.

[0163] The false information detection result obtained by the embodiment of the present invention is the predicted probability of the authenticity of the information to be detected.

[0164] Intuitively, when people judge the authenticity of multimodal news, they usually need to consider both the text-oriented perspective and the visual-oriented perspective. Specifically, the text-oriented perspective focuses on judging the text content while considering the visual content. In contrast, the visual-oriented perspective focuses on judging the visual content while considering the text content. Figure 1 As we have seen in the example above, paying attention to both text and images helps detect fake news. This is consistent with the regional network interactions in our brains when performing cognitive tasks.

[0165] Taking into account the dual-view learning process, the present invention proposes a new false information detection method, which is a multimodal detection method based on mutual learning (Multimodal Fusion via Mutual learning Network, MMNet). The mutual learning mechanism of MMNet can realize information transfer between different perspectives and better fuse multimodal information. Specifically, the model of the present invention consists of two key modules: a text-oriented classifier and a vision-oriented classifier. In each module, a new multimodal fusion block is designed to extract text / vision-oriented multimodal features, in which the interaction between the two modalities is fully characterized. In particular, the multimodal fusion block first captures cross-modal interactions through collaborative attention, and then extracts key information through self-attention for further false news classification.

[0166] The effectiveness of the method of the present invention will be verified by more specific examples below.

[0167] 1.1 Experimental Setup

[0168] 1.1.1 Dataset.

[0169] To evaluate the performance of the proposed MMNet, the present embodiment utilizes two widely used datasets: Weibo and Twitter. The Weibo dataset is collected by Xinhua News Agency and Weibo. Each post contains text, attached pictures, and social information. Fake news is captured from May 2012 to January 2016 and verified by Weibo's official rumor-busting system. The Twitter dataset is released for the Verified Multimedia Usage Task, which aims to detect false multimedia content on social media. Each tweet in the dataset includes text, pictures, and social context information associated with it. The Weibo dataset contains 9528 unique pictures, and the Twitter dataset contains 514 unique pictures. The present embodiment divides the Weibo dataset into a training set and a test set in a ratio of 8:2. For the Twitter dataset, the present invention adopts the existing preprocessing method to obtain its training set test set. Table 1 shows the statistics of the two datasets.

[0170] Table 1 Statistics of Weibo dataset and Twitter data

[0171] news Weibo Twitter Real News 4779 6026 Fake News 4749 7898

[0172] 1.1.2 Implementation details.

[0173] For word embedding, the embodiment of the present invention uses pre-trained BERT to extract text features with a dimension of 768. Specifically, bert-base-chinese is used to obtain word embedding information on the Weibo dataset, and bert-base-cased is used to learn word embedding information on the Twitter dataset. For image patch embedding, the embodiment of the present invention uses vit_base_patch16_224 to extract visual features with a dimension of 768, where the image size is resized to 224 and the patch size is 16. The embodiment of the present invention uses the adaptive moment estimation (Adam) optimizer to seek the optimal parameters, and the initial learning rate is 0.0001.

[0174] In addition to the accuracy index, the embodiment of the present invention also provides the precision, recall and F1 score of different methods for false news and true news.

[0175] 1.2 Baseline

[0176] The embodiment of the present invention compares the MMNet model with a unimodal model and a multimodal model, as shown below.

[0177] Unimodal Model. This embodiment of the present invention compares MMNet with the following model that only uses text for fake news detection.

[0178] SVM-TS: SVM-TS uses a linear SVM classifier and heuristic rules to detect fake news.

[0179] GRU: GRU uses a multi-layer GRU network to model text semantic information for fake news detection.

[0180] Multimodal model. This embodiment of the present invention compares MMNet with the following seven multimodal models.

[0181] att_RNN: att_RNN applies attention-based RNN to fuse textual, visual, and social context features for fake news detection. In this experiment, the part that processes social context features is removed.

[0182] EANN: EANN uses an auxiliary event adversarial neural network to remove specific features of events and keep shared features between events for detecting fake news. The embodiment of the present invention uses a simplified version of EANN in the experiment, excluding the event discriminator.

[0183] MVAE: MVAE combines a variational autoencoder with a binary classifier for fake news detection.

[0184] SpotFake: SpotFake uses pre-trained BERT to learn text features and leverages VGG-19 pre-trained on ImageNet to learn image features to detect fake news.

[0185] SpotFake+: SpotFake+ upgrades the pre-trained language model in SpotFake to XLNet to effectively detect fake news.

[0186] HMCAN: HMCAN adopts a hierarchical multimodal contextual attention network to detect fake news by jointly modeling multimodal contextual information and hierarchical semantics of text.

[0187] MCAN: MCAN adopts multiple collaborative attention layers to fuse multimodal features for the fake news detection task.

[0188] 1.3 Results and Analysis

[0189] Table 2 shows the overall performance comparison of different methods on the two datasets. The best result is in bold, and the second best result is underlined.

[0190] Table 2: Comparison results of different models on Weibo and Twitter datasets

[0191]

[0192] From Table 2, we can draw the following conclusions:

[0193] 1) The proposed MMNet outperforms other models in terms of accuracy and F1 score. Compared with the best baseline model, the accuracy of MMNet on Weibo and Twitter datasets is improved by about 0.78% and 3.7% respectively.

[0194] 2) The model that considers multimodal information is better than the model that only considers unimodal information. This confirms that integrating multiple modalities is advantageous for the task of fake news detection.

[0195] 3) Compared with methods that simply concatenate modal features or utilize auxiliary tasks (e.g., SpotFake, EANN), co-attention based models perform better, indicating that they can better fuse multimodal information.

[0196] 4) The MMNet of the embodiment of the present invention further surpasses the collaborative attention-based methods (i.e., HMCAN and MCAN). We believe that this is because the mutual learning network enables information transfer between the text-oriented perspective and the visual-oriented perspective, promoting multimodal fusion for fake news detection. In addition, the multimodal fusion block designed in the embodiment of the present invention can not only capture the interaction between multimodal information, but also select key information to identify fake news.

[0197] 1.4 Ablation studies

[0198] In order to verify the importance of each module in MMNet, this embodiment of the present invention compares MMNet with the following variants.

[0199] MMNet-T: A variant of MMNet based only on text-oriented classifiers.

[0200] MMNet-V: A variant of MMNet that only uses vision-oriented classification

[0201] MMNet-Co: A variant of MMNet that replaces the multimodal fusion block with the traditional co-attention network.

[0202] MMNet-Avg: A variant of MMNet that combines MMNet-T and MMNet-V by averaging the predicted probabilities.

[0203] Figure 5 shows the results of the ablation study, Figure 5 In (a) and (b), the models corresponding to each group of indicators are MMNet-T, MMNet-V, MMNet-Co, MMNet-Avg and MMNet of the present invention from left to right. Figure 5 The following observations can be obtained:

[0204] 1) The model MMNet of the present invention outperforms all the variants. The main reason is that the mutual learning between the text-oriented classifier and the vision-oriented classifier can transfer information between each other and promote multimodal fusion.

[0205] 2) Both MMNet-T and MMNet-V achieved relatively low performance, indicating that using only text-oriented or vision-oriented classifiers is a suboptimal choice for fake news detection. On the Weibo dataset, MMNet-T performs much better than MMNet-V. The reason may be that the text on Weibo is relatively long and contains more fake news detection information.

[0206] 3) MMNet consistently outperforms MMNet-Co on both datasets, demonstrating the effectiveness of the multimodal fusion block designed in this paper, which can not only capture the interaction between modalities but also select key information for fake news detection.

[0207] 4) Compared with MMNet-Avg, a method that combines text-oriented classifiers and vision-oriented classifiers by averaging, the performance of MMNet is significantly improved. This further demonstrates the effectiveness of mutual learning between different perspectives for fake news detection.

[0208] 1.5 Case Study

[0209] In addition, in order to have an intuitive understanding of the mutual learning strategy, the embodiment of the present invention visualizes the word attention weights on the image block calculated by formula (5) and formula (6). For the convenience of explanation, the embodiment of the present invention reflects the attention weight on the opacity of the image block. If the attention value is greater than the average attention weight, the opacity is set to 255; otherwise, the opacity is set to 76. The embodiment of the present invention shows the visualization results of MMNet-Avg and MMNet. Figure 6 . Figure 6 The attention weights of words are visualized on image patches. Each example consists of text (at the top, the words used for attention visualization are "dog" and "explosion") and three images, including the original image (on the left), the attention image without mutual learning obtained by MMNet-Avg (in the middle), and the attention image obtained by MMNet of an embodiment of the present invention (on the right).

[0210] from Figure 6 As can be seen from (a) in , for the word "dog", MMNet noticed the corresponding object, while the objects in MMNet-Avg without mutual learning were scattered in various corners of the image. Figure 6 In (b), it is found that the word "explosion" gives more attention to the corresponding area in the MMNet of the embodiment of the present invention. These cases show that the mutual learning between the text-oriented perspective and the vision-oriented perspective can better capture the interdependence of multimodal information.

[0211] 1.6λ KL The impact of value

[0212] To explore the weight λ of the mutual learning loss KL The impact on model performance, the embodiment of the present invention will KL From 1e -5 changes to 0.1, and Figure 7The accuracy of the two datasets is shown in Figure 2. It can be observed that on both datasets, the accuracy of fake news detection generally increases first and then decreases in 1e -5 to 0.01, and when λ KL When λ is greater than 0.01, the accuracy starts to decrease. KL When is 0.01, MMNet can best balance the individual classifiers and their mutual learning.

[0213] The embodiment of the present invention proposes a new model MMNet based on mutual learning network for multimodal fake news detection. The model of the embodiment of the present invention can transfer information between text-oriented and vision-oriented perspectives and promote multimodal fusion to enhance fake news detection. In addition, the embodiment of the present invention designs a new multimodal fusion block that can not only capture the interaction between modalities, but also select key information for fake news detection. A large number of experiments conducted on two public real-world datasets verify the effectiveness of the model MMNet of the embodiment of the present invention.

[0214] The above are only a few embodiments of the present application and do not constitute any form of limitation to the present application. Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the profession, without departing from the scope of the technical solution of the present application, using the technical content disclosed above to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A false information detection model training method, characterized in that: include: Acquire training information, wherein the training information includes text and images related to the text; Input the text and the image into respective encoders to obtain text representation and image representation; Interacting the text representation with the image representation to determine text-oriented multimodal features and vision-oriented multimodal features; Input the query vector represented by the text and the key vector and value vector represented by the image into the unit using the collaborative attention mechanism and the unit using the self-attention mechanism to obtain a text-oriented multimodal feature; input the query vector represented by the image and the key vector and value vector in the text representation into the unit using the collaborative attention mechanism and the unit using the self-attention mechanism to obtain a vision-oriented multimodal feature; Inputting the text-oriented multimodal features and the vision-oriented multimodal features into respective classifiers to obtain a text-oriented loss value and a vision-oriented loss value; An objective function of a false information detection model is determined according to the text-oriented loss value and the vision-oriented loss value, and the false information detection model is trained according to the objective function.

2. The false information detection model training method according to claim 1 is characterized in that: The text and the image are input into respective encoders to obtain text representation and image representation, which specifically includes: Obtaining embedding information of the text, and inputting the embedding information of the text into a text encoder to obtain a text representation; Obtaining embedded information of the image, and inputting the embedded information of the image into an image encoder to obtain an image representation; The text encoder and the image encoder are both Transformer encoders; The text representation and the image representation both include a query vector, a key vector and a value vector.

3. The false information detection model training method according to claim 2 is characterized in that: The text representation and the image representation are interacted to determine text-oriented multimodal features and vision-oriented multimodal features, specifically including: Inputting the query vector in the text representation and the key vector and the value vector in the image representation into a text-oriented multimodal fusion block to obtain text-oriented multimodal features; The query vector in the image representation and the key vector and the value vector in the text representation are input into a vision-oriented multimodal fusion block to obtain vision-oriented multimodal features.

4. The false information detection model training method according to claim 3 is characterized in that: The text-oriented multimodal fusion block includes a first modal interaction unit and a first key information selection unit; The query vector in the text representation and the key vector and value vector in the image representation are input into a text-oriented multimodal fusion block to obtain text-oriented multimodal features, specifically including: Inputting the query vector in the text representation and the key vector and the value vector in the image representation into a first modality interaction unit to obtain preliminary multimodal fusion features; Inputting the preliminary multimodal fusion features into the first key information selection unit to obtain text-oriented multimodal features; The mechanism used in the first modality interaction unit is a collaborative attention mechanism; The mechanism used by the first key information selection unit is the self-attention mechanism.

5. The false information detection model training method according to claim 3 is characterized in that: The vision-oriented multimodal fusion block includes a second modality interaction unit and a second key information selection unit; Inputting the query vector in the image representation and the key vector and value vector in the text representation into a vision-oriented multimodal fusion block to obtain vision-oriented multimodal features, specifically including: Inputting the query vector in the image representation and the key vector and the value vector in the text representation into a second modality interaction unit to obtain preliminary multimodal fusion features; Input the preliminary multimodal fusion features into the second key information selection unit to obtain vision-oriented multimodal features; The mechanism used in the second modality interaction unit is a collaborative attention mechanism; The mechanism used by the second key information selection unit is the self-attention mechanism.

6. The false information detection model training method according to any one of claims 1 to 5, characterized in that: The text-oriented multimodal features and the vision-oriented multimodal features are respectively input into respective classifiers to obtain text-oriented loss values ​​and vision-oriented loss values, specifically including: Inputting the text-oriented multimodal features into a text-oriented classifier to obtain a first prediction probability of the training information, and determining a text-oriented loss value according to the first prediction probability and the true value of the training information; The vision-oriented multimodal features are input into a vision-oriented classifier to obtain a second prediction probability of the training information, and a vision-oriented loss value is determined according to the second prediction probability and the true value of the training information.

7. The false information detection model training method according to claim 6 is characterized in that: The objective function of the false information detection model is determined based on the text-oriented loss value and the vision-oriented loss value, including: Determining a first interaction loss value from the text-oriented classifier to the vision-oriented classifier and a second interaction loss value from the vision-oriented classifier to the text-oriented classifier according to the first prediction probability and the second prediction probability; An objective function of a false information detection model is determined according to the text-oriented loss value, the vision-oriented loss value, the first interaction loss value, and the second interaction loss value.

8. The false information detection model training method according to claim 7, characterized in that: Determining a first interaction loss value from the text-oriented classifier to the vision-oriented classifier and a second interaction loss value from the vision-oriented classifier to the text-oriented classifier according to the first prediction probability and the second prediction probability specifically includes: According to the first prediction probability and the second prediction probability, a first interaction loss value from the text-oriented classifier to the vision-oriented classifier and a second interaction loss value from the vision-oriented classifier to the text-oriented classifier are determined using information divergence.

9. The false information detection model training method according to claim 7, characterized in that: Determining an objective function of a false information detection model according to the text-oriented loss value, the vision-oriented loss value, the first interaction loss value, and the second interaction loss value specifically includes: The objective function of the false information detection model is determined using the first formula, where the first formula is: ， In the formula, is the text-oriented loss value, is the vision-oriented loss value, is the first interaction loss value, is the second interaction loss value, is the weight of mutual learning loss.

10. A false information detection method, characterized in that: include: Obtain the information to be tested; The information to be detected is input into a false information detection model trained by the false information detection model training method described in any one of claims 1 to 9 to obtain a false information detection result.

Images