A multi-modal fake information detection method, system, device and storage medium

Abstract

The application discloses a kind of multi-modal false information detection method, system, equipment and storage medium, belong to false information detection technical field, the text input text representation network of multi-modal information is obtained text representation, the image input image representation network of multi-modal information is obtained image representation, text representation and image representation are input shared classifier, obtain text prediction information label, image prediction information label and text weight and image weight, based on text weight and image weight, by text prediction information label and image prediction information label weighting obtain prediction information label.The application is realized by combining text and image modal information, and using adaptive weight distribution mechanism, efficient, accurate multi-modal information processing is realized, and the application range is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of false information detection, and particularly relates to a multi-modal false information detection method, system, device and storage medium. BACKGROUND

[0002] Nowadays, with the rise of mobile Internet, the number of netizens is expanding, and more and more people begin to publish information on social network platforms such as microblog, twitter and so on, share their daily life or express their views and opinions on current affairs. However, these published information or articles are also full of false information, which often fabricate or distort facts, use highly provocative and exaggerated language, and publish false views with hatred or prejudice, in order to quickly attract traffic and attract people's attention. These unverified rumors have caused great harm after being published, but it is too costly to manually review and check each rumor, and it is often time-consuming and laborious, so there is an urgent need for a technical means that can automatically detect and identify false information.

[0003] The false information detection algorithm based on deep learning can be roughly divided into two types based on content and social context. The false information detection algorithm based on content usually starts from the content in the information to be detected, learns the features in the content to complete the classification task, and is particularly suitable for early detection scenarios. Among them, the single-modal false information detection algorithm usually only uses text or images in the information to extract single-modal features to complete the classification task. The commonly used method is based on various texts, which takes the text content of the information as the model input, obtains word embedding in the way of Word2Vec or BERT, and extracts text features through LSTM to classify samples. However, there is a certain limitation in using only single-modal information, and the model discrimination ability is general.

[0004] In addition, many studies have also proposed different multi-modal false information detection algorithms, which can fully utilize the information contained in the text and images in these information, learn the features therein to identify false information. Although these methods are based on different innovative ideas, they can all be classified into a category: first extract features from different single-modal, and then fuse them to obtain multi-modal representation for the final classification task, such as Figure 1However, as a task derived from natural language processing, the contribution and proportion of features from different modalities such as images and texts to the final result are different, and there are also significant differences in the data distribution. Some methods simply and roughly use the splicing method to fuse the image and text features. Some researches use attention mechanisms and other methods to weight and fuse the image and text, but there is a problem of modality inertia. A certain modality is given a smaller weight, so that its gradient becomes smaller when backpropagating, and the update of some parameters in the corresponding model becomes slower, and finally this modality is ignored by the model during training and is not fully learned. In addition, in order to learn multi-modal representation, many methods use complex fusion mechanisms such as attention, which increases the number of parameters of the model and sharply increases the computational complexity. At the same time, the samples sent into the model for calculation also need complete images and texts, which cannot deal with the problem of missing modalities and is difficult to deal with real-world scenarios. SUMMARY

[0005] In order to solve the problem of modality inertia in the weighted fusion of existing multi-modal fake information detection methods and the low accuracy of the prediction result, the present application provides a multi-modal fake information detection method, system, device and storage medium.

[0006] In order to achieve the above purpose, the present application provides the following technical solutions:

[0007] The present application provides a multi-modal fake information detection method, comprising:

[0008] inputting the text of the multi-modal information into a text representation network to obtain a text representation;

[0009] inputting the image of the multi-modal information into an image representation network to obtain an image representation;

[0010] inputting the text representation and the image representation into a shared classifier to obtain a text predicted information label, an image predicted information label, a text weight and an image weight, and obtaining a predicted information label by weighting the text predicted information label and the image predicted information label based on the text weight and the image weight.

[0011] The text representation network comprises a Bert model and a text encoder, and the text of the multi-modal information is input into the text representation network to obtain the text representation, specifically:

[0012] The text of the multi-modal information is converted into a text vector by the Bert model;

[0013] The text vector is processed by the text encoder to obtain the text representation, and the text encoder adopts a TextCNN.

[0014] The image representation network comprises a ResNet50 model and an image encoder, the image of the multi-modal information is input into the image representation network to obtain an image representation, and the image representation is specifically obtained by:

[0015] The image of the multi-modal information is input into the image representation network to obtain an image representation, and the image representation is specifically obtained by:

[0016] The image of the multi-modal information is input into the image representation network to obtain an image representation, and the image representation is specifically obtained by:

[0017] The text representation and the image representation are input into the shared classifier to obtain a text prediction information label, an image prediction information label, a text weight and an image weight, and the prediction information label is obtained by weighting the text prediction information label and the image prediction information label based on the text weight and the image weight, and the prediction information label is specifically obtained by:

[0018] The text representation and the image representation are input into the shared classifier to obtain a text prediction information label, an image prediction information label, a text weight and an image weight, and the prediction information label is obtained by weighting the text prediction information label and the image prediction information label based on the text weight and the image weight, and the prediction information label is specifically obtained by:

[0019] The text representation and the image representation are input into the shared classifier to obtain a text prediction information label, an image prediction information label, a text weight and an image weight, and the prediction information label is obtained by weighting the text prediction information label and the image prediction information label based on the text weight and the image weight, and the prediction information label is specifically obtained by:

[0020] The text representation and the image representation are input into the shared classifier to obtain a text prediction information label, an image prediction information label, a text weight and an image weight, and the prediction information label is obtained by weighting the text prediction information label and the image prediction information label based on the text weight and the image weight, and the prediction information label is specifically obtained by:

[0021] The multi-modal false information detection method provided by the application further comprises:

[0022] The sample multi-modal information and the true information label of the sample multi-modal information are obtained.

[0023] The text of the sample multi-modal information is input into a preset text representation network to obtain a text representation.

[0024] The text representation is input into a preset shared classifier to obtain a text prediction information label

[0025] The text loss is calculated according to the text prediction information label and the true information label, and the parameters of the text encoder in the text representation network and the preset shared classifier are updated according to the text loss.

[0026] The image of the sample multi-modal information is input into a preset image representation network to obtain an image representation.

[0027] The image representation is input into the shared classifier updated according to the text loss to obtain an image prediction information label.

[0028] The image classification loss is calculated according to the image prediction information label and the real information label; meanwhile, the image contrast loss is obtained by using the contrast learning mode, and the parameters of the image encoder in the image representation network and the shared classifier after the text loss are updated according to the image loss including the image classification loss and the image contrast loss.

[0029] The preset text representation network, the preset image representation network and the preset shared classifier are iteratively trained until a preset convergence condition is reached to obtain the text representation network, the image representation network and the shared classifier for multi-modal false information detection.

[0030] The text loss is obtained based on a cross-entropy loss function, the image classification loss is obtained based on a cross-entropy loss function, and the image contrast loss is obtained based on an InfoNCE loss function.

[0031] The parameters in the preset shared classifier are updated according to the text loss, and the parameters of the shared classifier after the text loss are updated are updated according to the image loss, and specifically:

[0032] Based on the gradient correction mechanism, the text loss and the image loss are iteratively trained on the preset shared classifier until a preset convergence condition is reached to obtain the shared classifier.

[0033] The application also provides a multi-modal false information detection system, comprising:

[0034] A text representation module is configured to input the text of the multi-modal information into a text representation network to obtain a text representation.

[0035] An image representation module is configured to input the image of the multi-modal information into an image representation network to obtain an image representation.

[0036] A prediction module is configured to input the text representation and the image representation into a shared classifier to obtain a text prediction information label, an image prediction information label, a text weight and an image weight, and obtain a prediction information label by weighting the text prediction information label and the image prediction information label based on the text weight and the image weight.

[0037] The application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-modal false information detection method when executing the computer program.

[0038] The application also provides a computer readable storage medium storing a computer program, wherein the computer program is executable on a processor to implement the steps of the multi-modal false information detection method.

[0039] Compared with the prior art, the application has the following advantages:

[0040] The method combines the information of the two modalities of text and image, respectively extracts the features of each other through a text representation network and an image representation network, realizes sufficient utilization of information, and then uses a shared classifier to predict the representation of the two modalities; the weight coefficient of the two modalities in the final result is determined according to the entropy of the single modality, the reliability of the modality is reflected through the weight distribution mechanism, the negative influence of the modality with greater uncertainty on the final result is effectively reduced, and the accuracy of the prediction result is improved. At the same time, the present application does not use a complex multi-modal fusion mechanism, reduces the computational complexity, improves the reasoning speed, and makes the application more efficient in practical application; when a certain modality is missing, the present application can still maintain good prediction performance.

[0041] Further, in the training stage, first, the respective single modality representation is extracted from the picture and the text, instead of weighted fusion of representation at the beginning; then we take the alternating training strategy, independently train the picture and text representation, and then use the picture representation and the text representation to classify and predict the sample respectively, and update the network parameters according to the loss calculated from the prediction result; ensure that each modality can be fully learned and optimized, and at the same time, in view of the problem that the discriminative ability of the picture modality may be limited when it is trained alone, a contrast learning mechanism is introduced, which can improve the prediction effect of the picture modality, so that it can more accurately capture the key features in the training process. The present application effectively avoids the modality inertia problem and enhances the discriminative ability of the model, realizes the overall improvement of the model performance. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and should not be regarded as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 A multi-modal feature fusion schematic diagram in the prior art;

[0044] Figure 2 A flowchart of the multi-modal false information detection method of the present application;

[0045] Figure 3 A schematic diagram of the multi-modal false information detection method of the present application;

[0046] Figure 4 A multi-modal feature fusion schematic diagram in the present application;

[0047] Figure 5 A text encoder TextCNN structure diagram of the present application;

[0048] Figure 6 A schematic diagram of the multi-modal fake information detection method of the present application;

[0049] Figure 7 A picture comparison learning structure diagram of the present application;

[0050] Figure 8 A multi-modal fake information detection system structure schematic diagram of the preferred embodiment of the present application;

[0051] Figure 9 An electronic device structure schematic diagram of the preferred embodiment of the present application. DETAILED DESCRIPTION

[0052] To enable persons skilled in the art to understand the characteristics and effects of the present application, the following is a general description and definition of the terms and phrases mentioned in the specification and claims. Unless otherwise specified, all technical and scientific words used herein have their usual meanings to those skilled in the art of the present application, and in case of conflict, the definition in the specification shall prevail.

[0053] Theories or mechanisms described and disclosed in the present application, whether correct or not, should not in any way limit the scope of the present application, i.e. the content of the present application can be implemented without being limited by any particular theory or mechanism.

[0054] In the present application, all features defined in the form of numerical range or percentage range, such as numerical value, quantity, content and concentration, are for the sake of brevity and convenience. Therefore, the description of numerical range or percentage range should be considered to have covered and specifically disclosed all possible sub-ranges and individual numerical values within the range (including integers and fractions).

[0055] In the present application, unless otherwise specified, "comprising", "including", "containing", "having" or similar terms encompass the meaning of "consisting of" and "consisting essentially of", for example, "A comprising a" encompasses the meaning of "A comprising a and other" and "A comprising only a".

[0056] As shown in Figures 2-4 The present application provides a multi-modal fake information detection method, which comprises:

[0057] S1: inputting the text of multi-modal information into a text representation network to obtain a text representation, specifically:

[0058] The text representation network is obtained by preset text representation network training, and the preset text representation network comprises a Bert model and a text encoder;

[0059] The text of multi-modal information is a continuous list of words wherein, is the length of the sequence, the text is obtained by the Bert model wherein, is wi the output of the last hidden layer of the Bert, , dw is the dimension of the word embedding in the Bert;

[0060] As shown in Figure 6 , the TextCNN is used as the text encoder, and the text representation is further extracted by the text encoder from the text vector , specifically:

[0061] The window size of a certain convolution kernel is set to s , and the convolution kernel is used to perform convolution operation on the consecutive words token in the text vector s to obtain the i th feature.

[0062]

[0063] wherein, and b c are the weights and constant bias terms of the convolution kernel, represents the concatenation of the i th word token to the th word token, is the ReLu activation function;

[0064] The convolution kernel will scan and operate on all the words of the text vector using the window s to obtain the feature map , ; then the most important information in the feature map is obtained by using the maximum pooling operation as the feature of the corresponding convolution kernel, and is obtained; in order to obtain different levels of semantics, the window size of n s different windows is set, each window corresponds to n c convolution kernels, and the features of all convolution kernels are spliced to obtain the text representation , wherein .

[0065] S2: input the image of the multi-modal information into the image representation network to obtain the image representation, specifically:

[0066] The image representation network is trained using a pre-defined image representation network, which includes a ResNet50 model and an image encoder.

[0067] Multimodal information images Features Features were analyzed using the ResNet50 model. Initial encoding is performed, using the output of the penultimate layer of the ResNet50 model as the image. Image vectors ,in , , , The number of channels, width, height, and dimensions of the output features of the penultimate layer of ResNet50 are respectively;

[0068] A linear layer encoder is used as the image encoder, which processes the image vector. Processing to obtain image representation .

[0069] In an embodiment of the present invention, a ResNet50 model is obtained by using a ResNet model pre-trained on the ImageNet1K dataset.

[0070] S3: Input the text representation and image representation into the shared classifier to obtain text prediction information labels, image prediction information labels, text weights, and image weights. Based on the text weights and image weights, obtain the prediction information labels by weighting the text prediction information labels and image prediction information labels, specifically:

[0071] The shared classifier is obtained by training a pre-defined shared classifier.

[0072] Input the text representation and image representation into the shared classifier to obtain the text prediction information label and the image prediction information label, as shown in the following formula;

[0073]

[0074]

[0075] in, Predict information labels for text. Predict information labels for images. R t For text representation, R v For image representation, W h , b h The parameters of the shared classifier h;

[0076] In order to avoid modal inertia, the features of the picture and the text are not fused, but are separately trained, and they are different in sharing the final result and should be given different weights. As shown in Figure 3 , the text representation and the picture representation of the multi-modal information are obtained by using the trained text representation network and the image representation network respectively, and then the shared classifier is used to generate the text prediction information label and the image prediction information label, and the prediction information label is obtained by weighting, as follows:

[0077]

[0078] wherein, λ t is the text weight, λ v is the image weight;

[0079] In an embodiment of the present application, for a certain specific modal , the entropy output by the shared classifier is , wherein, pm is the probability of the sample being divided into each category; the text entropy is , and the image entropy is .

[0080] In an embodiment of the present application, for a certain specific modal , when the modal shows higher uncertainty in the inference process, it is more likely to produce an erroneous prediction, and therefore, the weight of the modal to the final result λ m should be smaller; and the entropy output by each modal after the shared classifier h can well reflect this uncertainty. When the entropy output by the modal m is larger, it means that the credibility of the prediction result is lower, and the corresponding weight should be smaller. Therefore, the weight of the modal m can be obtained by subtracting the entropy e m of the modal m from the maximum value of the entropies of the two modes by using the softmax, and the weight λ m of the modal can be obtained by the following formula:

[0081]

[0082] wherein, is the maximum value of the entropies of the text modal and the image modal, e m is the entropy of the modal mentropy of the text, λ m weight of the modality m .

[0083] The shared classifier is a graph-text shared classifier of a multilayer perception machine.

[0084] In the embodiment of the application, the text feature extraction network, the image encoder and the shared classifier can be obtained by jointly training a preset text representation network, a preset image representation network and a preset shared classifier, the preset text representation network comprising a Bert model and a preset text encoder; the image representation network is obtained by training a preset image representation network, the preset image representation network comprising a ResNet50 model and an image encoder.

[0085] As Figure 6 shown, the multi-modal false information detection method provided by the embodiment of the application can further comprise:

[0086] S401: acquiring sample multi-modal information and a true information label of the sample multi-modal information;

[0087] In the embodiment of the application, sample multi-modal information is acquired from social media as training data, and a preset multi-modal information , wherein, W is text of the multi-modal information, I is an image of the multi-modal information. The true information label of the sample multi-modal information is , the predicted information label is and ; the multi-modal false information detection method is to determine whether the multi-modal false information is false information ( ) or true information ( ) according to the text feature and the image feature in the multi-modal information;

[0088] S402: inputting the text of the sample multi-modal information into a preset text representation network to obtain text representation;

[0089] S403: inputting the text representation into a preset shared classifier to obtain a text predicted information label

[0090] S404: calculating a text loss according to the text predicted information label and the true information label, and updating a text encoder in the text representation network and parameters in the preset shared classifier according to the text loss;

[0091] S405: inputting the image of the sample multi-modal information into a preset image representation network to obtain image representation;

[0092] S406: inputting the image representation into the shared classifier updated by the text classification loss to obtain an image predicted information label;

[0093] S407: Calculate the image classification loss according to the image prediction information label and the real information label; at the same time, obtain the image contrast loss by using the contrast learning method, and update the parameters of the image encoder in the image representation network and the shared classifier after the text loss according to the image loss including the image classification loss and the image contrast loss;

[0094] S408: Based on the updated preset text representation network, preset image representation network and preset shared classifier, repeat steps S402-S407 for iterative training until a preset convergence condition is reached to obtain the text representation network, image representation network and shared classifier for multi-modal false information detection.

[0095] In the embodiment of the application, the text loss is calculated by a cross-entropy loss function, as follows:

[0096]

[0097] wherein, loss t is the text loss, y is the real information label, is the text prediction information label

[0098] In the embodiment of the application, the image loss includes the image classification loss and the image contrast loss, as follows:

[0099]

[0100] wherein, loss v is the image loss, loss clf is the image classification loss, loss con is the image contrast loss;

[0101] The image classification loss is calculated by a cross-entropy loss function, as follows:

[0102]

[0103] wherein, loss clf is the image classification loss, y is the real information label, is the image prediction information label.

[0104] After considering the limitation of using only image features for false information discrimination, the application introduces a contrast learning technology to enhance the performance of the image encoder , especially for complex samples that are difficult to distinguish by traditional methods. For example, Figure 6As shown, this invention employs momentum contrastive learning, constructing a query dictionary from images of all multimodal information samples in a queue to address the problem of insufficient samples for comparison, and then using momentum to update the image encoder. To avoid inconsistencies in sample features during each comparison, the following steps are taken: For each multimodal information image, it is first treated as an anchor sample. The result of random dropout of all multimodal information images is used as a positive sample. Then, the images from all multimodal information images are selected as anchor samples. Each sample other than itself is considered a negative sample. , All positive and negative samples are used as keys in the query dictionary, using the image encoder. Linear layer encoder with the same structure As a momentum encoder, it encodes the positive sample features to obtain To minimize the distance between anchor samples and positive samples while maximizing the distance between anchor samples and negative samples, vector multiplication is used to measure the similarity between samples, and the InfoNCE loss function is employed to calculate the image contrast loss. loss con As shown in the following formula:

[0105]

[0106] in, For positive sample image representation, Representation of negative sample images; This represents the anchor point sample image.

[0107] It should be noted that a size of [size missing] is used. A queue is used to construct a query dictionary that stores the features of all negative samples. When the images of each batch of multimodal information are input sequentially, the features of all negative samples stored within are used... Used for calculating the contrast loss. Furthermore, the query dictionary is dynamically updated; after each batch's calculation, the positive sample features of the current batch are enqueued as negative samples for the next batch, while the oldest batch's feature vectors are dequeued. The query dictionary always stores a small subset of the training data, ensuring it represents the newest data in each training epoch. To reduce computational overhead, the linear layer encoder... Since it does not participate in parameter updates during backpropagation, the features of each key stored in the query dictionary queue have no gradients. However, to avoid inconsistencies in the features of the contrasting samples, after each batch completes backpropagation, the image encoder is updated using momentum. The parameters are copied to the linear layer encoder. In the embodiment, the momentum updating method is as follows:

[0108]

[0109] In the embodiment, the parameters of the preset shared classifier are updated through a text loss and an image loss, specifically:

[0110] As shown in Figure 7 , the picture and the text modal use a shared classifier to capture the cross-modal fusion features. In order to avoid modal forgetting, the application introduces a gradient correction mechanism, and the text modal and the image modal are alternately trained. In each round of iterative learning back propagation gradient calculation, the gradient value of the shared classifier parameter is modified, so that the shared classifier can take into account the data in the previous round of training, thereby avoiding forgetting the learned modal. In the first round of back propagation parameter update , it is assumed that the current learning modal is , , and the update is as follows:

[0111]

[0112] wherein is a gradient correction matrix, is the parameter of the shared classifier , and is the gradient of the loss , and is the learning rate.

[0113] In the initial stage, , the conventional gradient descent method is used for updating; but in each subsequent round of back propagation (i.e. ), the gradient correction matrix can modify the corresponding gradient value before updating the parameters. In the m th round of back propagation, the average value of the final representation of all samples in a sample batch after passing through the encoder of the modal is , which can be obtained by the following recursive least squares method, as follows:

[0114]

[0115] wherein is a constant term, is the initial value of the gradient correction matrix, and the unit matrix.

[0116] Embodiment

[0117] In this embodiment, the Chinese short text dataset Weibo21 and the English long text dataset GossipCop are used respectively. Since the present application mainly discusses the multi-modal fake news detection algorithm, in this embodiment, the pure text samples without pictures are removed, and for the samples with multiple pictures, one of them is randomly selected. Each dataset is strictly divided into training set, validation set and test set according to the ratio of 7:1:2.

[0118] The present application is implemented on Python using Pytorch. In this embodiment, the random gradient descent is used to update the parameters, and the iteration epochs is set to 100. For Chinese and English texts, the pre-trained models bert-base-chinese and bert-base-uncased from HuggingFace are used to obtain text vectors respectively. At the same time, the ResNet model pre-trained on ImageNet1K dataset in PyTorch is used to extract image features. The parameters of all the above pre-trained models remain fixed in the training and do not participate in gradient calculation and parameter update. The server memory used in the experiment is 90GB, the CPU is 12-core Xeon(R) Platinum 8352V, and the GPU is Nvidia GeForce RTX 4090(24GB).

[0119] Based on repeated experiments on the dataset, the best network structure and hyperparameters are set as shown in Table 1.

[0120] Table 1 Experimental settings

[0121]

[0122] In order to clarify the effectiveness and advantages of the present application, a series of comparisons are made with the prior art. Document 【1】: SpotFake: A Multi-Modal Framework for Fake News Detection, which uses a pre-trained Bert model and a VGG model to extract text and image features respectively, and then splices them to form the final representation into the classifier. Document 【2】: EANN: Event Adversarial Neural Networks for Multi-Modal Fake News Detection, which uses adversarial training to avoid overfitting on samples in a certain field based on splicing single-modal features, improving the generalization ability of the model. Document 【3】: Multimodal Fusion with Co-Attention Networks for Fake News Detection, which uses discrete cosine transform to extract the frequency domain features of the image, and then stacks multiple co-attention layers to continuously fuse them with text features and spatial domain features of the image to obtain the multi-modal representation of the sample. Document 【4】: Hierarchical Multi-Modal Contextual Attention Network for Fake News Detection, which divides the 12 hidden layers of Bert into three groups on average, and adds their outputs to obtain more rich hierarchical semantic features, and then uses a complex upper Transformer to complete multi-modal fusion. Document 【5】: Cross-Modal Ambiguity Learning for Multimodal Fake News Detection, which additionally introduces an auxiliary task of cross-modal alignment to learn cross-modal ambiguity with a variational autoencoder, and calculates the KL divergence of the distribution of different modal data as the weight of single-modal and multi-modal feature fusion.

[0123] This invention selects classic evaluation metrics—accuracy, precision, recall, and F1 score—as the evaluation criteria for classification comparison. Comparisons were conducted on two real-world datasets, Weibo21 and GossipCop, with results shown in Table 2. As shown in Table 2, this invention employs alternating training of text and images, avoiding weighted fusion of multimodal features during training, which improves the model's classification performance to some extent. The experimental data shows a significant improvement over other models, achieving the highest accuracy on both Weibo21 and GossipCop (0.933 and 0.884 respectively), and also achieving good results in precision, recall, and F1 score. Among all methods, SpotFake performed the worst because it only uses the simplest method to concatenate text and images, failing to fully learn multimodal features. EANN, which also uses a concatenation method, improves the model's generalization ability to some extent due to its adversarial training. MCAN, HMCAN, and CAFE all employ attention mechanisms to learn richer multimodal features of text and images, achieving good results. MCAN considers the frequency domain information of tampered or forged images, achieving a recall rate of 0.935 for fake information on Weibo21. HMCAN, on the other hand, learns hierarchical semantic features, achieving a precision rate of 0.943 for true information on Weibo21. CAFE uses KL divergence to balance unimodal and multimodal features, achieving first place in both fake information precision and true information recall on GossipCop. However, these baseline models cannot avoid the problem of modal inertia during training and still cannot fully learn the features of images and text.

[0124] Table 2 Comparative experimental results of the present invention and existing methods

[0125]

[0126] To analyze the function of each key module in this invention, the following variations were designed for ablation experiments: 1) w / o T: Remove the text classification module and use only images for classification training; 2) w / o I: Remove the image classification module and use only text for classification training; 3) w / o C: Remove the contrastive learning module in image classification training; 4) w / o S: Do not use a shared classifier in alternating image and text training. but each uses a classifier to output the final result; 5) w / o G: no gradient correction in the image-text sharing classifier; 6) w / o E: in the prediction reasoning, the entropy is not used as the dynamic fusion mechanism of the weight, but the classification results of the picture and the text output are directly added as the final result. The experimental results are shown in Table 3. After removing different key modules respectively, the effects of these variants are not good. The effects of w / o T and w / o I, which remove the text module and the image module respectively, have a large decline on the two data sets, especially the accuracy of w / o T is even less than 80%, which shows that the text information is more important for the multi-modal fake information detection task. After removing the sharing classifier, the accuracy also decreases by about 2 percentage points, because it loses important information of cross-modal interaction between images and texts, and the information provided by single-modal features is quite limited. At the same time, the gradient correction mechanism in the sharing classifier also plays a great role in solving the problem of modal forgetting. The accuracy of w / o G without gradient correction is only 0.913 and 0.877 on Weibo21 and GossipCop data sets respectively. In the reasoning stage, the entropy of each modal output is used as the weight of the modal to dynamically fuse the image and text. w / o E simply adds the outputs of the image and the text, simply and roughly treats them as the same, and ignores the different contributions of the picture and the text to the fake information detection task. The performance on Weibo21 is even worse than w / o S without sharing classifier and w / o G without gradient correction.

[0127] Table 3 Ablation experiment results of the present application

[0128]

[0129] As Figure 8 shown, another object of the present application is to provide a multi-modal fake information detection system, comprising:

[0130] a text representation module configured to input the text of the multi-modal information into a text representation network to obtain a text representation;

[0131] an image representation module configured to input the image of the multi-modal information into an image representation network to obtain an image representation;

[0132] a prediction module configured to input the text representation and the image representation into a sharing classifier to obtain a text predicted information label, an image predicted information label, a text weight and an image weight, and obtain a predicted information label by weighting the text predicted information label and the image predicted information label based on the text weight and the image weight.

[0133] As Figure 9According to the third aspect of the present application, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-modal false information detection method when executing the computer program.

[0134] The multi-modal false information detection method comprises:

[0135] inputting the text of the multi-modal information into a text representation network to obtain a text representation;

[0136] inputting the image of the multi-modal information into an image representation network to obtain an image representation;

[0137] inputting the text representation and the image representation into a shared classifier to obtain a text predicted information label, an image predicted information label, a text weight, and an image weight, and obtaining a predicted information label by weighting the text predicted information label and the image predicted information label based on the text weight and the image weight.

[0138] According to the fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, wherein the computer program implements the steps of the multi-modal false information detection method when executed by a processor.

[0139] The multi-modal false information detection method comprises:

[0140] inputting the text of the multi-modal information into a text representation network to obtain a text representation;

[0141] inputting the image of the multi-modal information into an image representation network to obtain an image representation;

[0142] inputting the text representation and the image representation into a shared classifier to obtain a text predicted information label, an image predicted information label, a text weight, and an image weight, and obtaining a predicted information label by weighting the text predicted information label and the image predicted information label based on the text weight and the image weight.

[0143] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0144] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks.

[0145] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks.

[0146] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks.

[0147] Finally, it should be noted that the above-mentioned embodiments are merely intended to illustrate the technical solutions of the present application, but not to limit it. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced, and any modification or replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A multi-modal fake news detection method, characterized in that, The method comprises the following steps: inputting the text of the multi-modal information into a text representation network to obtain text representation; inputting the image of the multi-modal information into an image representation network to obtain image representation; training the text representation into a shared classifier to obtain text prediction information label and text entropy, and training the image representation into the shared classifier to obtain image prediction information label and image entropy; obtaining text weight through the text entropy and obtaining image weight through the image entropy; based on the text weight and the image weight, the prediction information label is obtained by weighting the text prediction information label and the image prediction information label, specifically as follows: wherein, λ t is a text weight, λ v is an image weight, is a text prediction information tag, is an image prediction information tag, wherein H is is the maximum of the entropy of the text modality and the image modality, e m is the entropy of the modality m , λ m is the weight of the modality m ; e v is the text entropy; e t is the image entropy.

2. The multi-modal fake information detection method of claim 1, wherein, The text representation network comprises a Bert model and a text encoder, and the text of the multi-modal information is inputted into the text representation network to obtain the text representation, specifically as follows: the text of the multi-modal information is converted into a text vector through the Bert model; the text vector is processed based on the text encoder to obtain the text representation, and the text encoder adopts a TextCNN.

3. The multi-modal fake information detection method of claim 1, wherein, The image representation network comprises a ResNet50 model and an image encoder, and the image of the multi-modal information is inputted into the image representation network to obtain the image representation, specifically as follows: the image of the multi-modal information is converted into an image vector through the ResNet50 model; the image vector is processed based on the image encoder to obtain the image representation, and the image encoder adopts a linear layer encoder.

4. The multi-modal fake information detection method of claim 1, wherein, Further comprising: obtaining sample multi-modal information and real information label of the sample multi-modal information; inputting the text of the sample multi-modal information into a preset text representation network to obtain text representation; inputting the text representation into a preset shared classifier to obtain text prediction information label calculating text loss according to the text prediction information label and the real information label, and updating the parameters of the text encoder in the text representation network and the preset shared classifier according to the text loss; inputting the image of the sample multi-modal information into a preset image representation network to obtain image representation; inputting the image representation into the shared classifier updated by the text loss to obtain image prediction information label; calculating image classification loss according to the image prediction information label and the real information label; at the same time, obtaining image contrast loss by using contrast learning, and updating the parameters of the image encoder in the image representation network and the shared classifier updated by the text loss according to the image loss comprising the image classification loss and the image contrast loss; iteratively training based on the updated preset text representation network, preset image representation network and preset shared classifier until a preset convergence condition is reached to obtain the text representation network, image representation network and shared classifier for multi-modal false information detection.

5. The multi-modal fake information detection method of claim 4, wherein, The text loss is obtained based on a cross-entropy loss function, the image classification loss is obtained based on a cross-entropy loss function, and the image contrast loss is obtained based on an InfoNCE loss function.

6. The multi-modal fake information detection method of claim 4, wherein, The parameters of the preset shared classifier are updated according to the text loss, and the parameters of the shared classifier updated by the text loss are updated according to the image loss, specifically as follows: based on a gradient correction mechanism, the preset shared classifier is iteratively trained by the text loss and the image loss until a preset convergence condition is reached to obtain the shared classifier.

7. A multi-modal disinformation detection system, characterized in that, ​ The text representation module is configured to input the text of the multi-modal information into a text representation network to obtain a text representation. The image representation module is configured to input the image of the multi-modal information into an image representation network to obtain an image representation. The prediction module is configured to input the text representation into a shared classifier to obtain a text prediction information label and a text entropy, input the image representation into the shared classifier to obtain an image prediction information label and an image entropy, obtain a text weight through the text entropy, obtain an image weight through the image entropy, and obtain a prediction information label through the text prediction information label and the image prediction information label based on the text weight and the image weight, specifically as follows: wherein, λ t is a text weight, λ v is an image weight, is a text prediction information tag, is an image prediction information tag, wherein H is is the maximum of the entropy of the text modality and the image modality, e m is the entropy of the modality m , λ m is the weight of the modality m ; e v is the text entropy; e t is the image entropy. 8.An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-modal false information detection method of any one of claims 1-6 when executing the computer program. 9.A computer readable storage medium storing a computer program, wherein the computer program is executable on a processor to implement the steps of the multi-modal false information detection method of any one of claims 1-6.

Images