Multi-modal false information detection method, system and device and storage medium

Abstract

The invention discloses a multi-modal false information detection method, system and device and a storage medium, and belongs to the technical field of false information detection.The method comprises the steps that a text of multi-modal information is input into a text representation network to obtain text representation, and an image of the multi-modal information is input into an image representation network to obtain image representation; and inputting the text representation and the image representation into a shared classifier to obtain a text prediction information label, an image prediction information label, a text weight and an image weight, and weighting the text prediction information label and the image prediction information label based on the text weight and the image weight to obtain a prediction information label. According to the method, the text and image modal information is combined, and an adaptive weight distribution mechanism is adopted, so that efficient and accurate multi-modal information processing is realized, the prediction performance is remarkably improved, and the application range is widened.

Description

Technical Field

[0001] The present invention relates to the technical field of false information detection, and in particular to a multimodal false information detection method, system, device and storage medium. Background Art

[0002] Nowadays, with the rise of mobile Internet, the number of netizens continues to expand, and more and more people begin to post information on social networking platforms such as Weibo and Twitter to share their daily lives or express their views and opinions on current affairs. However, these information or articles are also full of false information. They often fabricate or distort facts, use inflammatory and exaggerated remarks, and express wrong views with hatred or prejudice and discrimination, in order to quickly attract traffic and attract attention. These unconfirmed rumors have caused great harm after being released, but the cost of manually reviewing and checking these rumor information one by one is too high, and it is often time-consuming and laborious. Therefore, there is an urgent need for technical means to automatically detect and identify false information.

[0003] False information detection algorithms based on deep learning can be roughly divided into two types: content-based and social context-based. Content-based false information detection algorithms usually start from the content of the information to be detected, learn the features in the content to complete the classification task, and are particularly suitable for early detection scenarios. Among them, unimodal false information detection algorithms usually only use the text or images in the information to extract unimodal features to complete the classification task. The more common methods are based on various types of text. They use the text content of the information as the model input, obtain word embeddings through methods such as Word2Vec or BERT, and extract text features through LSTM to classify samples. However, using only unimodal information has certain limitations, and the model's discrimination ability is average.

[0004] In addition, many studies have proposed different multimodal false information detection algorithms. These methods can make full use of the information contained in the text and images in these information, learn the features in them to identify false information. Although these methods are based on different innovative ideas, they can all be classified into one category: first extract features from different single modalities, and then fuse them to obtain multimodal representations for the final classification task, such as Figure 1As shown. However, as a task derived from natural language processing, false information detection has different contributions and proportions from features of different modalities such as images and texts to the final result, and they also have significant differences in data distribution. Some methods ignore this point and simply and crudely use splicing to fuse image and text features. Although some studies use methods such as attention mechanisms to perform weighted fusion of images and texts, there is a problem of modal inertia. A certain modality is given a smaller weight, resulting in a smaller gradient during back propagation, and some parameters in the corresponding model are updated more slowly. In the end, this modality is ignored by the model during training and is not fully learned. In addition, in order to learn multimodal representations, many methods use complex fusion mechanisms such as attention, which results in more model parameters and a sharp increase in computational complexity. At the same time, the samples sent to the model for calculation also require complete images and texts, which cannot cope with the problem of missing modalities and are difficult to cope with scenes in the real world. Summary of the invention

[0005] In order to solve the problem that the existing multimodal false information detection method has modal inertia in weighted fusion, resulting in low accuracy of prediction results, the present invention provides a multimodal false information detection method, system, device and storage medium.

[0006] In order to achieve the above object, the present invention provides the following technical solutions: The present invention provides a multimodal false information detection method, comprising: Input the text of multimodal information into the text representation network to obtain text representation; Input the image of multimodal information into the image representation network to obtain the image representation; The text representation and the image representation are input into a shared classifier to obtain text prediction information labels, image prediction information labels, text weights, and image weights. Based on the text weights and image weights, the prediction information labels are weighted by the text prediction information labels and the image prediction information labels to obtain prediction information labels.

[0007] The text representation network includes a Bert model and a text encoder. The text of the multimodal information is input into the text representation network to obtain the text representation, which is specifically: The text of multimodal information is obtained through the Bert model to obtain text vectors; The text vector is processed based on a text encoder to obtain a text representation, and the text encoder adopts TextCNN.

[0008] The image representation network includes a ResNet50 model and an image encoder. The image of multimodal information is input into the image representation network to obtain image representation, specifically: The image with multimodal information is passed through the ResNet50 model to obtain the image vector; The image vector is processed based on an image encoder to obtain an image representation, and the image encoder adopts a linear layer encoder.

[0009] The text representation and the image representation are input into the shared classifier to obtain the text prediction information label, the image prediction information label, the text weight and the image weight. Based on the text weight and the image weight, the prediction information label is obtained by weighting the text prediction information label and the image prediction information label, specifically: Input the text representation and the image representation into the shared classifier to obtain the text prediction information label, the image prediction information label, the text entropy and the image entropy; Through text entropy and image entropy, text weight and image weight are obtained; Based on the text weight and the image weight, the prediction information label is obtained by weighting the text prediction information label and the image prediction information label.

[0010] The present invention provides a multimodal false information detection method, further comprising: Obtain sample multimodal information and true information labels of the sample multimodal information; Inputting the text of the sample multimodal information into a preset text representation network to obtain text representation; Input the text representation into the preset shared classifier to obtain the text prediction information label Calculate the text loss based on the text prediction information label and the true information label, and update the parameters of the text encoder in the text representation network and the preset shared classifier based on the text loss; Inputting the image of the sample multimodal information into a preset image representation network to obtain an image representation; Input the image representation into the shared classifier after text loss update to obtain the image prediction information label; The image classification loss is calculated based on the image prediction information label and the real information label. At the same time, the image contrast loss is obtained by contrastive learning. The parameters of the image encoder in the image representation network and the shared classifier after the text loss update are updated based on the image loss including the image classification loss and the image contrast loss. Iterative training is performed based on the updated preset text representation network, preset image representation network and preset shared classifier until the preset convergence condition is reached to obtain the text representation network, image representation network and shared classifier for multimodal false information detection.

[0011] The text loss is obtained based on the cross entropy loss function, the image classification loss is obtained based on the cross entropy loss function, and the image contrast loss is obtained based on the InfoNCE loss function.

[0012] The updating of the parameters in the preset shared classifier according to the text loss and the updating of the parameters of the shared classifier after the text loss is updated according to the image loss are specifically: Based on the gradient correction mechanism, the preset shared classifier is iteratively trained with text loss and image loss until the preset convergence condition is reached to obtain the shared classifier.

[0013] The present invention also provides a multimodal false information detection system, comprising: A text representation module is used to input the text of multimodal information into a text representation network to obtain text representation; An image representation module, used for inputting an image of multimodal information into an image representation network to obtain an image representation; The prediction module is used to input the text representation and the image representation into the shared classifier to obtain the text prediction information label, the image prediction information label, the text weight and the image weight. Based on the text weight and the image weight, the prediction information label is obtained by weighting the text prediction information label and the image prediction information label.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned multimodal false information detection method when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned multimodal false information detection method are implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The method of the present invention combines the information of the two modalities of text and image, extracts the features of each through the text representation network and the image representation network, and realizes the full utilization of information; then uses a shared classifier to predict the representation of the two modalities; determines the weight coefficients of the two modalities in the final result according to the entropy of the single modality, reflects the reliability of the modality through the weight distribution mechanism, effectively reduces the negative impact of the modality with greater uncertainty on the final result, and improves the accuracy of the prediction result. At the same time, the present invention does not adopt a complex multi-modal fusion mechanism, reduces the computational complexity, improves the reasoning speed, and makes it more efficient in practical applications; when a certain modality is missing, the present invention can still maintain good prediction performance.

[0017] Furthermore, in the training phase, we first extract the respective single-modal representations from the image and text, instead of weighted fusion of the representations at the beginning; then we adopt an alternating training strategy to train the image and text representations independently, and then use the image representation and text representation to classify and predict the samples respectively, and update the network parameters based on the loss calculated according to the prediction results; this ensures that each modality can be fully learned and optimized. At the same time, in view of the limited discrimination ability of the image modality when it is trained alone, a comparative learning mechanism is introduced to improve the prediction effect of the image modality, so that it can capture key features more accurately during the training process. The present invention effectively avoids the problem of modal inertia and enhances the discrimination ability of the model, achieving an overall improvement in model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a schematic diagram of multimodal feature fusion in the prior art; Figure 2 It is a flow chart of the multimodal false information detection method of the present invention; Figure 3 Schematic diagram of the multimodal false information detection method of the present invention; Figure 4 Schematic diagram of multimodal feature fusion in the present invention; Figure 5 This is a structural diagram of the text encoder TextCNN of the present invention; Figure 6 Schematic diagram of the multimodal false information detection method of the present invention; Figure 7 It is a picture comparison learning structure diagram of the present invention; Figure 8 This is a schematic diagram of the structure of a multimodal false information detection system according to a preferred embodiment of the present invention; Fig. 9 It is a schematic diagram of the structure of an electronic device according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to understand the characteristics and effects of the present invention, the following is a general description and definition of the terms and expressions mentioned in the specification and claims. Unless otherwise specified, all technical and scientific terms used in the text are the common meanings understood by those skilled in the art for the present invention. In the event of a conflict, the definition in this specification shall prevail.

[0021] The theories or mechanisms described and disclosed in the present invention, whether correct or incorrect, should not limit the scope of the present invention in any way, that is, the present invention can be implemented without being limited by any specific theory or mechanism.

[0022] In the present invention, all features such as values, quantities, contents and concentrations defined in the form of numerical ranges or percentage ranges are for simplicity and convenience only. Accordingly, the description of numerical ranges or percentage ranges should be deemed to have included and specifically disclosed all possible secondary ranges and individual values ​​within the range (including integers and fractions).

[0023] In the present invention, unless otherwise specified, “comprising”, “including”, “containing”, “having” or similar terms cover the meanings of “consisting of” and “mainly consisting of”, for example, “A includes a” covers the meanings of “A includes a and other” and “A only includes a”.

[0024] like Figure 2~Figure 4 As shown, the present invention provides a multimodal false information detection method, comprising: S1: Input the text of multimodal information into the text representation network to obtain text representation, specifically: The text representation network is obtained by training a preset text representation network, which includes a Bert model and a text encoder; Text of multimodal information A list of consecutive words ,in, is the length of the sequence, and the text is obtained through the Bert model Text vector ,in, for wi After the output of the last hidden layer of Bert, , dw is the dimension of word embedding in Bert; like Figure 6 As shown, TextCNN is used as the text encoder, and the text vector is encoded by the text encoder. Further extraction to obtain text representation , specifically: Set the window size of a convolution kernel to s , convolution kernel text vector Continuous s The word token is convolved to get the i Features.

[0025]

[0026] in, and b c are the weights and constant bias terms of the convolution kernel, respectively. Representative i word token to The concatenation of word tokens, is the ReLu activation function; The convolution kernel will be applied to the text vector All words in the window s are scanned and calculated to obtain the feature map , ; Then use the maximum pooling operation to obtain the feature map The most important information in is taken as the feature of the corresponding convolution kernel, and we get ; In order to obtain different levels of semantics, set n s windows of different sizes, each corresponding to n c convolution kernels, the features of all convolution kernels Concatenate to get text representation ,in .

[0027] S2: Input the image of multimodal information into the image representation network to obtain the image representation, which is specifically: The image representation network is trained by a preset image representation network, which includes a ResNet50 model and an image encoder.

[0028] Images of multimodal information Features , through the ResNet50 model to feature Perform preliminary encoding and use the output of the penultimate layer in the ResNet50 model as the image Image vector ,in , , , The number of channels, width, height of the image and the dimension of the output features of the penultimate layer of ResNet50 respectively; A linear layer encoder is used as the image encoder. The image vector Processing to obtain image representation .

[0029] In an embodiment of the present invention, a ResNet50 model is obtained by using a ResNet model pre-trained with the ImageNet1K dataset.

[0030] S3: Input the text representation and the image representation into the shared classifier to obtain the text prediction information label, the image prediction information label, the text weight and the image weight. Based on the text weight and the image weight, the prediction information label is obtained by weighting the text prediction information label and the image prediction information label. Specifically, The shared classifier is trained by the preset shared classifier.

[0031] Input the text representation and image representation into the shared classifier to obtain the text prediction information label and the image prediction information label, as shown in the following formula:

[0032]

[0033] in, Predict information labels for text, Predict informative labels for images, R t For text representation, R v For image representation, W h , b h is the parameter of the shared classifier h; In order to avoid modal inertia, the features of images and text are not fused, but trained separately. They contribute differently to the final result and should be given different weights. Figure 3 As shown in the figure, the trained text representation network and image representation network are used to obtain the text representation and image representation of multimodal information, and then the shared classifier Generate text prediction information labels and image prediction information labels, and obtain the prediction information labels by weighting, as shown in the following formula:

[0034] in, λ t is the text weight, λ v is the image weight; In an embodiment of the present invention, for a particular mode , the entropy output by the shared classifier is ,in, pm is the probability of the sample being classified into each category; then the text entropy is , the image entropy is .

[0035] In an embodiment of the present invention, for a particular mode , when the modality shows higher uncertainty in the reasoning process, it is more likely to produce wrong predictions, so the weight of the modality on the final result λ m should be smaller; and each mode passes through the shared classifier h The entropy of the output can well reflect this uncertainty. m The larger the output entropy, the lower the credibility of its prediction result, and the smaller the corresponding weight should be. Therefore, the maximum entropy of the two modes can be obtained. With modal m Entropy e m Subtract, and then use softmax to get the mode m Weight λ m , both text weight and image weight can be obtained by the following formula:

[0036] Among them, is the maximum value of the entropy of the text mode and the image mode, e m For modal m The entropy of λ m For modal m The weight of .

[0037] The shared classifier is a picture-text shared classifier of a multi-layer perceptron.

[0038] In an embodiment of the present invention, a text feature extraction network, an image encoder and a shared classifier can be obtained by jointly training a preset text representation network, a preset image representation network and a preset shared classifier, wherein the preset text representation network includes a Bert model and a preset text encoder; the image representation network is obtained by training a preset image representation network, wherein the preset image representation network includes a ResNet50 model and an image encoder.

[0039] like Figure 6 As shown, a multimodal false information detection method provided by an embodiment of the present invention may also include: S401: Obtain sample multimodal information and real information labels of the sample multimodal information; In an embodiment of the present invention, sample multimodal information is obtained from social media as training data, and a multimodal information is preset. ,in, Wis the text of multimodal information, I is an image of multimodal information. The true information label of the sample multimodal information is , the predicted information labels are and ; The multimodal false information detection method is to judge whether the multimodal false information is false information based on the text features and image features in the multimodal information ( ) or true information ( ); S402: Inputting the text of the sample multimodal information into a preset text representation network to obtain text representation; S403: Input the text representation into a preset shared classifier to obtain a text prediction information label S404: Calculate text loss according to the text prediction information label and the true information label, and update the parameters of the text encoder in the text representation network and the preset shared classifier according to the text loss; S405: Inputting the image of the sample multimodal information into a preset image representation network to obtain an image representation; S406: Input the image representation into the shared classifier after the text classification loss is updated to obtain an image prediction information label; S407: Calculate the image classification loss according to the image prediction information label and the real information label; at the same time, use contrastive learning to obtain the image contrast loss, and update the parameters of the image encoder in the image representation network and the shared classifier after the text loss is updated according to the image loss including the image classification loss and the image contrast loss; S408: Based on the updated preset text representation network, preset image representation network and preset shared classifier, repeat steps S402 to S407 for iterative training until the preset convergence condition is reached to obtain the text representation network, image representation network and shared classifier for multimodal false information detection.

[0040] In this embodiment of the present invention, the text loss is calculated by a cross entropy loss function, as shown in the following formula:

[0041] in, loss t For text loss, y is the real information label, Predict informative labels for text In an embodiment of the present invention, the image loss includes image classification loss and image contrast loss, as shown in the following formula:

[0042] in, loss v is the image loss, lossclf is the image classification loss, loss con is the image contrast loss; The image classification loss is calculated by the cross entropy loss function as follows:

[0043] in, loss clf is the image classification loss, y is the real information label, Predict informative labels for images.

[0044] After considering the limitations of using only image features for false information identification, the present invention introduces contrastive learning technology to strengthen the image encoder performance, especially for complex samples that are difficult to distinguish by traditional methods. Figure 6 As shown, the present invention adopts momentum contrast learning to construct a query dictionary by queuing all images of multimodal information of samples to solve the problem of too few samples for comparison, and then uses momentum to update the image encoder , to avoid the problem of inconsistent sample features each time. Specifically, for each sample multimodal information image, first treat it as an anchor sample, and take the result of random dropout of all sample multimodal information images as a positive sample, and select all sample multimodal information images. Other samples except themselves are regarded as negative samples , , all positive and negative samples are used as keys in the query dictionary, using the image encoder Linear layer encoder with the same structure As a momentum encoder, we encode the positive sample features and get In order to shorten the distance between the anchor sample and the positive sample as much as possible and increase the distance between the anchor sample and the negative sample, the vector product is used to measure the similarity between samples, and the InfoNCE loss function is used to calculate the image contrast loss. loss con , as follows:

[0045] in, is the positive sample image representation, Representation of negative sample images; is the anchor point sample image representation.

[0046] It should be noted that a size of The query dictionary is constructed by using a queue to store the features of all negative samples. When each batch of multimodal information images are input in sequence, the features of all negative samples stored in it are Used for the calculation of contrast loss. In addition, the content of the query dictionary is also dynamically replaced and updated. After completing the calculation of each batch, the positive sample features of the current batch are queued as negative samples of the next batch and wait for subsequent use. At the same time, the first batch of data in the queue (that is, the feature vectors of the oldest batch) is dequeued. The query dictionary always stores a small subset of the sampled training data, and it is guaranteed to be the newer data in each training round. In order to reduce computational overhead, the linear layer encoder It does not participate in the parameter update in the back propagation. Therefore, the features of each key stored in the query dictionary queue have no gradient. However, in order to avoid the problem of inconsistent features of the comparison samples, after each batch completes the back propagation, the image encoder after the back propagation is updated in the momentum update method. The parameters are copied to the linear layer encoder In , the momentum is updated as follows:

[0047] In an embodiment of the present invention, the parameters of the preset shared classifier are updated by text loss and image loss, specifically: like Figure 7 As shown in the figure, the image and text modalities use a shared classifier to capture the features of cross-modal fusion. In order to avoid modality forgetting, the present invention introduces a gradient correction mechanism. The text modality and the image modality are trained alternately. When back propagation is used to calculate the gradient in each round of iterative learning, the gradient value of the shared classifier parameters is modified so that the shared classifier can take into account the data in the previous round of training, thereby avoiding forgetting the previous learned modality. Back propagation to update parameters , assuming that the mode currently being learned is , Update according to the following formula:

[0048] in, is the gradient correction matrix, is the parameter of the shared classifier About Loss The gradient of is the learning rate.

[0049] In the initial stage, Updated in the usual gradient descent manner; however, in each subsequent round of backpropagation (i.e. ), gradient correction matrix The corresponding gradient value can be modified before updating the parameters. During the back propagation, all samples in a sample batch pass through the modal m The final representation obtained by the encoder The average value of ,but It can be obtained by the following recursive least squares method, as follows:

[0050] in, is a constant term, is the initial value of the gradient correction matrix, the identity matrix.

[0051] Example In this embodiment, the Chinese short text dataset Weibo21 and the English long text dataset GossipCop are used respectively. Since the present invention mainly discusses the multimodal false information detection algorithm, the plain text samples without pictures are removed in this embodiment, and one of the samples with multiple pictures is randomly selected. Each dataset is randomly divided into training set, validation set and test set in strict accordance with the ratio of 7:1:2.

[0052] The present invention is implemented using Pytorch on Python. This embodiment uses stochastic gradient descent to update parameters, and sets the iteration epochs to 100. For Chinese and English texts, the pre-trained models bert-base-chinese and bert-base-uncased from HuggingFace are used to obtain text vectors. At the same time, the ResNet model pre-trained on the ImageNet1K dataset in PyTorch is used to extract image features. The parameters of all the above pre-trained models remain fixed during training and do not participate in gradient calculation and parameter update. The server memory used in the experiment is 90GB, the CPU is a 12-core Xeon(R) Platinum 8352V, and the GPU is Nvidia GeForce RTX 4090 (24GB).

[0053] Based on repeated experiments on the dataset, we set the optimal network structure and hyperparameters, as shown in Table 1.

[0054] Table 1 Experimental settings

[0055] In order to illustrate the effectiveness and advantages of the present invention, a series of comparisons are made between the present invention and existing methods. Reference [1]: SpotFake: A Multi-Modal Framework for Fake News Detection. This model uses the pre-trained Bert model and VGG model to extract the features of text and images respectively, and then concatenates the two to form the final representation and feeds it into the classifier. Reference [2]: EANN: Event Adversarial Neural Networks for Multi-Modal FakeNews Detection. Based on the concatenation of single-modal features, this model uses adversarial training to avoid overfitting on samples in a certain field, thereby improving the generalization ability of the model. Reference [3]: Multimodal Fusion with Co-Attention Networks for Fake News Detection. This model uses discrete cosine transform to extract the frequency domain features of the image, and then stacks multiple co-attention layers to continuously fuse them with the text features and the spatial domain features of the image to obtain a multimodal representation of the sample. Reference [4]: ​​Hierarchical Multi-Modal Contextual Attention Network for Fake News Detection, which divides the adjacent layers of Bert's 12 hidden layers into three groups on average and accumulates their outputs to obtain richer hierarchical semantic features, and then uses complex upper and lower Transformers to complete multimodal fusion. Reference [5]: Cross-Modal Ambiguity Learning for Multimodal Fake News Detection, this model additionally introduces the auxiliary task of cross-modal alignment, uses variational autoencoders to learn cross-modal ambiguity, and calculates the KL divergence of the distribution of different modal data as the weight for the fusion of single-modal and multi-modal features.

[0056] The present invention selects the classic evaluation indicators of accuracy, precision, recall and F1 as the evaluation criteria for classification comparison. By comparing on two real-world data sets, Weibo21 and GossipCop, the results are shown in Table 2. As shown in Table 2, the present invention adopts the method of alternating training of images and texts to avoid weighted fusion of multimodal features during training, which improves the classification effect of the model to a certain extent. The experimental data is significantly better than other models, and the highest accuracy rates of 0.933 and 0.884 were achieved on Weibo21 and GossipCop, respectively, and good results were also achieved in the three indicators of precision, recall and F1 value. Among all the methods, SpotFake has the worst effect because it only uses the simplest way to splice images and texts, and cannot fully learn multimodal features. EANN, which also uses the splicing method, improves the generalization ability of the model to a certain extent due to the use of event adversarial training. MCAN, HMCAN and CAFE all use the attention mechanism to learn richer multimodal features of images and texts, and have achieved good results. MCAN takes into account the frequency domain information of tampered or forged images, and achieved a recall rate of 0.935 for fake information on Weibo21. HMCAN can learn hierarchical semantic features, and achieved a precision rate of 0.943 for real information on Weibo21. CAFE uses KL divergence to balance unimodal and multimodal features, and ranked first in both the false information precision rate and the real information recall rate of GossipCop. However, the above baseline models cannot avoid the problem of modal inertia during training, and still cannot fully learn the features of images and texts.

[0057] Table 2 Comparative experimental results of the present invention and the existing method

[0058] In order to analyze the role of each key module in the present invention, the present invention designs the following different variants for ablation experiments. 1) w / o T: remove the text classification module and only use the image for classification training; 2) w / o I: remove the image classification module and only use the text for classification training; 3) w / o C: remove the contrastive learning module in the image classification training; 4) w / o S: do not use the shared classifier in the alternating image and text training , but each uses a separate classifier to output the final result; 5) w / o G: No gradient correction is performed in the shared classifier for images and texts; 6) w / o E: During prediction and reasoning, entropy is not used as a dynamic fusion mechanism for weights, but the classification results of the image and text outputs are directly added as the final result. The experimental results are shown in Table 3. After removing different key modules, the effects of these variants are not good. The effects of w / o T without the text module and w / o I without the image module on both datasets have dropped significantly, especially the accuracy of w / o T is even less than 80%, which shows that text information is more important for the multimodal false information detection task. After removing the shared classifier, the accuracy also dropped by about 2 percentage points, because it lost important information about the cross-modal interaction between images and texts, while the information provided by single-modal features is quite limited. At the same time, the gradient correction mechanism in the shared classifier also plays a big role in solving the problem of modal forgetting. The accuracy of w / o G without gradient correction on Weibo21 and GossipCop datasets is only 0.913 and 0.877 respectively. In the reasoning stage, the present invention uses the entropy of each modal output as the modal weight to dynamically fuse the image and text. w / o E simply adds the output of each image and text, and simply and crudely treats the two completely the same, ignoring the different contributions of images and text to the false information detection task. The performance on Weibo21 is not even as good as w / o S without the shared classifier and w / o G without gradient correction.

[0059] Table 3 Ablation experimental results of the present invention

[0060] like Figure 8 As shown, another object of the present invention is to provide a multimodal false information detection system, comprising: A text representation module is used to input the text of multimodal information into a text representation network to obtain text representation; An image representation module, used for inputting an image of multimodal information into an image representation network to obtain an image representation; The prediction module is used to input the text representation and the image representation into the shared classifier to obtain the text prediction information label, the image prediction information label, the text weight and the image weight. Based on the text weight and the image weight, the prediction information label is obtained by weighting the text prediction information label and the image prediction information label.

[0061] like Fig. 9 As shown, the third object of the present invention is to provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multimodal false information detection method when executing the computer program.

[0062] The multimodal false information detection method comprises: Input the text of multimodal information into the text representation network to obtain text representation; Input the image of multimodal information into the image representation network to obtain the image representation; The text representation and the image representation are input into a shared classifier to obtain text prediction information labels, image prediction information labels, text weights, and image weights. Based on the text weights and image weights, the prediction information labels are weighted by the text prediction information labels and the image prediction information labels to obtain prediction information labels.

[0063] A fourth object of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal false information detection method are implemented.

[0064] The multimodal false information detection method comprises: Input the text of multimodal information into the text representation network to obtain text representation; Input the image of multimodal information into the image representation network to obtain the image representation; The text representation and the image representation are input into a shared classifier to obtain text prediction information labels, image prediction information labels, text weights, and image weights. Based on the text weights and image weights, the prediction information labels are weighted by the text prediction information labels and the image prediction information labels to obtain prediction information labels.

[0065] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0067] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal false information detection method, characterized in that: include: Input the text of multimodal information into the text representation network to obtain text representation; Input the image of multimodal information into the image representation network to obtain the image representation; The text representation and the image representation are input into a shared classifier to obtain text prediction information labels, image prediction information labels, text weights, and image weights. Based on the text weights and image weights, the prediction information labels are weighted by the text prediction information labels and the image prediction information labels to obtain prediction information labels.

2. A multimodal false information detection method according to claim 1, characterized in that: The text representation network includes a Bert model and a text encoder. The text of the multimodal information is input into the text representation network to obtain the text representation, which is specifically: The text of multimodal information is obtained through the Bert model to obtain text vectors; The text vector is processed based on a text encoder to obtain a text representation, and the text encoder adopts TextCNN.

3. A multimodal false information detection method according to claim 1, characterized in that: The image representation network includes a ResNet50 model and an image encoder. The image of multimodal information is input into the image representation network to obtain image representation, specifically: The image with multimodal information is passed through the ResNet50 model to obtain the image vector; The image vector is processed based on an image encoder to obtain an image representation, and the image encoder adopts a linear layer encoder.

4. A multimodal false information detection method according to claim 1, characterized in that: The text representation and the image representation are input into the shared classifier to obtain the text prediction information label, the image prediction information label, the text weight and the image weight. Based on the text weight and the image weight, the prediction information label is obtained by weighting the text prediction information label and the image prediction information label, specifically: Input the text representation and the image representation into the shared classifier to obtain the text prediction information label, the image prediction information label, the text entropy and the image entropy; Through text entropy and image entropy, text weight and image weight are obtained; Based on the text weight and the image weight, the prediction information label is obtained by weighting the text prediction information label and the image prediction information label.

5. A multimodal false information detection method according to claim 1, characterized in that: Also includes: Obtain sample multimodal information and true information labels of the sample multimodal information; Inputting the text of the sample multimodal information into a preset text representation network to obtain text representation; Input the text representation into the preset shared classifier to obtain the text prediction information label Calculate the text loss based on the text prediction information label and the true information label, and update the parameters of the text encoder in the text representation network and the preset shared classifier based on the text loss; Inputting the image of the sample multimodal information into a preset image representation network to obtain an image representation; Input the image representation into the shared classifier after text loss update to obtain the image prediction information label; The image classification loss is calculated based on the image prediction information label and the real information label. At the same time, the image contrast loss is obtained by contrastive learning. The parameters of the image encoder in the image representation network and the shared classifier after the text loss update are updated based on the image loss including the image classification loss and the image contrast loss. Iterative training is performed based on the updated preset text representation network, preset image representation network and preset shared classifier until the preset convergence condition is reached to obtain the text representation network, image representation network and shared classifier for multimodal false information detection.

6. A multimodal false information detection method according to claim 5, characterized in that: The text loss is obtained based on the cross entropy loss function, the image classification loss is obtained based on the cross entropy loss function, and the image contrast loss is obtained based on the InfoNCE loss function.

7. A multimodal false information detection method according to claim 5, characterized in that: The updating of the parameters in the preset shared classifier according to the text loss and the updating of the parameters of the shared classifier after the text loss is updated according to the image loss are specifically: Based on the gradient correction mechanism, the preset shared classifier is iteratively trained with text loss and image loss until the preset convergence condition is reached to obtain the shared classifier.

8. A multimodal false information detection system, characterized in that: include: A text representation module is used to input the text of multimodal information into a text representation network to obtain text representation; An image representation module, used for inputting an image of multimodal information into an image representation network to obtain an image representation; The prediction module is used to input the text representation and the image representation into the shared classifier to obtain the text prediction information label, the image prediction information label, the text weight and the image weight. Based on the text weight and the image weight, the prediction information label is obtained by weighting the text prediction information label and the image prediction information label.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multimodal false information detection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the multimodal false information detection method according to any one of claims 1 to 7.

Images