Multimodal false news detection method, device, equipment and medium
Through joint decision-making of multi-modal fake news models, combined with images, title text, OCR text and associated text, the problem of poor accuracy in multimodal fake news detection in the existing technology is solved, and more efficient false news recognition is achieved.
Patent Information
- Application Number
- CN202411249513.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, deep learning models that rely solely on cross-modal consistency or a single function to detect multimodal fake news, the detection accuracy is poor and it is impossible to effectively deal with the various formation mechanisms and expressions of false news.
Using multimodal fake news detection method, by obtaining images, title text, OCR text and associated text, input text analysis model, graphic comparison model, knowledge verification model and visual analysis model, comprehensive prediction scores for detection, and use multivariate experts The model makes joint decisions.
It improves the accuracy of fake news detection, adapts to the multimodal characteristics of news samples, has dynamic decision-making capabilities, and can more effectively capture and identify cunning and hidden fake news characteristics.
Smart Images

Figure CN118762377B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of information detection technology, and specifically to a multimodal false news detection method, device, equipment and computer-readable storage medium. Background Art
[0002] With the rapid development of social media platforms, the number of its users has also increased rapidly. Some users create false news for various purposes. These false news are spread quickly and affect public judgment. How to accurately identify false news is becoming more and more important.
[0003] In the related technology, detection is mainly performed through the following methods: one is to use a single-function deep learning model to extract text and image features respectively, and fuse the text and image features, and perform detection based on the fused features; the other is to perform detection by identifying the cross-modal consistency between text and image.
[0004] However, the formation mechanism and manifestation forms of fake news are very rich. The formation mechanism involves various means such as text tampering, image tampering, exaggeration, misleading, etc.; the manifestation forms include screenshots, real photos or animations, etc. Given that the formation mechanism and manifestation forms of fake news present various situations, if we only rely on cross-modal consistency or single-function deep learning models to detect multimodal fake news, the detection accuracy will be poor. Summary of the invention
[0005] The present application provides a multimodal fake news detection method, apparatus, device and computer-readable storage medium, which can solve the technical problem in the prior art that multimodal fake news detection only relies on cross-modal consistency or a single-function deep learning model, resulting in poor detection accuracy.
[0006] In a first aspect, an embodiment of the present application provides a multimodal false news detection method, the multimodal false news detection method comprising:
[0007] Obtain target data of the news to be tested, the target data including images, title text, OCR text and associated text, wherein the OCR text is text obtained by recognizing the news to be tested through optical character recognition technology;
[0008] Input the title text and OCR text into the text analysis model and output the first prediction score;
[0009] Input the title text and the image into the image-text comparison model and output a second prediction score;
[0010] Input the title text and the associated text into the knowledge verification model and output a third prediction score;
[0011] Input the title text and the image into the visual analysis model and output a fourth prediction score;
[0012] The first prediction score, the second prediction score, the third prediction score and the fourth prediction score are combined to determine the detection result of the news to be tested.
[0013] In combination with the first aspect, in one implementation, the title text and the OCR text are input into a text analysis model, and a first prediction score is output, including:
[0014] Input the title text and OCR text into BERT to generate the first word embedding vector and the second word embedding vector. BERT is a deep bidirectional encoder.
[0015] The first word embedding vector is processed using a masked self-attention network to obtain a first sentence-level embedding vector;
[0016] The second word embedding vector is processed using a masked self-attention network to obtain a second sentence-level embedding vector;
[0017] Based on the first word embedding vector and the second word embedding vector, the first output feature of Text-CNN is determined. The corresponding formula is:
[0018] ,
[0019] Among them, w t is the first word embedding vector, w o is the second word embedding vector, is the learnable parameter of the i-th Text-CNN, r i is the first output feature, For splicing symbols, Text-CNN i is the i-th convolutional neural network;
[0020] Based on the first sentence-level embedding vector and the second sentence-level embedding vector, the first post-gate weight is determined. The corresponding formula is:
[0021] ,
[0022] Among them, e t is the first sentence level embedding vector, e o is the second sentence-level embedding vector, φ is the learnable parameter of the gating network, and w i is the weight after the first gate, softmax is the activation function, and G is the gated network;
[0023] Based on the first output feature and the first post-gate weight, the first prediction score is determined, and the corresponding formula is:
[0024] ,
[0025] in, is the first prediction score and MLP is the classifier.
[0026] In combination with the first aspect, in one implementation, inputting the title text and the image into the image-text comparison model and outputting the second prediction score includes:
[0027] Input the title text into BERT to generate the first classification vector;
[0028] Input the image into Vision Transformer to generate the second classification vector. Vision Transformer is a visual encoder.
[0029] Perform dimensionality reduction on the first classification vector and the second classification vector respectively;
[0030] Connect the first classification vector after dimensionality reduction and the second classification vector after dimensionality reduction to obtain a multimodal feature;
[0031] The multimodal features are processed using an activation function to obtain a second prediction score.
[0032] In combination with the first aspect, in one implementation, the title text and the associated text are input into the knowledge verification model, and the third prediction score is output, including:
[0033] Input the title text and related text into BERT to generate the third word embedding vector and the fourth word embedding vector;
[0034] The third word embedding vector is processed using a masked self-attention network to obtain a third sentence-level embedding vector;
[0035] The fourth word embedding vector is processed using the masked self-attention network to obtain the fourth sentence-level embedding vector;
[0036] Based on the third word embedding vector and the fourth word embedding vector, determine the second output feature of Text-CNN;
[0037] Determine a second post-gate weight based on the third sentence-level embedding vector and the fourth sentence-level embedding vector;
[0038] Based on the second output feature and the second post-gated weight, a third prediction score is determined.
[0039] In combination with the first aspect, in one implementation, inputting the title text and the image into a visual analysis model and outputting a fourth prediction score includes:
[0040] Extract the query vector of the last hidden state of the image;
[0041] Extract the key vector and value vector of the last hidden state of the title text;
[0042] Based on the query vector, key vector and value vector, determine the image embedding vector that interacts with the title text. The corresponding formula is:
[0043] w f = ,
[0044] Among them, w f is the image embedding vector, Q is the query vector, K is the key vector, V is the value vector, D is the feature dimension, T is the matrix transpose, Attention is the attention function, and softmax is the activation function;
[0045] The image embedding vector is optimized by using a hybrid expert network to obtain an optimized image embedding vector;
[0046] Based on the optimized image embedding vector, the fourth prediction score is determined, and the corresponding formula is:
[0047] ,
[0048] Among them, e f is the optimized image embedding vector, is the fourth prediction score, and MLP is the classifier.
[0049] In combination with the first aspect, in one implementation, the first prediction score, the second prediction score, the third prediction score, and the fourth prediction score are combined to determine the detection result of the news to be tested, including:
[0050] Calculate the differences between the first prediction score, the second prediction score, the third prediction score, the fourth prediction score and the score threshold respectively, and take the absolute values to obtain the calculation results;
[0051] Determine the maximum value among all calculation results, and use the prediction score corresponding to the maximum value as the target prediction score;
[0052] Compare the target prediction score with the score threshold to obtain a comparison result;
[0053] Based on the comparison result, the detection result of the news to be tested is determined.
[0054] In combination with the first aspect, in one implementation, determining the detection result of the news to be tested based on the comparison result includes:
[0055] If the comparison result is that the target prediction score is greater than the score threshold, it is determined that the detection result of the news to be tested is fake news;
[0056] If the comparison result is that the target prediction score is less than or equal to the score threshold, it is determined that the detection result of the news to be tested is true news.
[0057] In a second aspect, an embodiment of the present application provides a multimodal false news detection device, the multimodal false news detection device comprising:
[0058] An acquisition module is used to acquire target data of the news to be tested, wherein the target data includes images, title texts, OCR texts and associated texts, wherein the OCR texts are texts obtained by recognizing the news to be tested through optical character recognition technology;
[0059] A first input-output module, used for inputting the title text and the OCR text into the text analysis model, and outputting a first prediction score;
[0060] A second input-output module, used for inputting the title text and the image into the image-text comparison model, and outputting a second prediction score;
[0061] A third input-output module, used for inputting the title text and the associated text into the knowledge verification model, and outputting a third prediction score;
[0062] a fourth input-output module, configured to input the title text and the image into the visual analysis model and output a fourth prediction score;
[0063] The determination module is used to comprehensively determine the detection result of the news to be tested by combining the first prediction score, the second prediction score, the third prediction score and the fourth prediction score.
[0064] In a third aspect, an embodiment of the present application provides a multimodal false news detection device, comprising a processor, a memory, and a multimodal false news detection program stored on the memory and executable by the processor, wherein when the multimodal false news detection program is executed by the processor, the steps of the multimodal false news detection method as described in any one of the first aspects are implemented.
[0065] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a multimodal false news detection program is stored. When the multimodal false news detection program is executed by a processor, the steps of the multimodal false news detection method as described in any one of the first aspects are implemented.
[0066] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:
[0067] By obtaining target data of the news to be tested, the target data includes images, title text, OCR text and associated text; inputting the title text and OCR text into the text analysis model, outputting the first prediction score; inputting the title text and image into the image-text comparison model, outputting the second prediction score; inputting the title text and associated text into the knowledge verification model, outputting the third prediction score; inputting the title text and image into the visual analysis model, outputting the fourth prediction score; comprehensively combining the first prediction score, the second prediction score, the third prediction score and the fourth prediction score to determine the detection result of the news to be tested, realizing joint decision-making using a multivariate expert model, adapting well to the multimodal characteristics of news samples, and having dynamic decision-making capabilities, so as to more effectively capture and identify cunning and hidden characteristics of false news. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] Figure 1 This is a flowchart of the first embodiment of the multimodal fake news detection method of the present application;
[0069] Figure 2 This is a functional module diagram of an embodiment of a multimodal fake news detection device of the present application;
[0070] Figure 3 This is a schematic diagram of the hardware structure of a multimodal fake news detection device involved in the embodiment of the present application. DETAILED DESCRIPTION
[0071] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0072] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0073] In a first aspect, an embodiment of the present application provides a multimodal fake news detection method.
[0074] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the multimodal fake news detection method of this application. Figure 1 As shown, the multimodal fake news detection method includes:
[0075] Step 110, obtaining target data of the news to be tested, the target data including images, title text, OCR text and associated text;
[0076] The news to be tested is a multimodal news in the form of "text-image" news pairs. Text recognition technology is used to recognize the title text in the news to be tested, and image recognition technology is used to recognize the image in the news to be tested. Optical character recognition technology (OCR) is further used to recognize OCR text in the image. The OCR text is the text obtained by recognizing the news to be tested through optical character recognition technology. The title text is input into the large language model to obtain the associated text of the news to be tested.
[0077] Among them, the title text includes the title and text of the news to be tested; the OCR text includes the text in the image and the publisher's watermark; the associated text is the report related to the news to be tested; the large language model includes but is not limited to ERNIE-Bot-4.
[0078] When using a large language model, the prompt words used are: "This is a message posted by a media user: [news title]. Do you think this message is false, such as fake, rumor-mongering, inconsistent with common sense, etc. Based on your knowledge, tell us your reasoning." [News title] is replaced by the specific news text. The supplementary information and reasoning results output by the large language model are the corresponding associated text.
[0079] Step 120: Input the title text and the OCR text into a text analysis model, and output a first prediction score;
[0080] The text analysis model is implemented based on the hybrid expert network architecture, which is obtained by continuously optimizing and training the text analysis model based on the loss function of the predicted label and the real label. The sub-experts of the hybrid expert network use the Text-CNN structure, which is a convolutional neural network.
[0081] During the specific implementation, the title text and OCR text are input into the text analysis model. The text analysis model combines all available texts from the news to be tested, including: title, text, OCR text in pictures, etc., and evaluates from the text perspective to determine whether the news to be tested contains false traces such as exaggeration and misleading in the text, and outputs the first prediction score.
[0082] Step 130: input the title text and the image into the image-text comparison model, and output a second prediction score;
[0083] Among them, the image-text comparison model trains a shared projection head (Projection Head), which is a two-layer linear layer structure with ReLU activation function activated in the middle to minimize the distance between text embedding and image embedding in real samples and maximize the distance of false samples.
[0084] In the specific implementation, the title text and image are input into the image-text comparison model, and the authenticity of the news pair to be tested is evaluated from the perspective of cross-modal consistency, and the second prediction score is output; in order to achieve cross-modal consistency learning, samples with high consistency and samples with low consistency are required, and the existing news dataset lacks samples with low consistency. In this context, this application proposes that cross-modal consistency learning can be achieved by constructing an auxiliary consistency dataset. We sample real news pairs from the original news dataset and cross-exchange the sampled news pairs to construct Specifically, two real news Cross-matched to form mixed pairs and As negative sample pairs, the original pair of news is regarded as a positive sample pair with higher consistency. j is the image of the jth news in the original news dataset, T j is the text of the jth news in the original news dataset, I k is the image of the kth news in the original news dataset, T k is the text of the kth news in the original news dataset, D real is the original news collection. It contains positive samples and negative samples, where the positive samples come from the real news in the original data set, and the negative samples are obtained by cross-exchanging two real news. The negative samples in , whose text and image come from different pairs, may naturally have low consistency.
[0085] Step 140: input the title text and the associated text into the knowledge verification model, and output a third prediction score;
[0086] Among them, the knowledge verification model is also implemented based on the hybrid expert network architecture. The difference between the knowledge verification model and the text analysis model is that the inputs of the knowledge verification model are different. The inputs of the knowledge verification model are title text and associated text, while the inputs of the text analysis model are title text and OCR text. That is, only the OCR text is replaced with the associated text output by the large language model. The sub-experts of the hybrid expert network also use the Text-CNN structure. The knowledge verification model uses external knowledge and combines the large language model to make inferences and judgments.
[0087] For example, the title of a news article is "In 2010, 24 million kilograms of apples disappeared in a certain area", and the picture and title also match. Attempts to predict the authenticity of this news only from the perspective of image, text, cross-modal consistency, and OCR text have all ended in failure. When dealing with this news, the knowledge verification model of the present application infers that the "missing" in the title does not mean "disappeared", but means "during the blooming period of apple trees in 2010, the area frequently encountered hail weather, resulting in a decrease in apple production, which was 24 million kilograms less than in previous years, not 24 million kilograms of apples missing", and successfully detected this fake news. This example illustrates that in order to better detect fake news, it is necessary to use the reasoning ability of a large language model.
[0088] In specific implementation, the title text and associated text are input into the knowledge verification model. The knowledge verification model combines the title text and associated text from the news to be tested, evaluates from a textual perspective, determines the authenticity of the news to be tested, and outputs a third prediction score.
[0089] Step 150: Input the title text and the image into the visual analysis model, and output a fourth prediction score;
[0090] It is very challenging to detect fake news relying solely on images. The visual analysis model of this application focuses on detecting compressed and tampered images. For example, we observed the T-SNE dimensionality reduction visualization in the feature space when linear probing was performed using text features encoded by the BERT deep bidirectional encoder and image features encoded by the Vision Transformer visual encoder. The text features show clearer category boundaries, while the image features are more noisy. This difference between category clarity and noise level has been observed in different news datasets. This shows that relying solely on image information for effective fake news detection is not enough. The natural noise in news images can affect the ability of visual analysis experts to make judgments, so text is needed to provide assistance.
[0091] In the specific implementation, the title text and image are input into the visual analysis model. The visual analysis model mainly uses image information, and introduces text information as a supplement through the attention mechanism from image to text for joint judgment, and outputs the fourth prediction score.
[0092] Step 160: Determine the detection result of the news to be tested by combining the first prediction score, the second prediction score, the third prediction score and the fourth prediction score.
[0093] Multiple expert models are pre-trained using multiple information sources such as text, images, cross-modal consistency, OCR text, and external knowledge provided by a large language model, namely the text analysis model, image-text comparison model, knowledge verification model, and visual analysis model.
[0094] A multi-expert model consisting of sub-expert models such as text analysis model, image-text comparison model, knowledge verification model and visual analysis model is used for joint decision-making. The prediction scores output by each model are integrated, and the dynamic controller selects the appropriate expert model according to predefined rules to determine the detection results of the news to be tested. This process ensures that each news to be tested is comprehensively analyzed and an adaptability report of the news to be tested is obtained. Since each expert model is responsible for analyzing the causes and clues of a type of false news. This decoupled model design allows this multi-expert network to make full use of the information of each modality to make a more comprehensive decision, thereby more accurately detecting multimodal false news.
[0095] Among them, the dynamic controller can adaptively select the expert model according to the characteristics of each news sample and integrate the prediction results of the selected expert model.
[0096] In this embodiment, by obtaining target data of the news to be tested, the target data includes images, title text, OCR text and associated text; inputting the title text and OCR text into a text analysis model, and outputting a first prediction score; inputting the title text and image into a picture-text comparison model, and outputting a second prediction score; inputting the title text and associated text into a knowledge verification model, and outputting a third prediction score; inputting the title text and image into a visual analysis model, and outputting a fourth prediction score; comprehensively combining the first prediction score, the second prediction score, the third prediction score and the fourth prediction score, the detection result of the news to be tested is determined, and joint decision-making is achieved using a multivariate expert model, which is well adapted to the multimodal characteristics of news samples and has dynamic decision-making capabilities, so as to more effectively capture and identify cunning and hidden characteristics of false news.
[0097] Furthermore, in one embodiment, the title text and the OCR text are input into a text analysis model, and a first prediction score is output, including:
[0098] Step 210: Input the title text and the OCR text into BERT to generate a first word embedding vector and a second word embedding vector, where BERT is a deep bidirectional encoder.
[0099] Optionally, the title text and OCR text can be directly input into BERT, or the OCR text can be cleaned first, and then the cleaned OCR text and title text are marked and then input into BERT to generate the first word embedding vector w t and the second word embedding vector w o .
[0100] Step 220: Use the masked self-attention network to process the first word embedding vector to obtain the first sentence-level embedding vector e t ;
[0101] Step 230: Use the masked self-attention network to process the second word embedding vector to obtain the second sentence-level embedding vector e o ;
[0102] In specific implementation, the following formulas are used to process the first word embedding vector and the second word embedding vector respectively:
[0103] ,
[0104] ,
[0105] Among them, Mask_Attention is the masked self-attention network;
[0106] Step 240: Determine the first output feature of Text-CNN based on the first word embedding vector and the second word embedding vector. The corresponding formula is:
[0107] ,
[0108] Among them, w t is the first word embedding vector, w o is the second word embedding vector, is the learnable parameter of the i-th Text-CNN, r i is the first output feature, For splicing symbols, Text-CNN i is the i-th convolutional neural network;
[0109] In order to better integrate OCR features and text information during text analysis, the text analysis model uses multiple sub-experts and gating for adaptive fusion. Each sub-expert adopts a Text-CNN architecture with different convolution kernels to adapt to news texts of different lengths, thereby enhancing the generalization ability of the network. This application does not specifically limit the number of sub-experts and the size of the convolution kernel. Preferably, in the embodiment, 5 Text-CNNs are used as sub-experts. The convolution kernel sizes of these sub-experts are designed to be 1, 2, 3, 5, and 10, respectively.
[0110] Step 250: Determine the first post-gate weight based on the first sentence-level embedding vector and the second sentence-level embedding vector. The corresponding formula is:
[0111] ,
[0112] Among them, et is the first sentence level embedding vector, e o is the second sentence-level embedding vector, φ is the learnable parameter of the gated network, wi is the weight after the first gate, softmax is the activation function, and G is the gated network;
[0113] Gated Network It is a feedforward network. After passing the softmax activation function, the weight w after each gate i Represents the normalized weight of the corresponding expert.
[0114] Step 260: Determine a first prediction score based on the first output feature and the first post-gate weight. The corresponding formula is:
[0115] ,
[0116] in, is the first prediction score and MLP is the classifier.
[0117] Furthermore, the output feature r of Text-CNN is i AND gate weight w i After weighting, it passes through the MLP classifier and then the activation function to determine the prediction score. The MLP classifier contains two linear layers and a ReLU activation function. is the prediction score, whose value is between 0 and 1, and the true label y is 0 or 1, 0 represents true news and 1 represents false news.
[0118] The learnable parameters of Text-CNN, the learnable parameters of the gated network, and the parameters of the classifier are optimized by the cross entropy classification loss function between the predicted score and the true label y. The classification loss function is:
[0119] ,
[0120] in, is the classification loss.
[0121] In this embodiment, the title text and the OCR text are input into BERT to generate a first word embedding vector and a second word embedding vector; the first word embedding vector is processed by a masked self-attention network to obtain a first sentence-level embedding vector; the second word embedding vector is processed by a masked self-attention network to obtain a second sentence-level embedding vector; based on the first word embedding vector and the second word embedding vector, a first output feature of Text-CNN is determined; based on the first sentence-level embedding vector and the second sentence-level embedding vector, a first post-gate weight is determined; based on the first output feature and the first post-gate weight, a first prediction score is determined. The model uses multiple sub-experts and gated networks for adaptive fusion, so that OCR features and text information can be better integrated in text analysis. Each sub-expert uses a Text-CNN architecture with different convolution kernels to extract features from fine-grained to coarse-grained to adapt to news texts of different lengths, thereby enhancing the generalization ability of the network.
[0122] Furthermore, in one embodiment, the title text and the image are input into the image-text comparison model, and the second prediction score is output, including:
[0123] Step 310: Input the title text into BERT to generate a first classification vector;
[0124] Step 320: Input the image into Vision Transformer to generate a second classification vector, where Vision Transformer is a visual encoder.
[0125] In the specific implementation, the title text is input into BERT, the title text is classified, and the first classification vector p is generated. t ; Input the image into Vision Transformer, classify the image, and generate the second classification vector p o .
[0126] Step 330, respectively reducing the dimension of the first classification vector and the second classification vector;
[0127] In a specific implementation, a projection function is used to reduce the dimension of the first classification vector and the second classification vector so as to align the two in the same feature space.
[0128] Step 340: Connect the first classification vector after dimensionality reduction and the second classification vector after dimensionality reduction to obtain a multimodal feature;
[0129] Project the first classification vector after dimensionality reduction (p t ) and the second classification vector projection after dimensionality reduction (p o) is concatenated, that is, the two vectors are combined and spliced together to form a new vector, and then the multimodal feature v can be obtained after average pooling.
[0130] Step 350: Process the multimodal features using an activation function to obtain a second prediction score.
[0131] Furthermore, a multi-layer perceptron (MLP) network with a softmax output layer is used as a classifier. The multimodal features are processed using an activation function to obtain the second prediction score, and the corresponding formula is:
[0132]
[0133] Among them, the optimization of the image-text comparison model is achieved through the alignment loss and classification loss, that is,
[0134] , is calculated in the same way as the text analysis model. It is obtained by the following formula:
[0135]
[0136] Among them, positive refers to p t and p o From the same news; negative refers to p t and p o They come from different news respectively, d is a calibration parameter and can be flexibly set according to actual needs. In this embodiment, d is set to 0.2.
[0137] In this embodiment, the title text is input into BERT to generate a first classification vector; the image is input into VisionTransformer to generate a second classification vector; the first classification vector and the second classification vector are respectively reduced in dimension; the first classification vector after dimension reduction and the second classification vector after dimension reduction are connected to obtain a multimodal feature; the multimodal feature is processed using an activation function to obtain a second prediction score, thereby evaluating the authenticity of the news pair from the perspective of cross-modal consistency.
[0138] Furthermore, in one embodiment, the title text and the associated text are input into the knowledge verification model, and the third prediction score is output, including:
[0139] Step 410: Input the title text and the associated text into BERT to generate a third word embedding vector and a fourth word embedding vector;
[0140] Step 420: Process the third word embedding vector using the masked self-attention network to obtain a third sentence-level embedding vector;
[0141] Step 430: Process the fourth word embedding vector using the masked self-attention network to obtain a fourth sentence-level embedding vector;
[0142] Step 440: Determine a second output feature of Text-CNN based on the third word embedding vector and the fourth word embedding vector;
[0143] Step 450: Determine a second post-gate weight based on the third sentence-level embedding vector and the fourth sentence-level embedding vector;
[0144] Step 460: Determine a third prediction score based on the second output feature and the second post-gated weight.
[0145] It should be noted that the difference between the knowledge verification model and the text analysis model is that their inputs are different. The input of the knowledge verification model is the title text and the associated text, while the input of the text analysis model is the title text and the OCR text. That is, only the OCR text is replaced with the associated text output by the large language model. The process of determining the third prediction score by the knowledge verification model is the same as steps 210-260, which will not be repeated here.
[0146] In this embodiment, a third word embedding vector and a fourth word embedding vector are generated by inputting the title text and the associated text into BERT; the third word embedding vector is processed by a masked self-attention network to obtain a third sentence-level embedding vector; the fourth word embedding vector is processed by a masked self-attention network to obtain a fourth sentence-level embedding vector; based on the third word embedding vector and the fourth word embedding vector, a second output feature of Text-CNN is determined; based on the third sentence-level embedding vector and the fourth sentence-level embedding vector, a second post-gate weight is determined; based on the second output feature and the second post-gate weight, a third prediction score is determined. The model uses multiple sub-experts and gated networks for adaptive fusion, so that the title text and the associated text are better integrated in text analysis. Each sub-expert uses a Text-CNN architecture with different convolution kernels to extract features from fine-grained to coarse-grained to adapt to news texts of different lengths, thereby enhancing the generalization ability of the network.
[0147] Furthermore, in one embodiment, the title text and the image are input into the visual analysis model, and a fourth prediction score is output, including:
[0148] Step 510, extracting the query vector of the last hidden state of the image;
[0149] Step 520: extract the key vector and value vector of the last hidden state of the title text;
[0150] In specific implementation, the image is input into Vision Transformer, so that the query vector Q of the last hidden state of the image can be extracted; the title text is input into BERT, so that the key vector K and value vector V of the last hidden state of the title text can be extracted.
[0151] Step 530: Determine the image embedding vector that interacts with the title text based on the query vector, the key vector, and the value vector. The corresponding formula is:
[0152] w f = ,
[0153] Among them, w f is the image embedding vector, Q is the query vector, K is the key vector, V is the value vector, D is the feature dimension, T is the matrix transpose, Attention is the attention function, and softmax is the activation function;
[0154] Further, an attention function is performed on the normalized query vector, key vector, and value vector to obtain the image embedding vector w that interacts with the title text. f .
[0155] Step 540: Optimize the image embedding vector using the hybrid expert network to obtain an optimized image embedding vector;
[0156] Step 550: Determine a fourth prediction score based on the optimized image embedding vector, and the corresponding formula is:
[0157] ,
[0158] Among them, e f is the optimized image embedding vector, is the fourth prediction score, and MLP is the classifier.
[0159] The visual analysis model still uses the hybrid expert network to optimize features. The input feature of the hybrid expert network is w f ; The gated network is a linear layer – ReLU activation – linear layer – softmax output layer structure; The input accepted by the gated network is a pair of w f The output features after performing self-attention. The features optimized by the hybrid expert network are sent to the MLP layer for detection, and the fourth prediction score is determined according to the above formula.
[0160] The optimization of the visual analysis model is based on the classification loss function. The calculation method of the classification loss function is the same as that of the text analysis model, which will not be repeated here.
[0161] Furthermore, in one embodiment, the first prediction score, the second prediction score, the third prediction score and the fourth prediction score are combined to determine the detection result of the news to be tested, including:
[0162] Step 610: Calculate the differences between the first prediction score, the second prediction score, the third prediction score, the fourth prediction score and the score threshold respectively, and take the absolute values to obtain the calculation results;
[0163] Step 620: determine the maximum value among all calculation results, and use the prediction score corresponding to the maximum value as the target prediction score;
[0164] Step 630: compare the target prediction score with the score threshold to obtain a comparison result;
[0165] In the prior art, when a single-function deep learning model detects various fake news samples, when the positive and negative samples are balanced, the model prediction probability greater than 0.5 is usually considered to be fake news, and less than 0.5 is considered to be real news. This application develops a dynamic controller that selects a suitable expert model according to predefined rules to determine the detection results of the news to be tested.
[0166] The predefined rules are steps 610 to 640, and the score threshold is set to 0.5.
[0167] Exemplarily, the first prediction score is 0.8, the second prediction score is 0.7, the third prediction score is 0.3, and the fourth prediction score is 0.1. First, the difference between the first prediction score, the second prediction score, the third prediction score, and the fourth prediction score and the score threshold of 0.5 is calculated respectively, and the absolute value is taken to obtain the calculation results of 0.3, 0.2, 0.2, and 0.4. The maximum value of all the calculation results is determined to be 0.4. The farther from 0.5, the higher the confidence level of the corresponding sub-expert model prediction, and the higher the accuracy. The prediction score 0.1 corresponding to the maximum value 0.5 is used as the target prediction score; that is, the target prediction score is 0.1 at this time. Further, the target prediction score 0.1 is compared with the score threshold 0.5 to obtain a comparison result.
[0168] Step 640: Determine the detection result of the news to be tested based on the comparison result.
[0169] It can be understood that the comparison results include two situations, namely, the comparison result is that the target prediction score is greater than the score threshold or the comparison result is that the target prediction score is less than or equal to the score threshold. Different comparison results correspond to different detection results. If the comparison result is that the target prediction score is greater than the score threshold, the detection result of the news to be tested is determined to be false news; if the comparison result is that the target prediction score is less than or equal to the score threshold, the detection result of the news to be tested is determined to be true news.
[0170] Exemplarily, the target prediction score 0.1 is compared with the score threshold 0.5, and the comparison result is 0.1 is less than 0.5. At this time, it is determined that the detection result of the news to be tested is true news.
[0171] In this embodiment, the calculation results are obtained by respectively calculating the differences between the first prediction score, the second prediction score, the third prediction score, the fourth prediction score and the score threshold and taking the absolute values; then the maximum value among all the calculation results is determined, and the prediction score corresponding to the maximum value is used as the target prediction score; the target prediction score is further compared with the score threshold to obtain a comparison result; then based on the comparison result, the detection result of the news to be tested is determined, and the dynamic controller can adaptively select the prediction result of the optimal model in the multivariate expert model according to the characteristics of each sample, thereby improving the accuracy of multimodal false news detection.
[0172] Among them, the dynamic controller realizes the prediction result of the optimal model in the multivariate expert model adaptively according to the characteristics of each sample according to the top-k rule. For example, Top-1 means that step 620 only determines the maximum value of all calculation results, and takes the prediction score corresponding to the maximum value as the target prediction score; Top-2 means that step 620 arranges all calculation results from large to small, takes the first two larger calculation results, and averages the prediction scores corresponding to the first two larger calculation results, and takes the average score as the target prediction score; Top-3 means that step 620 arranges all calculation results from large to small, takes the first three larger calculation results, and averages the prediction scores corresponding to the first three larger calculation results, and takes the average score as the target prediction score.
[0173] For example, taking Top-1 as an example, for each sample, the selected sub-expert model is different. Some choose the text analysis model, and some choose the knowledge verification model, so it is a dynamic selection process.
[0174] In order to illustrate the accuracy of the multimodal fake news detection method in the present application, a news dataset is first obtained from a news website, and then verification is performed on the news dataset. Taking the first news dataset obtained from the first news website and the second news dataset obtained from the second news website as examples, Tables 1 and 2 are the verification results.
[0175] Table 1 Comparison of indicators of several methods on different news datasets
[0176]
[0177] Among them, EANN corresponds to the detection method used in the paper "Event Adversarial Neural Networks for Multi-ModalFake News Detection", MCAN corresponds to the detection method used in "Multimodal fusion with co-attention networks for fake news detection", CAFE corresponds to the detection method used in the paper "Cross-modal ambiguity learning for multimodal fake news detection", CFFN corresponds to the detection method used in the paper "Cross-modal Consistency Learning with Fine-grained FusionNetwork for Multimodal Fake News Detection", BMR corresponds to the detection method used in the paper "Bootstrapping multi-view representations for fake news detection", spotfake corresponds to the detection method used in the paper "Spotfake: A multi-modal framework for fake news detection", and DistilBert corresponds to the detection method used in the paper "DistilBERT, a distilled versionof BERT: smaller, faster, cheaper and lighter".
[0178] As can be seen from Table 1, for the first news dataset, the present invention outperforms the current most advanced method BMR by 1.7% in average score (90.8% → 92.5%) and improves accuracy by 0.6% (91.8% → 92.4%). For fake news, the present invention leads by 0.9% in average score (91.4% → 92.3%) and leads by 2.2% in real news (90.4% → 92.6%). For the second news dataset, the present invention leads by 1.2% in average score (88.7% → 89.9%) and exceeds 0.7% in accuracy (89.5% → 90.3%). For fake news, the present invention leads by 1.8% in average score (69.1% → 70.9%).
[0179] Table 2 Performance metrics of different controller methods
[0180]
[0181] Among them, Table 2 shows that the Top-1 method performs best with an accuracy of 92.44%. When the number of models in the selected multi-expert model increases, the accuracy steadily decreases from 92.44% to 92.22%, indicating that when multiple sub-expert models are selected to detect the same sample at the same time, the discrimination ability of the proposed framework will be slightly weakened. In addition, we calculated that under ideal conditions (i.e., at least one sub-expert model can make a correct prediction), the accuracy can reach 96.03%.
[0182] In a second aspect, an embodiment of the present application also provides a multimodal fake news detection device.
[0183] In one embodiment, referring to Figure 2 , Figure 2 This is a functional module diagram of an embodiment of a multi-modal fake news detection device of the present application. Figure 2 As shown, the multimodal false news detection device 200 includes:
[0184] The acquisition module 210 is used to acquire target data of the news to be tested, wherein the target data includes images, title texts, OCR texts and associated texts, wherein the OCR texts are texts obtained by recognizing the news to be tested through optical character recognition technology;
[0185] A first input-output module 220, for inputting the title text and the OCR text into the text analysis model and outputting a first prediction score;
[0186] A second input-output module 230, for inputting the title text and the image into the image-text comparison model and outputting a second prediction score;
[0187] A third input-output module 240, for inputting the title text and the associated text into the knowledge verification model and outputting a third prediction score;
[0188] A fourth input-output module 250, for inputting the title text and the image into the visual analysis model and outputting a fourth prediction score;
[0189] The determination module 260 is used to determine the detection result of the news to be tested by comprehensively considering the first prediction score, the second prediction score, the third prediction score and the fourth prediction score.
[0190] Furthermore, in one embodiment, the first input-output module is specifically used for:
[0191] Input the title text and OCR text into BERT to generate the first word embedding vector and the second word embedding vector. BERT is a deep bidirectional encoder.
[0192] The first word embedding vector is processed using a masked self-attention network to obtain a first sentence-level embedding vector;
[0193] The second word embedding vector is processed using a masked self-attention network to obtain a second sentence-level embedding vector;
[0194] Based on the first word embedding vector and the second word embedding vector, the first output feature of Text-CNN is determined. The corresponding formula is:
[0195] ,
[0196] Among them, w t is the first word embedding vector, w o is the second word embedding vector, is the learnable parameter of the i-th Text-CNN, r i is the first output feature, For splicing symbols, Text-CNN i is the i-th convolutional neural network;
[0197] Based on the first sentence-level embedding vector and the second sentence-level embedding vector, the first post-gate weight is determined. The corresponding formula is:
[0198] ,
[0199] Among them, e t is the first sentence level embedding vector, e o is the second sentence-level embedding vector, φ is the learnable parameter of the gating network, and w i is the weight after the first gate, softmax is the activation function, and G is the gated network;
[0200] Based on the first output feature and the first post-gate weight, the first prediction score is determined, and the corresponding formula is:
[0201] ,
[0202] in, is the first prediction score, softmax is the activation function, and MLP is the classifier.
[0203] Furthermore, in one embodiment, the second input-output module is specifically used for:
[0204] Input the title text into BERT to generate the first classification vector;
[0205] Input the image into Vision Transformer to generate the second classification vector. Vision Transformer is a visual encoder.
[0206] Perform dimensionality reduction on the first classification vector and the second classification vector respectively;
[0207] Connect the first classification vector after dimensionality reduction and the second classification vector after dimensionality reduction to obtain a multimodal feature;
[0208] The multimodal features are processed using an activation function to obtain a second prediction score.
[0209] Furthermore, in one embodiment, the third input-output module is specifically used for:
[0210] The title text and associated text are input into the knowledge verification model, and the third prediction score is output, including:
[0211] Input the title text and related text into BERT to generate the third word embedding vector and the fourth word embedding vector;
[0212] The third word embedding vector is processed using a masked self-attention network to obtain a third sentence-level embedding vector;
[0213] The fourth word embedding vector is processed using the masked self-attention network to obtain the fourth sentence-level embedding vector;
[0214] Based on the third word embedding vector and the fourth word embedding vector, determine the second output feature of Text-CNN;
[0215] Determine a second post-gate weight based on the third sentence-level embedding vector and the fourth sentence-level embedding vector;
[0216] Based on the second output feature and the second post-gated weight, a third prediction score is determined.
[0217] Furthermore, in one embodiment, the fourth input-output module is specifically used for:
[0218] Extract the query vector of the last hidden state of the image;
[0219] Extract the key vector and value vector of the last hidden state of the title text;
[0220] Based on the query vector, key vector and value vector, determine the image embedding vector that interacts with the title text. The corresponding formula is:
[0221] w f = ,
[0222] Among them, w f is the image embedding vector, Q is the query vector, K is the key vector, V is the value vector, D is the feature dimension, T is the matrix transpose, Attention is the attention function, and softmax is the activation function;
[0223] The image embedding vector is optimized by using a hybrid expert network to obtain an optimized image embedding vector;
[0224] Based on the optimized image embedding vector, the fourth prediction score is determined, and the corresponding formula is:
[0225] ,
[0226] Among them, e f is the optimized image embedding vector, is the fourth prediction score, and MLP is the classifier.
[0227] Further, in one embodiment, the determination module is specifically used for:
[0228] Calculate the differences between the first prediction score, the second prediction score, the third prediction score, the fourth prediction score and the score threshold respectively, and take the absolute values to obtain the calculation results;
[0229] Determine the maximum value among all calculation results, and use the prediction score corresponding to the maximum value as the target prediction score;
[0230] Compare the target prediction score with the score threshold to obtain a comparison result;
[0231] Based on the comparison result, the detection result of the news to be tested is determined.
[0232] Further, in one embodiment, the determination module is specifically used for:
[0233] If the comparison result is that the target prediction score is greater than the score threshold, it is determined that the detection result of the news to be tested is fake news;
[0234] If the comparison result is that the target prediction score is less than or equal to the score threshold, it is determined that the detection result of the news to be tested is true news.
[0235] Among them, the functional implementation of each module in the above-mentioned multimodal false news detection device corresponds to the various steps in the above-mentioned multimodal false news detection method embodiment, and its functions and implementation processes are no longer repeated here one by one.
[0236] In a third aspect, an embodiment of the present application provides a multimodal fake news detection device, which may be a device with data processing capabilities, such as a personal computer (PC), a laptop computer, or a server.
[0237] Reference Figure 3 , Figure 3The hardware structure diagram of the multimodal false news detection device involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the multimodal false news detection device may include a processor, a memory, a communication interface, and a communication bus.
[0238] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.
[0239] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the multimodal fake news detection device, as well as interfaces for interconnecting the multimodal fake news detection device with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.
[0240] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0241] The processor may be a general-purpose processor, which may call the multimodal false news detection program stored in the memory and execute the multimodal false news detection method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the multimodal false news detection program is called may refer to the various embodiments of the multimodal false news detection method of the present application, which will not be described in detail here.
[0242] Those skilled in the art will understand that Figure 3 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0243] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.
[0244] The computer-readable storage medium of the present application stores a multimodal false news detection program, wherein when the multimodal false news detection program is executed by a processor, the steps of the multimodal false news detection method as described above are implemented.
[0245] Among them, the method implemented when the multimodal false news detection program is executed can refer to the various embodiments of the multimodal false news detection method of the present application, and will not be repeated here.
[0246] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.
[0247] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.
[0248] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.
[0249] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.
[0250] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.
[0251] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.
[0252] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A multimodal false news detection method, characterized in that: The multimodal false news detection method comprises: Obtain target data of the news to be tested, the target data including images, title text, OCR text and associated text, wherein the OCR text is text obtained by recognizing the news to be tested through optical character recognition technology; Input the title text and OCR text into the text analysis model and output the first prediction score; Input the title text and the image into the image-text comparison model and output a second prediction score; Input the title text and the associated text into the knowledge verification model and output a third prediction score; Input the title text and the image into the visual analysis model and output a fourth prediction score; The first prediction score, the second prediction score, the third prediction score and the fourth prediction score are combined to determine the detection result of the news to be tested.
2. The multimodal false news detection method according to claim 1, characterized in that: Input the title text and OCR text into the text analysis model and output the first prediction score, including: Input the title text and OCR text into BERT to generate the first word embedding vector and the second word embedding vector. BERT is a deep bidirectional encoder. The first word embedding vector is processed using a masked self-attention network to obtain a first sentence-level embedding vector; The second word embedding vector is processed using a masked self-attention network to obtain a second sentence-level embedding vector; Based on the first word embedding vector and the second word embedding vector, the first output feature of Text-CNN is determined. The corresponding formula is: , Among them, w t is the first word embedding vector, w o is the second word embedding vector, is the learnable parameter of the i-th Text-CNN, r i is the first output feature, For splicing symbols, Text-CNN i is the i-th convolutional neural network; Based on the first sentence-level embedding vector and the second sentence-level embedding vector, the first post-gate weight is determined. The corresponding formula is: , Among them, e t is the first sentence level embedding vector, e o is the second sentence-level embedding vector, φ is the learnable parameter of the gating network, and w i is the weight after the first gate, softmax is the activation function, and G is the gated network; Based on the first output feature and the first post-gate weight, the first prediction score is determined, and the corresponding formula is: , in, is the first prediction score and MLP is the classifier.
3. The multimodal false news detection method according to claim 1, characterized in that: Input the title text and image into the image-text comparison model and output the second prediction score, including: Input the title text into BERT to generate the first classification vector; Input the image into Vision Transformer to generate the second classification vector. Vision Transformer is a visual encoder. Perform dimensionality reduction on the first classification vector and the second classification vector respectively; Connect the first classification vector after dimensionality reduction and the second classification vector after dimensionality reduction to obtain a multimodal feature; The multimodal features are processed using an activation function to obtain a second prediction score.
4. The multimodal false news detection method according to claim 1, characterized in that: The title text and associated text are input into the knowledge verification model, and the third prediction score is output, including: Input the title text and related text into BERT to generate the third word embedding vector and the fourth word embedding vector; The third word embedding vector is processed using a masked self-attention network to obtain a third sentence-level embedding vector; The fourth word embedding vector is processed using the masked self-attention network to obtain the fourth sentence-level embedding vector; Based on the third word embedding vector and the fourth word embedding vector, determine the second output feature of Text-CNN; Determine a second post-gate weight based on the third sentence-level embedding vector and the fourth sentence-level embedding vector; Based on the second output feature and the second post-gated weight, a third prediction score is determined.
5. The multimodal false news detection method according to claim 1, characterized in that: The title text and image are fed into the visual analysis model, which outputs the fourth prediction score, including: Extract the query vector of the last hidden state of the image; Extract the key vector and value vector of the last hidden state of the title text; Based on the query vector, key vector and value vector, determine the image embedding vector that interacts with the title text. The corresponding formula is: w f = , Among them, w f is the image embedding vector, Q is the query vector, K is the key vector, V is the value vector, D is the feature dimension, T is the matrix transpose, Attention is the attention function, and softmax is the activation function; The image embedding vector is optimized by using a hybrid expert network to obtain an optimized image embedding vector; Based on the optimized image embedding vector, the fourth prediction score is determined, and the corresponding formula is: , Among them, e f is the optimized image embedding vector, is the fourth prediction score, and MLP is the classifier.
6. The multimodal false news detection method according to claim 1, characterized in that: The first prediction score, the second prediction score, the third prediction score and the fourth prediction score are combined to determine the detection result of the news to be tested, including: Calculate the differences between the first prediction score, the second prediction score, the third prediction score, the fourth prediction score and the score threshold respectively, and take the absolute values to obtain the calculation results; Determine the maximum value among all calculation results, and use the prediction score corresponding to the maximum value as the target prediction score; Compare the target prediction score with the score threshold to obtain a comparison result; Based on the comparison result, the detection result of the news to be tested is determined.
7. The multimodal false news detection method according to claim 1, characterized in that: Based on the comparison results, the detection results of the news to be tested are determined, including: If the comparison result is that the target prediction score is greater than the score threshold, it is determined that the detection result of the news to be tested is fake news; If the comparison result is that the target prediction score is less than or equal to the score threshold, it is determined that the detection result of the news to be tested is true news.
8. A multimodal false news detection device, characterized in that: The multimodal false news detection device comprises: An acquisition module is used to acquire target data of the news to be tested, wherein the target data includes images, title texts, OCR texts and associated texts, wherein the OCR texts are texts obtained by recognizing the news to be tested through optical character recognition technology; A first input-output module, used for inputting the title text and the OCR text into the text analysis model, and outputting a first prediction score; A second input-output module, used for inputting the title text and the image into the image-text comparison model, and outputting a second prediction score; A third input-output module, used for inputting the title text and the associated text into the knowledge verification model, and outputting a third prediction score; a fourth input-output module, configured to input the title text and the image into the visual analysis model and output a fourth prediction score; The determination module is used to comprehensively determine the detection result of the news to be tested by combining the first prediction score, the second prediction score, the third prediction score and the fourth prediction score.
9. A multimodal fake news detection device, characterized in that: The multimodal false news detection device includes a processor, a memory, and a multimodal false news detection program stored on the memory and executable by the processor, wherein when the multimodal false news detection program is executed by the processor, the steps of the multimodal false news detection method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal false news detection program, wherein when the multimodal false news detection program is executed by a processor, the steps of the multimodal false news detection method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Distributed false news detection system based on federal learning
CN116775865A
Equipment operation and maintenance knowledge base construction method and system based on machine learning
CN117807239A
Systems and Methods for Explainable Fake News Detection
US20220036011A1
Systems and methods for annotating line charts in the wild
US20230274081A1
Method for factual event detection from online news based on deep learning
US20240070402A1
Cited By
Event knowledge graph improvement method and device based on data driving and medium
CN119849615A
A data-driven method, device, and medium for improving event knowledge graphs
CN119849615B
