A multimodal false news detection method and device for solving feature neutralization effect

Abstract

The present invention provides a multimodal false news detection method and device for solving the feature neutralization effect, the method comprising: for a multimodal false news detection task, a training data set is given; the training data set includes multiple groups of image modal and text modal content; a deep learning model is used to extract features of the image modal and text modal content, and mapped to the same feature space, to obtain deep features of the image modal and text modal in each group, to form feature pairs; to obtain single-modal authenticity prediction results corresponding to the two modalities; for each feature pair, the feature pair is combined with the single-modal authenticity prediction results to fuse the feature pair into a multimodal feature; and the authenticity of the multimodal feature is predicted. The present invention can solve the neutralization effect problem caused by multimodal feature fusion and improve the accuracy of false news detection.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a multimodal false news detection method and device for solving feature neutralization effect. Background Art

[0002] Most existing fake news detection methods use news text content for detection, using advanced artificial intelligence technology to learn content patterns and dissemination patterns from training data. However, most news on social platforms does not only contain text content, but also multimodal content such as text, pictures, and videos. The field of artificial intelligence has also begun to gradually focus on multimodal fake news detection tasks, encoding and mapping content in different modes to the same feature space, and then using a specific fusion module to fuse the features of multiple modes for final classification.

[0003] The latest multimodal fake news detection model is CAFE [Chen Y, Li D, Zhang P, et al. Cross-modal ambiguity learning for multimodal fake news detection [C] / / Proceedings of the ACM Web Conference 2022.2022: 2897-2905], which uses a pre-trained ResNet network structure and a BERT model to extract features from images and texts, respectively, and uses a multi-layer neural network to map the features of the two to the same space and concatenate them for final authenticity classification. The novelty of this model lies in modeling the inconsistency between modalities to assist in the above-mentioned feature concatenation operation. Specifically, it uses a variational encoder to map the features of each modality to a Gaussian distribution, and then uses the Kullback-Leibler divergence to calculate the distance between the distributions to model their inconsistency.

[0004] Fusing the features of multimodal news is a key step in multimodal fake news detection methods, while existing detection methods only consider the splicing of single-modal features, which leads to the problem of neutralization effect. Specifically, the neutralization effect is caused by the contradiction between the authenticity of different modalities. That is, the authenticity of each modality (such as images and texts) is always true and false, and in reality, if one modality in a multimodal news is false, then the authenticity of this news must be false. It is because of this phenomenon that when a piece of news appears, its two modalities are one true and one false, a simple feature splicing method like other methods will cause the neutralization of the authenticity of the two modalities, resulting in incorrect predictions. Figure 1 Shown is a formal description of the neutralization effect. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a multimodal false news detection method and device for solving the feature neutralization effect, so as to solve the neutralization effect problem caused by multimodal feature fusion existing in the prior art.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] On the one hand, a multimodal fake news detection method for solving feature neutralization effect is provided, comprising the following steps:

[0008] S1, for a multimodal fake news detection task, a training dataset is given; the training dataset includes multiple sets of image modality and text modality content;

[0009] S2, using a deep learning model to extract features from the contents of the image modality and text modality, and mapping them to the same feature space, obtaining deep features of the image modality and text modality in each group to form feature pairs;

[0010] S3, obtaining the single-modality authenticity prediction results corresponding to the two modalities;

[0011] S4, for each feature pair, combine the single-modal authenticity prediction results and fuse the feature pair into a multimodal feature;

[0012] S5, predict the authenticity of multimodal features.

[0013] Preferably, in step S1, the training data set is in is the content of image mode and text mode, y i ∈{0, 1} is the authenticity label corresponding to the news, 0 means false and 1 means true.

[0014] Preferably, in step S2, for each picture The deep features are For each text The deep features are The characteristic pairs are formed as

[0015] in and They represent the image encoder and text encoder based on deep learning models and their alignment operations respectively.

[0016] Preferably, in step S3, the problem of unimodal authenticity classification is transformed into a PU learning problem; when a piece of news is true, its two unimodal authenticity must be true. Based on this fact, the initial training data set is Transformed into a positive unimodal dataset and unlabeled unimodal datasets Where M represents the number of samples with true labels in the training data set;

[0017] In these two data sets Based on the data set, the single-modal authenticity classification problem is transformed into a PU learning problem. The variational PU learning method is used to Randomly sample a batch from in By positive subset and the unlabeled subset The deep learning objective function of this part is:

[0018]

[0019] in represents a unimodal authenticity classification operation based on a forward propagation network, used to predict z I and z T The authenticity of a single mode, i.e. true or false; log(·) represents the logarithm with the natural constant e as the base;

[0020] Because z I and z T In order to ensure that the same framework is used for calculations based on different modal data, a modal alignment objective function is proposed based on contrastive learning technology:

[0021]

[0022] Where τ is the temperature coefficient, cos(·,·) is the cosine similarity, and exp(·) represents the exponential with the natural constant e as the base; Combining the above two formulas, the training objective function of the unimodal authenticity classifier is as follows:

[0023]

[0024] Where γ is the trade-off coefficient. So far, the single-modal authenticity prediction results corresponding to the two modes are obtained. and

[0025] Preferably, in step S4, the single-modal authenticity prediction result and With unimodal features Integrate to obtain multimodal features Represents a multimodal feature fusion operation.

[0026] Preferably, in step S5, a multimodal authenticity classifier is constructed to predict the authenticity of the multimodal features. represents the multimodal authenticity classification operation; the supervised learning objective function of the multimodal authenticity classifier is:

[0027]

[0028] l CE (·,·) represents the standard cross entropy loss function; combined with the supervised learning objective function and The final objective function is:

[0029]

[0030] Where λ is a trade-off hyperparameter.

[0031] On the other hand, a multimodal false news detection device for solving the feature neutralization effect is provided, comprising:

[0032] A training data set setting module, for a multimodal fake news detection task, is given a training data set; the training data set includes multiple sets of image modality and text modality content;

[0033] The single-modal feature extraction module uses a deep learning model to extract features from the contents of the image modality and text modality, and maps them to the same feature space to obtain the deep features of the image modality and text modality in each group to form feature pairs;

[0034] Single-modal authenticity classifier, obtains the single-modal authenticity prediction results corresponding to the two modalities;

[0035] The multimodal feature fusion module combines the single-modal authenticity prediction results for each feature pair and fuses the feature pair into a multimodal feature;

[0036] Multimodal authenticity classifier,predicts the authenticity of multimodal features.

[0037] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned multimodal fake news detection method.

[0038] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned multimodal fake news detection method.

[0039] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0040] In view of the neutralization effect problem existing in the existing multimodal fake news detection methods, the present invention proposes a new multimodal fake news detection method. By designing a unimodal authenticity classifier and converting it into a PU learning problem for training, auxiliary unimodal authenticity prediction is used to complete multimodal feature fusion, thereby solving the neutralization effect problem and improving the accuracy of fake news identification and detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 It is a schematic diagram describing the form of the neutralization effect;

[0043] Figure 2 is a flow chart of a multimodal false news detection method provided by an embodiment of the present invention;

[0044] Figure 3 It is a schematic diagram of the principle of the multimodal fake news detection method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The embodiment of the present invention provides a multimodal false news detection method for solving the feature neutralization effect. Figure 2 is a flow chart of the multimodal fake news detection method, Figure 3 Schematic diagram of the principle of the multimodal fake news detection method. Figure 2 and Figure 3 , the method comprises the following steps:

[0047] S1. For a multimodal fake news detection task, a training dataset is given; the training dataset includes multiple sets of contents in image modalities and text modalities.

[0048] The training data set is in is the content of image mode and text mode, yi ∈{0, 1} is the authenticity label corresponding to the news, 0 means false and 1 means true. Given such a training dataset, the goal of the fake news detector is to give accurate authenticity predictions for new news.

[0049] S2, uses a deep learning model to extract features from the contents of the image modality and text modality, and maps them to the same feature space to obtain the deep features of the image modality and text modality in each group to form feature pairs.

[0050] Among them, for each picture The deep features are For each text The deep features are The characteristic pairs are formed as and Respectively Figure 3 The image encoder and text encoder based on the deep learning model and their alignment operation are shown.

[0051] S3, obtaining the single-modal authenticity prediction results corresponding to the two modalities.

[0052] For each feature pair Usually we need to merge them into a multimodal feature However, if the features are directly spliced ​​like other works, it will lead to the neutralization effect problem mentioned above. Therefore, the present invention uses the single-modal authenticity prediction results corresponding to the two modalities and Participate in the integration process, where and represents the probability that the truth of the two modalities is predicted to be true.

[0053] In order to make these unimodal predictions more accurate, the present invention transforms the unimodal authenticity classification problem into a positive and unlabeled (PU) learning problem. Specifically, when a piece of news is true, its two unimodal authenticity must be true. Based on this fact, the initial training data set is Transformed into a positive unimodal dataset and unlabeled unimodal datasets Where M represents the number of samples with true labels in the training data set;

[0054] In these two data sets Based on the data set, the single-modal authenticity classification problem is transformed into a PU learning problem. The variational PU learning method is used to Randomly sample a batch from in By positive subset and the unlabeled subset The deep learning objective function of this part is:

[0055]

[0056] in express Figure 3 The unimodal authenticity classification operation implemented by the unimodal authenticity classifier based on the forward propagation network shown is used to predict z I and z T The authenticity of a single mode, that is, true or false; log(·) represents the logarithm with the natural constant e as the base.

[0057] Because z I and z T To ensure that the data from different modalities can be calculated using the same framework, this paper proposes an objective function for modality alignment based on contrastive learning technology:

[0058]

[0059] Where τ is the temperature coefficient, cos(·,·) is the cosine similarity, and exp(·) represents the exponential with the natural constant e as the base; Combining the above two formulas, the training objective function of the unimodal authenticity classifier is as follows:

[0060]

[0061] Where γ is the trade-off coefficient. So far, the single-modal authenticity prediction results corresponding to the two modes are obtained. and

[0062] S4, for each feature pair, combine the single-modal authenticity prediction results and fuse the feature pair into a multimodal feature.

[0063] After the above steps, the single-modal authenticity prediction results of each modality have been obtained. and The single-modal authenticity prediction results and With unimodal features Integrate to obtain multimodal features express Figure 3 The multimodal feature fusion operation implemented by the multimodal feature fusion module shown.

[0064] S5, predict the authenticity of multimodal features.

[0065] Finally, a multimodal authenticity classifier is constructed to predict the authenticity of multimodal features. express Figure 3 The multimodal authenticity classification operation implemented by the multimodal authenticity classifier shown.

[0066] The supervised learning objective function of the multimodal authenticity classifier is:

[0067]

[0068] l CE (·,·) represents the standard cross entropy loss function. Combining the supervised learning objective function and Finally, the objective function proposed by the present invention is:

[0069]

[0070] Where λ is a trade-off hyperparameter.

[0071] In view of the neutralization effect problem existing in the existing multimodal fake news detection methods, the present invention proposes a new multimodal fake news detection method. By designing a unimodal authenticity classifier and converting it into a PU learning problem for training, auxiliary unimodal authenticity prediction is used to complete multimodal feature fusion, thereby solving the neutralization effect problem and improving the accuracy of fake news identification and detection.

[0072] The false news recognition accuracy of the proposed method on the existing data set can reach 93.4%, which exceeds the latest detection model CAFE with an accuracy of 84.0%.

[0073] Accordingly, an embodiment of the present invention further provides a multimodal false news detection device, the device comprising:

[0074] A training data set setting module, for a multimodal fake news detection task, is given a training data set; the training data set includes multiple sets of image modality and text modality content;

[0075] The single-modal feature extraction module uses a deep learning model to extract features from the contents of the image modality and text modality, and maps them to the same feature space to obtain the deep features of the image modality and text modality in each group to form feature pairs;

[0076] Single-modal authenticity classifier, obtains the single-modal authenticity prediction results corresponding to the two modalities;

[0077] The multimodal feature fusion module combines the single-modal authenticity prediction results for each feature pair and fuses the feature pair into a multimodal feature;

[0078] Multimodal authenticity classifier,predicts the authenticity of multimodal features.

[0079] The device of this embodiment can be used to perform Figure 2 and Figure 3 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0080] An embodiment of the present invention also provides an electronic device, which may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) and one or more memories, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the steps of the above-mentioned multimodal fake news detection method.

[0081] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the above-mentioned multimodal fake news detection method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0082] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0083] References in the specification to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments", etc. indicate that the described embodiments may include a particular feature, structure, or characteristic, but not every embodiment may include the particular feature, structure, or characteristic. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the relevant art to implement such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0084] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0085] A person skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal fake news detection method for solving feature neutralization effect, characterized in that: The following steps are involved: S1, for a multimodal fake news detection task, a training dataset is given; the training dataset includes multiple sets of image modality and text modality content; In step S1, the training data set is in is the content of image mode and text mode, y i ∈{0,1} is the authenticity label corresponding to the news, 0 means false and 1 means true; S2, using a deep learning model to extract features from the contents of the image modality and text modality, and mapping them to the same feature space, obtaining deep features of the image modality and text modality in each group to form feature pairs; In step S2, for each picture The deep features are For each text The deep features are The characteristic pairs are formed as in represents the image encoder based on the deep learning model and its alignment operation, Represents the text encoder based on the deep learning model and its alignment operation; S3, obtaining the single-modality authenticity prediction results corresponding to the two modalities; In step S3, the problem of unimodal authenticity classification is transformed into a PU learning problem; when a piece of news is true, its two unimodal authenticity must be true. Based on this fact, the initial training data set is Transformed into a positive unimodal dataset and unlabeled unimodal datasets Where M represents the number of samples with true labels in the training data set; In these two datasets Based on the data set, the single-modal authenticity classification problem is transformed into a PU learning problem. The variational PU learning method is used to Randomly sample a batch from in By positive subset and the unlabeled subset The deep learning objective function of this part is: in represents a unimodal authenticity classification operation based on a forward propagation network, used to predict z I and z T The authenticity of a single mode, i.e. true or false; log(·) represents the logarithm with the natural constant e as the base; Because z I and z T In order to ensure that the same framework is used for calculations based on different modal data, a modal alignment objective function is proposed based on contrastive learning technology: Where τ is the temperature coefficient, cos(·,·) is the cosine similarity, and exp(·) represents the exponential with the natural constant e as the base; Combining the above two formulas, the training objective function of the unimodal authenticity classifier is as follows: Where γ is the trade-off coefficient. So far, the single-modal authenticity prediction results corresponding to the two modes are obtained. and S4, for each feature pair, combine the single-modal authenticity prediction results and fuse the feature pair into a multimodal feature; S5, predict the authenticity of multimodal features.

2. The multimodal false news detection method according to claim 1, characterized in that: In step S4, the single-modal authenticity prediction result and With unimodal features Integrate to obtain multimodal features Represents a multimodal feature fusion operation.

3. The multimodal false news detection method according to claim 2, characterized in that: In step S5, a multimodal authenticity classifier is constructed to predict the authenticity of the multimodal features. represents the multimodal authenticity classification operation; the supervised learning objective function of the multimodal authenticity classifier is: l CE (·,·) represents the standard cross entropy loss function; combined with the supervised learning objective function and The final objective function is: Where λ is a trade-off hyperparameter.

4. A multimodal false news detection device for solving feature neutralization effect, the device is used to implement the multimodal false news detection method according to any one of claims 1 to 3, characterized in that: include: A training data set setting module, for a multimodal fake news detection task, is given a training data set; the training data set includes multiple sets of image modality and text modality content; The single-modal feature extraction module uses a deep learning model to extract features from the contents of the image modality and text modality, and maps them to the same feature space to obtain the deep features of the image modality and text modality in each group to form feature pairs; Single-modal authenticity classifier, obtains the single-modal authenticity prediction results corresponding to the two modalities; The multimodal feature fusion module combines the single-modal authenticity prediction results for each feature pair and fuses the feature pair into a multimodal feature; Multimodal authenticity classifier,predicts the authenticity of multimodal features.

5. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the multimodal fake news detection method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the multimodal fake news detection method as described in any one of claims 1-3.

Images