False news detection method and system based on uncertain semantic fusion

Abstract

The application discloses a false news detection method and system based on uncertain semantic fusion, relates to the technical field of data mining, and comprises the following steps: acquiring multi-modal data of news in a social network; extracting high-order semantic features of a specific mode from each mode of data by a mode coding module based on uncertainty; modeling uncertainty between modes by a variational attention fusion module, and fusing multi-modal features; and obtaining a category label of target news by using a false news classifier based on the fused multi-modal features. The application can extract more robust high-order features from different modal information of news, further effectively integrate more reliable multi-modal features, and effectively improve the accuracy of multi-modal false news detection.

Description

Technical Field

[0001] This invention relates to the field of data mining technology, specifically to a method and system for detecting fake news based on uncertain semantic fusion. Background Technology

[0002] In recent years, fake news has evolved from its traditional plain text form to a multimodal format. Compared to plain text-based fake information, fake news with visual elements is more eye-catching and spreads faster, causing significant harm and negative impacts on the online ecosystem and seriously affecting social stability and people's daily lives. Therefore, how to automatically detect multimodal fake news is an urgent research topic in the field of social network analysis, with significant research and practical application implications.

[0003] Multimodal information provides many complementary clues for identifying misinformation. Two key aspects of multimodal content understanding are the extraction and fusion of multimodal features. Firstly, regarding multimodal feature extraction, most multimodal fake news detection methods rely on pre-trained models to obtain high-order semantic features. However, multimodal data, especially from social media, is often low-quality and contains a lot of noise, causing semantic uncertainty within the image / text modality. Traditional encoders struggle to represent noisy data with inherent uncertainty using point vectors. Therefore, robustly extracting representations of specific modalities is crucial for better understanding multimodal content.

[0004] Secondly, fake news publishers may disguise data across any modality, leading to variations in the distribution of disguised data across different modalities and introducing uncertainty between modalities. In multimodal feature fusion, existing attention-based fusion methods, based on data features, learn modality-independent weights to fuse multimodal features, ignoring the impact of these aforementioned differences. Therefore, how to reasonably model the uncertainty between modalities is crucial for effectively fusing multimodal features. Summary of the Invention

[0005] The purpose of this invention is to address the following two main technical problems: first, how to model the inherent data uncertainty of different modal data and extract more robust modal features; second, how to consider the differences in modal masquerading distribution and more effectively fuse multimodal features; and to provide a fake news detection method and system that simultaneously models intramodal and intermodal uncertainties to improve the performance of fake news detection tasks.

[0006] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0007] A method for detecting fake news based on uncertain semantic fusion includes the following steps:

[0008] 1) Acquire multimodal data of news from social networks, including text and / or images;

[0009] 2) For the multimodal data, an uncertainty-based modal coding module is constructed. This modal coding module extracts sentence representations from the input modal data through a pre-trained language model, maps the sentence representations to a predefined dimension, and then reconstructs the mapped sentence representations using a latent Gaussian distribution to obtain the distributed semantic representation of the modal data.

[0010] 3) Based on the distributed semantic representations of different modalities obtained in step 2), a variational attention fusion module is constructed. This variational attention fusion module obtains cross-modal attention vectors through sampling, and then calculates and fuses the feature representations of different modalities through weighted calculation.

[0011] 4) The fused multimodal feature representation obtained in step 3) is processed using a fake news classifier to obtain the category label of the target news.

[0012] Preferably, step 1) involves preprocessing the text and / or images, including: segmenting the text into words, generating a token character index, and retaining the maximum sequence length; and converting the images to a preset size.

[0013] Preferably, in step 2), the expression representing the sentence is extracted from the input modal data using a pre-trained language model as follows:

[0014] x t =BERT({[CLS],w1,...,w n})

[0015] Where, x t The expression representing a sentence, [CLS] indicates the insertion of a special character at the beginning of the sentence, w i This represents the i-th word of the input text sequence, where n is the text length, and BERT represents a pre-trained language model.

[0016] Preferably, in step 2), the sentence representation is mapped to a predefined dimension based on the activation function and the trainable parameters of the pre-trained language model.

[0017] Preferably, in step 2), the expression for reconstructing the mapped sentence representation using a latent Gaussian distribution is as follows:

[0018]

[0019] Where p represents the reconstructed sentence representation, z t The distributed representation of the reconstruction, s t The mapped sentence representation, It follows a Gaussian distribution, μ tLet σ be the mean of the Gaussian distribution. t Let be the variance of the Gaussian distribution, and I be the identity matrix.

[0020] Preferably, in and This represents a parameter-independent multilayer perceptron, where θ1 and θ2 are the corresponding model parameters.

[0021] Preferably, in step 2), after reconstructing the mapped sentence representation using a latent Gaussian distribution, reparameterization is used to enable the calculation of the model gradient. The expression for the reconstructed distributed representation is as follows:

[0022] Preferably, the cross-modal attention vector in step 3) is calculated as follows:

[0023]

[0024]

[0025] in, Let a be the attention vector corresponding to mode k. k z is the attention vector corresponding to the regularized mode k. k For the distributed semantic representation of modality k, The mean value corresponding to mode k; q represents the variance corresponding to mode k, describing the relative confidence level of mode k; k′ represents the mode that takes values ​​from the set {v, t}, and q represents the relative confidence level of mode k. θ The posterior distribution of attention weights between modalities is given, where t represents the text modality and v represents the image modality.

[0026] Preferably, the calculation formula for weighted calculation and fusion of feature representations of different modalities in step 3) is as follows:

[0027]

[0028] Among them, W h These are trainable parameters.

[0029] A fake news detection system based on uncertain semantic fusion includes:

[0030] The data acquisition and preprocessing module is used to acquire multimodal data of news from social media platforms, including text and / or images;

[0031] The modality coding module is used to extract sentence representations from multimodal data using a pre-trained language model, map the sentence representations to a predefined dimension, and then reconstruct the mapped sentence representations using a latent Gaussian distribution to obtain a distributed semantic representation of the multimodal data.

[0032] The variational attention fusion module is used to obtain cross-modal attention vectors by sampling for distributed semantic representations under different modalities, and to calculate and fuse the feature representations of different modalities by weighting.

[0033] The fake news classification module is used to process the fused multimodal feature representation using a fake news classifier to predict the category label of the target news.

[0034] The technical solution of the present invention has the following advantages:

[0035] In multimodal feature extraction, this invention models the inherent uncertainties within a specific modality through feature regularization, enabling robust extraction of richer specific modality representations from noisy multimodal content, thus facilitating a better understanding of specific modality semantic information. In multimodal feature fusion, this invention learns an attention weight associated with modality masquerading distribution to estimate the contributions of different modalities, effectively fusing multimodal features and allowing the model to integrate effective information from more reliable modalities. By simultaneously modeling intramodal and intermodal uncertainties, this invention can more effectively understand and fuse multimodal content, significantly improving the performance of multimodal fake news detection. Experiments on public datasets show that this invention can effectively improve detection accuracy by 4.8% and 10.2% for Weibo and Twitter platforms, respectively. Attached Figure Description

[0036] Figure 1 This is a flowchart of a fake news detection method based on uncertain semantic fusion, according to an embodiment of the present invention.

[0037] Figure 2 This is a structural diagram of a fake news detection system based on uncertain semantic fusion, according to an embodiment of the present invention. Detailed Implementation

[0038] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the described embodiments are only intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0039] Figure 1 This is a flowchart of a fake news detection method provided by an embodiment of the present invention, which specifically includes the following steps:

[0040] Step A: Obtain multimodal data of news from social networks. For each sample i in the multimodal data, it includes text content and image content. Preprocessing of the text and images includes: for text content, performing word segmentation (English is generally segmented into WordPieces, and Chinese into characters), generating token character indices, and retaining the maximum sequence length; for image content, converting the image to a predefined size of 3×224×224 for feeding into the pre-trained encoder.

[0041] Step B: Based on the multimodal data from Step A, construct an uncertainty-based modality coding module. Use this module to process the data for each modality in the multimodal data. Taking the text modality as an example, the input sample text sequence X... t We model the inherent uncertainties within text modalities and obtain distributed semantic representations of text modalities.

[0042] The specific processing steps include:

[0043] Step B1: Extract sentence representations using a pre-trained language model Where D t The dimensions of the latent variables output by the model are as follows:

[0044] x t =BERT({[CLS],w1,...,w n})

[0045] Where [CLS] represents a special character inserted at the beginning of a sentence, w i This represents the i-th word of the input text sequence, where n is the text length, and BERT represents a pre-trained language model.

[0046] Step B2: Map the sentence representation to the Dh dimension, calculated as follows:

[0047]

[0048] in, and Here are the trainable parameters, and σ represents the activation function.

[0049] Step B3: Reconstruct the mapped sentence representation using a latent Gaussian distribution to model in-mode uncertainty. The parameterized Gaussian distribution is defined as follows:

[0050]

[0051] Where p represents the reconstructed sentence representation. It follows a Gaussian distribution, μ t Let σ be the mean of the Gaussian distribution. tLet be the variance of the Gaussian distribution. in, and Let I represent a parameter-independent multilayer perceptron, where θ1 and θ2 are the corresponding model parameters, and I is the identity matrix.

[0052] During implementation, since the sampling process is non-differentiable, reparameterization is used to make the model gradient computable. Therefore, the reconstructed distributed representation z... t The calculation method is as follows:

[0053]

[0054] Step C: Based on the high-order semantic features of different modalities obtained in Step B, i.e., distributed semantic representations, construct a variational attention fusion module to model the uncertainty between modalities and fuse multimodal features;

[0055] The specific processing steps include:

[0056] Step C1: To model the uncertainty between modalities, the traditional attention mechanism is improved by obtaining cross-modal attention vectors through sampling. Given distributed representations of different modalities, the cross-modal attention vectors are calculated as follows:

[0057]

[0058]

[0059] in, Let a be the attention vector corresponding to mode k. k z is the attention vector corresponding to the regularized mode k. k For the distributed semantic representation of modality k, The mean value corresponding to mode k; q represents the variance corresponding to mode k, describing the relative confidence level of mode k; k′ represents the mode that takes values ​​from the set {v, t}, and q represents the relative confidence level of mode k. θ The posterior distribution of attention weights between modalities is given, where t represents the text modality and v represents the image modality.

[0060] Step C2: Obtain the fused multimodal representation by weighted calculation of the feature representations of different modalities. Specifically:

[0061]

[0062] Among them, W h These are trainable parameters.

[0063] Step D: Based on the fused multimodal features h obtained in step C, use a fake news classifier to obtain the category label of the target news.

[0064] like Figure 2 As shown, this embodiment of the invention also provides a fake news detection system, specifically including:

[0065] The data acquisition and preprocessing module is used for news data acquisition from social media platforms. It acquires multimodal news data (such as text, images, etc.) and performs data preprocessing. The preprocessing includes: for text content, performing word segmentation (English is generally segmented into WordPieces, and Chinese is generally segmented into characters), generating token character indexes, and retaining the maximum sequence length; for image content, converting the image to a predefined size of 3×224×224 in order to feed it into the pre-trained encoder.

[0066] The uncertainty-based modality coding module is used to encode different modal data after preprocessing. It models the inherent uncertainty within the modality, that is, it extracts sentence representations through a pre-trained language model, maps the sentence representations to a predefined dimension, and then reconstructs the mapped sentence representations using a latent Gaussian distribution to generate distributed semantic representations under a specific modality.

[0067] The variational attention fusion module is used to obtain cross-modal attention vectors by sampling based on the distributed semantic representations under specific modalities obtained above, calculate the feature representations of different modalities by weighting, and fuse them to generate multimodal feature representations of news.

[0068] The fake news classification module is used to predict the category label of the sample based on the above multimodal feature representation and a fake news classifier.

[0069] Experimental test:

[0070] Experiments were conducted using publicly available multimodal fake news benchmark datasets from two well-known social networking platforms, Weibo and Twitter. The results were compared with classic single-modal detection methods (BERT, VGG-19) and multimodal detection methods (SAFE, att-RNN, MVAE, EANN, BDANN) in this field. Accuracy was used as the evaluation metric. The experimental results for the two datasets are shown in the table below. On the benchmark datasets of Weibo and Twitter, this invention can improve the accuracy by 4.8% and 10.2%, respectively.

[0071] Table 1

[0072] method Weibo Twitter BERT 0.804 0.706 VGG-19 0.633 0.596 SAFE 0.763 0.766 att-RNN 0.772 0.664 MVAE 0.824 0.745 EANN 0.794 0.648 BDANN 0.814 0.821 Method of the present invention 0.862 0.923 Increase percentage +4.8％ +10.2％

[0073] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting fake news based on uncertain semantic fusion, characterized in that, Includes the following steps: 1) Acquire multimodal data of news from social networks, including text and / or images; 2) For the aforementioned multimodal data, an uncertainty-based modality coding module is constructed. This module extracts sentence representations from the input modality data using a pre-trained language model, maps the sentence representations to a predefined dimension, and then reconstructs the mapped sentence representations using a latent Gaussian distribution to obtain the distributed semantic representation of the modality data. The expression for reconstructing the mapped sentence representations using a latent Gaussian distribution is as follows: in, This represents the reconstructed sentence representation. Representation of the reconstructed distributed representation, The mapped sentence representation, It follows a Gaussian distribution. The mean of a Gaussian distribution is given. Let V be the variance of the Gaussian distribution. For unit array; 3) Based on the distributed semantic representations of different modalities obtained in step 2), a variational attention fusion module is constructed. This module obtains cross-modal attention vectors through sampling, and then calculates and fuses the feature representations of different modalities through weighted calculation. The calculation formula for the cross-modal attention vector is as follows: in, This is the attention vector corresponding to mode k. This is the attention vector corresponding to the regularized mode k. For the distributed semantic representation of modality k, The mean value corresponding to mode k; The variance corresponding to mode k describes the relative confidence level of mode k; For a modality that takes values ​​from the set {v,t}, For the posterior distribution of intermodal attention weights, Represents text modality, Represents image modality; 4) The fused multimodal feature representation obtained in step 3) is processed using a fake news classifier to obtain the category label of the target news.

2. The method as described in claim 1, characterized in that, In step 1), the text and / or images are preprocessed, including: performing word segmentation on the text, generating a token character index, and retaining the maximum sequence length; and converting the images to a preset size.

3. The method as described in claim 1, characterized in that, In step 2), the expression representing the sentence is extracted from the input modal data using a pre-trained language model, as follows: in, An expression representing a sentence. This indicates a special character inserted at the beginning of a sentence. This represents the i-th word of the input text sequence, where n is the text length. This represents a pre-trained language model.

4. The method as described in claim 1, characterized in that, In step 2), sentence representations are mapped to predefined dimensions based on activation functions and trainable parameters of the pre-trained language model.

5. The method as described in claim 1, characterized in that, , ,in and This represents a parameter-independent multilayer perceptron. and These are the corresponding model parameters.

6. The method as described in claim 1, characterized in that, In step 2), after reconstructing the mapped sentence representation using a latent Gaussian distribution, reparameterization is used to enable the computation of the model gradient. The expression for the reconstructed distributed representation is as follows: .

7. The method as described in claim 1, characterized in that, The calculation formula for weighted calculation and fusion of feature representations of different modalities in step 3) is as follows: in, These are trainable parameters.

8. A fake news detection system based on uncertain semantic fusion, characterized in that, include: The data acquisition and preprocessing module is used to acquire multimodal data of news from social media platforms, including text and / or images; The modality coding module is used to extract sentence representations from multimodal data using a pre-trained language model, map these representations to a predefined dimension, and then reconstruct the mapped sentence representations using a latent Gaussian distribution to obtain a distributed semantic representation of the multimodal data. The expression for reconstructing the mapped sentence representations using a latent Gaussian distribution is as follows: in, This represents the reconstructed sentence representation. Representation of the reconstructed distributed representation, The mapped sentence representation, It follows a Gaussian distribution. The mean of a Gaussian distribution is given. Let V be the variance of the Gaussian distribution. For unit array; The variational attention fusion module is used to obtain cross-modal attention vectors through sampling for distributed semantic representations under different modalities. It then calculates and fuses the feature representations of different modalities through weighted calculation. The calculation formula for the cross-modal attention vector is as follows: in, This is the attention vector corresponding to mode k. This is the attention vector corresponding to the regularized mode k. For the distributed semantic representation of modality k, The mean value corresponding to mode k; The variance corresponding to mode k describes the relative confidence level of mode k; For a modality that takes values ​​from the set {v,t}, For the posterior distribution of intermodal attention weights, Represents text modality, Represents image modality; The fake news classification module is used to process the fused multimodal feature representation using a fake news classifier to predict the category label of the target news.

Images