Information entropy guided debiased multi-modal fake news detection method

Abstract

The application belongs to the field of news detection, and particularly relates to a multi-modal false news detection method based on information entropy guidance, which comprises the following steps: obtaining a training news data set, wherein the training news data set comprises a news image set, a news text set and a true label set; constructing a prediction model, training the prediction model using the training news data set, obtaining a prediction loss, a total classification loss and a decoupling total loss, summing the prediction loss, the total classification loss and the decoupling total loss to obtain a total loss, obtaining a prediction model parameter at a minimum total loss, obtaining an optimal parameter, updating the prediction model according to the optimal parameter, and obtaining an optimal prediction model; obtaining an actual news data set; inputting the actual news data set into the optimal prediction model to obtain a prediction result, wherein the prediction result is that the news is true or false. The application has the effect of improving the accuracy of news true or false judgment.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of news detection, and particularly relates to a multi-modal fake news detection method based on information entropy guidance. BACKGROUND

[0002] In the information technology era, the communication paradigm of news has changed from the traditional media one-way broadcast to the multi-element, high-speed and interactive diffusion based on the Internet and social platforms. As a typical harmful information, the large-scale generation and spread of fake news pose a serious threat to public awareness, social trust and even political stability, so it is very important to judge the truth of news.

[0003] In related technologies, fake news detection can be divided into three directions: multi-modal feature modeling, cross-modal interaction learning and pre-training model enhancement. The first direction focuses on learning comprehensive multi-modal representation by integrating text, image and metadata (usually metadata). For example, SAFE explicitly models the semantic correlation between text and image to improve classification performance. The second type of method emphasizes cross-modal interaction and semantic alignment. The third type of method uses large-scale pre-training models, such as CLIP, BERT and VGG, to inject rich semantic prior information.

[0004] For the above related technologies, existing methods usually focus on feature and interaction modeling, ignoring the double imbalance problems in multi-modal learning. First, the modality imbalance problem exposes the inherent optimization bias in multi-modal joint training: the model tends to quickly converge to the dominant and easy-to-learn modality, while ignoring effective representation learning for non-dominant modalities. Second, the class imbalance problem, most learning algorithms are based on the assumption that the prior distribution of classes in the training data is balanced. However, in practice, this assumption is rarely true. The class imbalance problem is particularly prominent in multi-modal fake news detection. SUMMARY

[0005] The technical problem to be solved by the application is to provide a multi-modal fake news detection method based on information entropy guidance to improve the accuracy of fake news detection.

[0006] A multi-modal fake news detection method based on information entropy guidance, comprising:

[0007] Obtain a training news data set, the training news data set comprising a news image set, a news text set and a true annotation set;

[0008] The prediction model is constructed, the prediction model is trained using the training news data set, prediction loss, total classification loss and decoupling total loss are obtained, the prediction loss, the total classification loss and the decoupling total loss are summed to obtain the total loss, the prediction model parameter when the total loss is minimum is obtained, the optimal parameter is obtained, the prediction model is updated according to the optimal parameter, and the optimal prediction model is obtained;

[0009] An actual news data set is obtained.

[0010] The actual news data set is input into the optimal prediction model to obtain a prediction result, wherein the prediction result is that the news is true or false.

[0011] The process of training the prediction model using the training news data set to obtain the prediction loss, the total classification loss and the decoupling total loss is as follows:

[0012] Features of the news image set are extracted to obtain image features, and features of the news text set are extracted to obtain text features.

[0013] The image features and the text features are subjected to global feature extraction, splicing, separation and mapping operations to obtain initial image features, initial text features and initial fusion features.

[0014] The decoupling total loss is obtained according to the initial fusion features, the initial image features and the initial text features.

[0015] Image embedding features and text embedding features are obtained according to the image features and the text features, the image embedding features and the text embedding features are subjected to nonlinear transformation to obtain nonlinear transformation features, the nonlinear transformation features are weighted to obtain attention scores, and the weighted features are obtained according to the attention scores and the nonlinear transformation features, wherein the weighted features include image weighted features and text weighted features.

[0016] The image weighted features, the text weighted features and the initial fusion features are input into a classifier to obtain entropy features, and entropy weights are calculated according to the entropy features, wherein the entropy weights include image entropy weights, text entropy weights and fusion entropy weights.

[0017] Image classifier features, text classifier features and fusion classifier features are obtained according to the image weighted features, the text weighted features and the initial fusion features, and image classification loss, text classification loss and fusion classification loss are obtained according to the image classifier features, the text classifier features, the fusion classifier features and a real label set.

[0018] Total classification loss is obtained according to the image classification loss, the text classification loss, the fusion classification loss, the image entropy weights, the text entropy weights and the fusion entropy weights.

[0019] According to the initial image feature, the initial text feature and the initial fusion feature, a difference vector is calculated, and according to the difference vector and the initial feature, a feedforward feature is obtained;

[0020] According to the feedforward feature, a query vector, a key vector and a value vector, attention update is performed to obtain an updated feature, the updated feature including an image updated feature, a text updated feature and a fusion updated feature, the query vector, the key vector and the value vector being obtained through the feedforward feature;

[0021] The image updated feature, the text updated feature and the fusion updated feature are spliced to obtain a final feature;

[0022] The final feature is input into a prediction classifier to obtain a prediction value, and according to the prediction value and the real label set, a prediction loss is obtained.

[0023] Optionally, the image feature and the text feature are subjected to global feature extraction, splicing, separation and mapping operations to obtain initial image features, initial text features and initial fusion features, including:

[0024] Global marks are extracted from the image feature and the text feature respectively to obtain image mark features and text mark features;

[0025] The image mark features and the text mark features are spliced to obtain a fusion feature;

[0026] Global marks are separated from the image feature and the text feature respectively to obtain image embedding features and text embedding features;

[0027] The image embedding features, the text embedding features and the fusion feature are respectively mapped to obtain initial image features, initial text features and initial fusion features.

[0028] Optionally, the decoupling total loss is obtained according to the initial fusion feature, the initial image feature and the initial text feature, including:

[0029] According to the initial fusion feature, the initial image feature and the initial text feature, a variational mutual information is calculated;

[0030] According to the variational mutual information, an image decoupling loss and a text decoupling loss are calculated, and the image decoupling loss and the text decoupling loss are summed to obtain a decoupling total loss.

[0031] Optionally, the image embedding feature and the text embedding feature are obtained according to the image feature and the text feature, the image embedding feature and the text embedding feature are nonlinearly transformed to obtain nonlinear transformation features, the nonlinear transformation features are weighted to obtain attention scores, the weighted features are obtained according to the attention scores and the nonlinear transformation features, and the weighted features include:

[0032] The image embedding feature and the text embedding feature are obtained according to the image feature and the text feature.

[0033] The image embedding feature and the text embedding feature are nonlinearly transformed to obtain image nonlinear transformation features and text nonlinear transformation features, and the image nonlinear transformation features and the text nonlinear transformation features are taken as the nonlinear transformation features.

[0034] A weight vector of the nonlinear transformation features is obtained.

[0035] The attention scores are obtained according to the nonlinear transformation features and the weight vector.

[0036] The attention scores are normalized to obtain normalized scores.

[0037] The weighted features are obtained according to the normalized scores and the nonlinear transformation features.

[0038] Optionally, the image weighted features, the text weighted features and the fusion features are input into a classifier to obtain entropy features, and entropy weights are calculated according to the entropy features, and the entropy weights include:

[0039] The image weighted features, the text weighted features and the fusion features are respectively input into a classifier to obtain image classifier features, text classifier features and fusion classifier features, and the image classifier features, the text classifier features and the fusion classifier features are taken as the entropy features.

[0040] Entropy feature probability distributions are obtained according to the entropy features.

[0041] Entropy values are obtained according to the entropy feature probability distributions, the entropy values are normalized to obtain entropy weights, and the entropy weights include image entropy weights, text entropy weights and fusion entropy weights.

[0042] Optionally, the difference vector is calculated according to the initial image feature, the initial text feature and the initial fusion feature, and the feedforward features are obtained according to the difference vector and the initial features, and the feedforward features include:

[0043] The initial image feature and the initial text feature are linearly transformed to obtain a query vector, a key vector and a value vector.

[0044] The difference vector is calculated according to the query vector and the key vector.

[0045] Based on the difference vector, initial image features, and initial text features, layer-normalized features are obtained;

[0046] The layer normalized features are input into the feedforward neural network to obtain feedforward features, which include image feedforward features, text feedforward features, and fused feedforward features.

[0047] Optionally, the final features are input into the prediction classifier to obtain predicted values, and the prediction loss is obtained based on the predicted values ​​and the ground truth set, expressed as:

[0048]

[0049] in, Indicates the true label, Represents the categories estimated from the training set. The prior probability, For predicted values, As a feature of fusion, For a set of categories, It is a set of categories Any category in the table indicates that the predictions for all categories are summed when calculating the loss function.

[0050] A system for identifying genuine and fake news includes:

[0051] The first acquisition module is used to acquire the training news dataset, which includes a news image set, a news text set, and a ground truth annotation set.

[0052] The model building and training module is used to build a prediction model. The prediction model is trained using a training news dataset to obtain the prediction loss, total classification loss, and decoupling total loss. The prediction loss, total classification loss, and decoupling total loss are summed to obtain the total loss. The prediction model parameters that minimize the total loss are obtained to obtain the optimal parameters. The prediction model is updated according to the optimal parameters to obtain the optimal prediction model.

[0053] The second acquisition module is used to acquire the actual news dataset;

[0054] The prediction module is used to input the actual news dataset into the optimal prediction model to obtain the prediction result, which is whether the news is true or false.

[0055] The process of training the prediction model using the training news dataset to obtain the prediction loss, total classification loss, and decoupling total loss is as follows:

[0056] Extract features from the news image set to obtain image features; extract features from the news text set to obtain text features.

[0057] performing global feature extraction, splicing, separation and mapping operations on the image features and the text features to obtain initial image features, initial text features and initial fusion features;

[0058] obtaining a decoupling total loss according to the initial fusion features, initial image features and initial text features;

[0059] obtaining image embedding features and text embedding features according to the image features and the text features, performing nonlinear transformation on the image embedding features and the text embedding features to obtain nonlinear transformation features, weighting the nonlinear transformation features to obtain attention scores, obtaining weighted features according to the attention scores and the nonlinear transformation features, the weighted features including image weighted features and text weighted features;

[0060] inputting the image weighted features, the text weighted features and the initial fusion features into a classifier to obtain entropy features, calculating entropy weights according to the entropy features, the entropy weights including image entropy weights, text entropy weights and fusion entropy weights;

[0061] obtaining image classifier features, text classifier features and fusion classifier features according to the image weighted features, the text weighted features and the fusion features, obtaining image classification loss, text classification loss and fusion classification loss according to the image classifier features, the text classifier features, the fusion classifier features and a real label set;

[0062] obtaining a total classification loss according to the image classification loss, the text classification loss, the fusion classification loss, the image entropy weights, the text entropy weights and the fusion entropy weights;

[0063] calculating a difference vector according to the initial image features, the initial text features and the initial fusion features, and obtaining a feedforward feature according to the difference vector and the initial features;

[0064] performing attention update according to the feedforward feature, a query vector, a key vector and a value vector to obtain an updated feature, the updated feature including image updated features, text updated features and fusion updated features, the query vector, the key vector and the value vector being obtained through the feedforward feature;

[0065] splicing the image updated features, the text updated features and the fusion updated features to obtain a final feature;

[0066] inputting the final feature into a prediction classifier to obtain a prediction value, and obtaining a prediction loss according to the prediction value and the real label set.

[0067] A terminal device comprises a memory, a processor, the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, an information entropy guided debiasing multi-modal fake news detection method is adopted.

[0068] A computer readable storage medium, the computer readable storage medium stores a computer program, and when the processor loads and executes the computer program, an information entropy guided debiasing multi-modal fake news detection method is adopted.

[0069] The beneficial effects of the present application are:

[0070] 1. Mutual information constraint is used to realize modal feature decoupling, and an information entropy adaptive mechanism is used to dynamically adjust the learning weight of each modal. The modal rebalancing mechanism guided by entropy models the uncertainty specific to the modal to adaptively adjust the loss weight. By assigning a stronger learning signal to the weak modal, EMR realizes dynamic rebalancing and promotes cross-modal collaborative optimization, extracts more accurate features and improves the accuracy of news true or false judgment.

[0071] 2. In each modal branch, the present application uses a self-attention mechanism to capture the interdependence within the modal, thereby strengthening the key information in each modal. For cross-modal interaction, the present application introduces a cross-attention mechanism, which not only models the interdependence within the modal, but also effectively captures the complementary information between the modal. Specifically, the text, image and cross-modal features are spliced as the key (Key) and value (Value) in the cross-attention, while the cross-modal features themselves are used as the query (Query), thereby promoting information transmission and fusion between different modalities, extracting more accurate features and improving the accuracy of fake news detection. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 The flowchart of the information entropy guided debiasing multi-modal fake news detection method of the present application is shown in the figure.

[0073] Figure 2 The training flowchart of the prediction model of the present application is shown in the figure. DETAILED DESCRIPTION

[0074] An information entropy guided debiasing multi-modal fake news detection method, as shown in the figure, comprises: Figure 1

[0075] S1, obtain a training news data set, the training news data set comprises a news image set, a news text set and a true label set.

[0076] Specifically, the training news data set is represented as: ​

[0077]

[0078] wherein, represents a set of news texts, is the Nth news text, represents a corresponding news image, represents the Nth news image, is a set of true labels for binary classification, wherein when represents fake news, represents a real news sample, is the true or false of the Nth news.

[0079] S2, a prediction model is constructed, the prediction model is trained using the training news data set, a prediction loss, a total classification loss and a decoupling total loss are obtained, the prediction loss, the total classification loss and the decoupling total loss are summed to obtain a total loss, the prediction model parameters at which the total loss is minimum are obtained, the optimal parameters are obtained, the prediction model is updated according to the optimal parameters, and the optimal prediction model is obtained.

[0080] Specifically, the total loss is represented as:

[0081]

[0082] wherein, is the total loss, is the prediction loss, is the decoupling total loss, is the total classification loss, and is a hyperparameter for controlling the relative importance of the MDR loss (decoupling total loss) and the EMR loss (total classification loss) in the overall objective.

[0083] S3, an actual news data set is obtained.

[0084] S4, the actual news data set is input into the optimal prediction model to obtain a prediction result, and the prediction result is that the news is true or false.

[0085] As shown in Figure 2 , the process of training the prediction model using the training news data set to obtain the prediction loss, the total classification loss and the decoupling total loss is as follows:

[0086] S20, extracting features of the news image set to obtain image features, and extracting features of the news text set to obtain text features.

[0087] Specifically, in order to extract discriminative semantic information from different modalities, the present application uses independent encoders to obtain feature representations of text and images respectively. Given an input text sequence The present application utilizes The context-aware wordpiece embedding (text feature) is obtained and represented as:

[0088]

[0089] wherein, is the text feature, represents the encoded wordpiece embedding set, is the batch size, is the wordpiece sequence length, is the embedding dimension. Each wordpiece embedding encodes the syntactic and semantic contextual information of the th wordpiece.

[0090] MAE is used to extract image block-level visual feature representation:

[0091]

[0092] wherein, is the Nth news image, is the image feature.

[0093] S21, global feature extraction, splicing, separation and mapping operations are performed on the image feature and the text feature to obtain initial image features, initial text features and initial fusion features.

[0094] Global feature extraction, splicing, separation and mapping operations are performed on the image feature and the text feature to obtain initial image features, initial text features and initial fusion features, including:

[0095] S211, global markers are extracted from the image feature and the text feature respectively to obtain image marker features and text marker features.

[0096] S212, the image marker features and the text marker features are spliced to obtain fusion features.

[0097] S213, global markers are separated from the image feature and the text feature respectively to obtain image embedding features and text embedding features.

[0098] S214, the image embedding features, the text embedding features and the fusion features are respectively mapped to obtain initial image features, initial text features and initial fusion features.

[0099] Specifically, the present application constructs a multi-modal shared branch to form a joint representation by extracting and splicing the global marker [CLS], represented as:

[0100]

[0101] in, Label features for images, For text tagging features, This is a feature of fusion. and Global semantic vectors from text and image modalities, respectively. The final result... It incorporates shared semantic information across modalities.

[0102] Simultaneously, this invention extracts the global [CLS] representation from the sequence embedding to obtain modality-specific features:

[0103]

[0104] in, For text embedding features, These are image embedding features.

[0105] To ensure compatibility between subsequent multimodal modeling and attention-based fusion, this invention utilizes a modality-specific nonlinear MLP to... , and Projected into a unified embedded space :

[0106]

[0107] in, As initial text features, As initial image features, These are the initial fusion features.

[0108] In traditional multimodal learning, modality-specific information and shared information are often entangled in the same representation space, leading to significant redundancy and overlap. This hybrid representation not only limits the amount of effective information that can be captured in a fixed dimension, but also makes weak modalities more susceptible to being suppressed by the dominant modality.

[0109] This problem is particularly prominent in fake news detection tasks because different types of forgery often manifest in different modalities. To address this challenge, this invention proposes a Multi-modal Disentangled Representation (MDR) module, which jointly models the initial text features during the feature extraction stage. Initial image features and initial fusion features .

[0110] Through explicit mutual information constraints, MDR decouples shared and modality-specific information, enabling each subspace to carry independent and complementary features. This decoupling not only improves information density and representational capability but also provides a clearer learning path for weak modes, thereby alleviating modality imbalance and enhancing overall robustness. Furthermore, the separation of shared and modality-specific features lays a solid foundation for subsequent cross-modal collaboration and dynamic modulation.

[0111] The MDR module minimizes shared representations. Mutual information (MI) between modality-specific representations is used to suppress redundant dependencies, i.e.:

[0112]

[0113] in, This represents the shared representation that minimizes the initial fused features and the initial text features. To minimize the shared representation between the initial fused features and the initial image features.

[0114] S22. Based on the initial fusion features, initial image features, and initial text features, obtain the total decoupling loss.

[0115] Based on the initial fusion features, initial image features, and initial text features, the total decoupling loss is obtained, including:

[0116] S221. Based on the initial fusion features, initial image features, and initial text features, the variational mutual information is calculated.

[0117] S222. Based on variational mutual information, calculate the image decoupling loss and the text decoupling loss, and sum the image decoupling loss and the text decoupling loss to obtain the total decoupling loss.

[0118] Specifically, mutual information (MI) measures the information of two random variables. and The statistical dependencies between them, for , for or The definition is as follows:

[0119]

[0120] in, For joint distribution, For variables marginal distribution, For variables marginal distribution, As expected.

[0121] Because it is difficult to accurately obtain the conditional distribution during training. This invention employs the CLUB approximation (Contrastive Log-ratio Upper Bound), which provides a computable upper bound for the MI.

[0122] = / .

[0123] For a sample pair For small batches of data, the unbiased estimate is as follows:

[0124]

[0125] in, For conditional distribution, it means that under certain conditions... In this case, The distribution, and To represent the first in a small batch In a sample pair and feature, Indicates the first in a small batch one sample feature, This represents the total number of samples. This is about joint distribution. Down, The expectation. That is, given all possible... u and v In the case of sample pairs, conditional probability The expected value of the logarithm, marginal distribution and Down, The expectation, this expectation first of all u marginal distribution Take the expectation, and then... v marginal distribution Take the expected value.

[0126] However, in DeMIND, the initial fusion features Compared with initial image features and initial text features ( and The actual conditional distribution is obtained simultaneously, but in practice it is difficult to directly access the true conditional distribution. Therefore, this invention introduces a variational approximation, which... Using a multilayer perceptron (MLP) Parameterization, where For learnable parameters, The variational approximation of the conditional distribution is denoted by , which represents the result obtained through training. The conditional probability distribution is used to approximate the true conditional distribution. .

[0127] The variational form of mutual information estimation—vCLUB—is expressed as:

[0128] .

[0129] In order to improve For true conditional distribution To maximize the approximation capability of observed sample pairs, this invention maximizes the log-likelihood of observed sample pairs, expressed as:

[0130]

[0131] in, This is the log-likelihood loss.

[0132] Therefore, the decoupling losses for text-specific and image-specific features are defined as follows:

[0133]

[0134]

[0135] in, For text decoupling loss, Image decoupling loss, The log-likelihood loss is calculated for the initial fused features and the initial text features. The log-likelihood loss is calculated using the initial fused features and the initial image features. The mutual information between the initial fused features and the initial text features. For the mutual information between the initial fused features and the initial image features, For the variational approximation parameters of the text, represents the variational approximation parameters of the image.

[0136] The total decoupling loss of the MDR module is defined as:

[0137] .

[0138] S23. Obtain image embedding features and text embedding features based on image features and text features. Perform nonlinear transformation on the image embedding features and text embedding features to obtain nonlinear transformation features. Weight the nonlinear transformation features to obtain attention scores. Based on the attention scores and nonlinear transformation features, obtain weighted features, which include image weighted features and text weighted features.

[0139] Specifically, in multimodal fake news detection, although each modality contains discriminative cues, joint training often leads to the dominant modality gaining an advantage, thereby suppressing the contributions of weak modalities. When a fixed loss weight is used, the gradient from the dominant modality further reinforces this imbalance. To address this issue, this invention proposes an entropy-guided modality rebalancing (EMR) mechanism, which models modality-specific uncertainties during training to adaptively adjust the loss weights. By assigning stronger learning signals to weak modalities, EMR achieves dynamic rebalancing and promotes cross-modal collaborative optimization.

[0140] Since the original modal features in fake news detection often contain noise or redundant information, directly applying EMR may inadvertently weaken key signals. To address this, this invention introduces a token-level attention mechanism in the text and image modal branches to achieve fine-grained weighting at the token level, thereby highlighting task-related information and suppressing irrelevant factors.

[0141] For any mode The sequence length of this invention is denoted as... (i.e., text) and images ).set up For the first Embedding of a word or image patch, where For a unified embedding dimension, Text embedding features Or image embedding features .

[0142] Image embedding features and text embedding features are obtained based on image features and text features. These features are then subjected to a nonlinear transformation to obtain nonlinear transformed features. These nonlinear transformed features are then weighted to obtain an attention score. Based on the attention score and the nonlinear transformed features, a weighted feature is obtained. The weighted feature includes image-weighted features and text-weighted features, including:

[0143] S231. Obtain image embedding features and text embedding features based on image features and text features.

[0144] S232. Perform nonlinear transformation on the image embedding features and text embedding features to obtain image nonlinear transformation features and text nonlinear transformation features, and use the image nonlinear transformation features and text nonlinear transformation features as nonlinear transformation features.

[0145] S233. Obtain the weight vector of the nonlinear transformation characteristics.

[0146] S234. Based on the nonlinear transformation characteristics and weight vector, obtain the attention score.

[0147] S235. Normalize the attention score to obtain the normalized score.

[0148] S236. Based on the normalized score and nonlinear transformation characteristics, the weighted features are obtained.

[0149] S24. Input the image weighted features, text weighted features, and fusion features into the classifier to obtain the entropy features. Calculate the entropy weights based on the entropy features. The entropy weights include the image entropy weight, text entropy weight, and fusion entropy weight.

[0150] Specifically, each embedding passes through a biased fully connected layer. And through nonlinear functions Transformation yields an intermediate representation:

[0151]

[0152] Among them, when o = i hour, For image nonlinear transformation features, For image embedding features, when o= t hour, This is a nonlinear transformation feature of text. For text embedding features.

[0153] Introduce a learnable weight vector To calculate the attention score:

[0154]

[0155] in, To score attention, Let T be the weight vector, and T be the transpose.

[0156] Attention scores are normalized within the sequence using the softmax function:

[0157]

[0158] in, To score the normalization, For this is the first Attention score for each token or patch j is an index in the sequence, indicating the modality. In, all Each of the Tokens or Patches.

[0159] Modal The aggregation representation is obtained by weighted summation, and is expressed as:

[0160]

[0161] in, These are weighted features.

[0162] Image-weighted features, text-weighted features, and fused features are input into the classifier to obtain entropy features. Entropy weights are then calculated based on these entropy features, including:

[0163] S241. Input the image weighted features, text weighted features, and fusion features into the classifier respectively to obtain image classifier features, text classifier features, and fusion classifier features. Use the image classifier features, text classifier features, and fusion classifier features as entropy features.

[0164] Specifically, to quantify the prediction uncertainty of each modality, this invention calculates the conditional information entropy of text, image, and cross-modal shared representations. Specifically, the text and image modal representations after word-level attention aggregation are respectively... and Cross-modal shared branches remain as For each modality, the present invention adds an independent classifier. To generate the corresponding logits:

[0165]

[0166] in, For text classifier features, For image classifier features, To fuse classifier features.

[0167] S242. Obtain the probability distribution of entropy features based on the entropy features.

[0168] Specifically, logits are processed through a softmax layer to obtain the class probabilities for each modality:

[0169]

[0170] in, Indicates the batch sample index. Indicates category index, Indicates the first Each sample in modality Subcategories The predicted probability is given by , where j represents the index of all categories (i.e., j=1,…,K). Let be the logit of the b-th sample pair for class k under modality z (i.e., the original output of the model).

[0171] S243. Based on the probability distribution of entropy features, obtain the entropy value. Normalize the entropy value to obtain the entropy weight. The entropy weight includes image entropy weight, text entropy weight, and fusion entropy weight.

[0172] Specifically, the entropy value is calculated as follows:

[0173]

[0174] in, The entropy value is the value obtained from it. , and It can serve as a quantitative indicator of modality-specific uncertainty. A higher entropy value indicates lower prediction confidence, suggesting that the modality has weaker discriminative ability in the current training phase. Entropy-based metrics can naturally achieve adaptive weighting across modalities, allowing weak modalities to be explicitly strengthened during optimization.

[0175] The entropy value leads to the normalized weights as follows:

[0176]

[0177] in, For entropy weights, These normalized weights , and It is used to adjust the contribution of each modality during training, so that weak modalities can be given higher weights.

[0178] S25. Based on the image weighted features, text weighted features, and initial fusion features, obtain the image classifier features, text classifier features, and fusion classifier features. Based on the image classifier features, text classifier features, fusion classifier features, and the ground truth set, obtain the image classification loss, text classification loss, and fusion classification loss.

[0179] Each modality branch adjusts the classification loss using logit. Supervision is represented as:

[0180]

[0181] in, For the real labeled set, when z = t hour, For text classification loss, when z = i hour, For text classification loss, when z = m hour, To incorporate classification losses.

[0182] S26. Based on the image classification loss, text classification loss, fusion classification loss, image entropy weight, text entropy weight, and fusion entropy weight, obtain the total classification loss.

[0183] Specifically, the overall loss function of EMR is defined as:

[0184]

[0185] in, To fuse entropy weights, For text entropy weights, Image entropy weights This represents the total classification loss.

[0186] During training, EMR dynamically updates the modal weights based on the entropy values ​​of each modality. This adaptively enhances the gradient contribution of weak modes. An entropy-guided weighting mechanism can improve modal complementarity, alleviate modal imbalance, and thus enhance the robustness and discriminative ability of multimodal models.

[0187] S27. Based on the initial image features, initial text features, and initial fusion features, calculate the difference vector. Based on the difference vector and the initial features, obtain the feedforward features.

[0188] Specifically, to effectively model the interaction relationships between text, images, and cross-modal features, this invention proposes an attention-based multi-modal fusion module. This module combines self-attention and cross-attention mechanisms to dynamically capture intermodal dependencies and interaction information, ensuring that the information from each modality is fully optimized in the final fused representation.

[0189] Within each modal branch, this invention employs a self-attention mechanism to capture intramodal dependencies, thereby reinforcing key information within each modality. For cross-modal interactions, this invention introduces a cross-attention mechanism, concatenating text, images, and cross-modal features as keys and values, and using the cross-modal features themselves as queries, thereby capturing complementary information between modalities.

[0190] Based on the initial image features, initial text features, and initial fusion features, a difference vector is calculated. Based on the difference vector and the initial features, feedforward features are obtained, including:

[0191] S271. Linearly transform the initial image features and initial text features to obtain the query vector, key vector, and value vector.

[0192] S272. Calculate the difference vector based on the query vector and the key vector.

[0193] S273. Based on the difference vector and the initial features, the layer normalized features are obtained.

[0194] S274. Input the layer normalized features into the feedforward neural network to obtain feedforward features, which include image feedforward features, text feedforward features, and fused feedforward features.

[0195] S28. Based on the feedforward features, query vector, key vector, and value vector, perform attention update to obtain updated features. The updated features include image update features, text update features, and fused update features. The query vector, key vector, and value vector are obtained through the feedforward features.

[0196] Specifically, firstly, in the text and image branches, this invention models the dependencies within a modality through a self-attention mechanism. For modalities... The present invention first uses input features Linear transformation to query vector ( ), key vector ( ) and value vector ( ).

[0197]

[0198]

[0199]

[0200] in, , and This is a linear projection matrix. Subsequently, the feature representation is updated using a self-attention mechanism. The update processes for image and text features are as follows:

[0201]

[0202]

[0203]

[0204] in, It is a difference vector. For layer normalization, It is a feedforward neural network. To update features, It is a feedforward feature.

[0205] In the cross-modal branch, this invention utilizes a cross-attention mechanism to model the interaction between modalities. Specifically, text, images, and cross-modal features are concatenated as a key ( ) and value ( And using the cross-modal feature itself as the query ( The feature update process is as follows:

[0206]

[0207] Specifically, the attention update process is as follows:

[0208]

[0209]

[0210]

[0211] in, To integrate and update features, To integrate feedforward features.

[0212] S29. Concatenate the image update features, text update features, and fused update features to obtain the final features.

[0213] Specifically, the splicing process is as follows:

[0214]

[0215] in, For splicing features, Update features for text. Update features for the image. To integrate and update features.

[0216] S30. Input the final features into the predictive classifier to obtain the predicted value. Based on the predicted value and the true labeled set, obtain the prediction loss.

[0217] Specifically, This information is then fed into the final classifier to predict the veracity of the news, represented as:

[0218]

[0219] Specifically, will Recorded as Input into the classifier Indicates the true label, Represents the categories estimated from the training set. The prior probability, For predicted values, As a feature of fusion, For a set of categories, It is a set of categories Any category in the table indicates that the predictions for all categories are summed when calculating the loss function.

[0220] A debiased multimodal fake news detection system based on information entropy guidance includes:

[0221] The first acquisition module is used to acquire the training news dataset, which includes a set of news images, a set of news texts, and a set of ground truth annotations.

[0222] The model building and training module is used to build a prediction model. It trains the prediction model using a training news dataset to obtain the prediction loss, total classification loss, and decoupling total loss. The prediction loss, total classification loss, and decoupling total loss are summed to obtain the total loss. The prediction model parameters that minimize the total loss are obtained to get the optimal parameters. The prediction model is updated based on the optimal parameters to obtain the optimal prediction model.

[0223] The second acquisition module is used to acquire the actual news dataset.

[0224] The prediction module is used to input the actual news dataset into the optimal prediction model and obtain the prediction result, which is whether the news is true or false.

[0225] The process of training the prediction model using the training news dataset and obtaining the prediction loss, total classification loss, and decoupling total loss is as follows:

[0226] Extract features from the news image set to obtain image features, and extract features from the news text set to obtain text features.

[0227] Global feature extraction, concatenation, separation, and mapping operations are performed on image features and text features to obtain initial image features, initial text features, and initial fused features.

[0228] The total decoupling loss is obtained based on the initial fusion features, initial image features, and initial text features.

[0229] Image embedding features and text embedding features are obtained based on image features and text features. The image embedding features and text embedding features are then subjected to nonlinear transformation to obtain nonlinear transformation features. The nonlinear transformation features are then weighted to obtain attention scores. Based on the attention scores and the nonlinear transformation features, weighted features are obtained, including image weighted features and text weighted features.

[0230] Image weighted features, text weighted features, and initial fusion features are input into the classifier to obtain entropy features. Entropy weights are calculated based on the entropy features. Entropy weights include image entropy weights, text entropy weights, and fusion entropy weights.

[0231] Based on image weighted features, text weighted features, and fusion features, we obtain image classifier features, text classifier features, and fusion classifier features. Based on image classifier features, text classifier features, fusion classifier features, and the ground truth annotation set, we obtain image classification loss, text classification loss, and fusion classification loss.

[0232] The total classification loss is obtained by considering the image classification loss, text classification loss, fusion classification loss, image entropy weight, text entropy weight, and fusion entropy weight.

[0233] Based on the initial image features, the initial text features, and the initial fusion features, a difference vector is calculated, and based on the difference vector and the initial features, a feedforward feature is obtained.

[0234] Based on the feedforward features, query vector, key vector, and value vector, attention is updated to obtain updated features. The updated features include image updated features, text updated features, and fused updated features. The query vector, key vector, and value vector are obtained through the feedforward features.

[0235] The image update features, text update features, and fused update features are concatenated to obtain the final features.

[0236] The final features are input into the predictive classifier to obtain the predicted values. Based on the predicted values ​​and the true labeled set, the prediction loss is obtained.

[0237] This application also discloses a terminal device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor loads and executes the computer program, it employs a de-biased multimodal fake news detection method guided by information entropy.

[0238] The terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server. The terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0239] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it.

[0240] The memory can be an internal storage unit of the terminal device, such as a hard disk or RAM of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) equipped on the terminal device. Furthermore, the memory can be a combination of internal storage units and external storage devices of the terminal device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0241] In this terminal device, the information entropy-guided de-biased multimodal fake news detection method described in the above embodiments is stored in the terminal device's memory and loaded and executed on the terminal device's processor for convenient use.

[0242] This application also discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it employs an information entropy-guided method for detecting biased multimodal fake news as described in the above embodiments.

[0243] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0244] The above-described method for detecting biased multimodal fake news based on information entropy guidance is stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the method.

[0245] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0246] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A method for detecting biased multimodal fake news based on information entropy guidance, characterized in that, include: Obtain a training news dataset, which includes a set of news images, a set of news texts, and a set of ground truth annotations; Construct a prediction model, train the prediction model using the training news dataset, and obtain the prediction loss, total classification loss, and decoupling total loss. Sum the prediction loss, total classification loss, and decoupling total loss to obtain the total loss. Obtain the prediction model parameters that minimize the total loss to obtain the optimal parameters. Update the prediction model according to the optimal parameters to obtain the optimal prediction model. Obtain actual news datasets; The actual news dataset is input into the optimal prediction model to obtain the prediction result, which indicates whether the news is true or false. The process of training the prediction model using the training news dataset to obtain the prediction loss, total classification loss, and decoupling total loss is as follows: Extract features from the news image set to obtain image features; extract features from the news text set to obtain text features. The image features and text features are subjected to global feature extraction, concatenation, separation and mapping operations to obtain initial image features, initial text features and initial fused features; The decoupling total loss is obtained based on the initial fusion features, initial image features, and initial text features. Image embedding features and text embedding features are obtained based on the image features and text features. The image embedding features and text embedding features are then subjected to a nonlinear transformation to obtain nonlinear transformation features. The nonlinear transformation features are then weighted to obtain an attention score. Based on the attention score and the nonlinear transformation features, a weighted feature is obtained. The weighted feature includes image weighted features and text weighted features. The image weighted features, text weighted features, and initial fusion features are input into the classifier to obtain entropy features. Entropy weights are calculated based on the entropy features. The entropy weights include image entropy weights, text entropy weights, and fusion entropy weights. Based on the image weighted features, text weighted features, and initial fusion features, image classifier features, text classifier features, and fusion classifier features are obtained. Based on the image classifier features, text classifier features, fusion classifier features, and the ground truth set, image classification loss, text classification loss, and fusion classification loss are obtained. The total classification loss is obtained by considering the image classification loss, text classification loss, fusion classification loss, image entropy weight, text entropy weight, and fusion entropy weight. Based on the initial image features, the initial text features, and the initial fusion features, a difference vector is calculated, and based on the difference vector and the initial features, a feedforward feature is obtained. Based on the feedforward features, query vector, key vector, and value vector, attention is updated to obtain updated features. The updated features include image updated features, text updated features, and fused updated features. The query vector, key vector, and value vector are obtained through the feedforward features. The image update features, the text update features, and the fused update features are concatenated to obtain the final features; The final features are input into the prediction classifier to obtain the predicted value. Based on the predicted value and the real labeled set, the prediction loss is obtained.

2. The information entropy-guided de-biased multimodal fake news detection method as described in claim 1, characterized in that, The image features and text features are subjected to global feature extraction, concatenation, separation, and mapping operations to obtain initial image features, initial text features, and initial fused features, including: Global markers are extracted from the image features and text features respectively to obtain image marker features and text marker features; The image marker features and text marker features are concatenated to obtain the fused features; Global markers are separated from the image features and text features respectively to obtain image embedding features and text embedding features; The image embedding features, text embedding features, and fusion features are mapped to obtain initial image features, initial text features, and initial fusion features.

3. The information entropy-guided de-biased multimodal fake news detection method as described in claim 1, characterized in that, The process of obtaining the total decoupling loss based on the initial fusion features, initial image features, and initial text features includes: Variational mutual information is calculated based on the initial fusion features, initial image features, and initial text features. Based on the variational mutual information, the image decoupling loss and the text decoupling loss are calculated. The image decoupling loss and the text decoupling loss are summed to obtain the total decoupling loss.

4. The information entropy-guided de-biased multimodal fake news detection method as described in claim 1, characterized in that, The process involves obtaining image embedding features and text embedding features based on the image features and text features, performing a nonlinear transformation on the image embedding features and text embedding features to obtain nonlinear transformation features, weighting the nonlinear transformation features to obtain an attention score, and obtaining weighted features based on the attention score and the nonlinear transformation features, including: Based on the image features and text features, image embedding features and text embedding features are obtained; The image embedding features and text embedding features are subjected to nonlinear transformation to obtain image nonlinear transformation features and text nonlinear transformation features, which are then used as nonlinear transformation features. Obtain the weight vector of the nonlinear transformation characteristics; Based on the nonlinear transformation characteristics and weight vector, the attention score is obtained; The attention score is normalized to obtain the normalized score; The weighted features are obtained based on the normalized score and the nonlinear transformation characteristics.

5. The information entropy-guided de-biased multimodal fake news detection method as described in claim 1, characterized in that, The process of inputting the image-weighted features, text-weighted features, and fused features into a classifier to obtain entropy features, and calculating entropy weights based on the entropy features, includes: The image weighted features, text weighted features, and fusion features are respectively input into the classifier to obtain image classifier features, text classifier features, and fusion classifier features. The image classifier features, text classifier features, and fusion classifier features are used as entropy features. The probability distribution of entropy features is obtained based on the entropy characteristics. Based on the entropy feature probability distribution, the entropy value is obtained. The entropy value is normalized to obtain the entropy weight, which includes image entropy weight, text entropy weight and fusion entropy weight.

6. The information entropy-guided de-biased multimodal fake news detection method as described in claim 1, characterized in that, The step of calculating a difference vector based on the initial image features, the initial text features, and the initial fusion features, and obtaining feedforward features based on the difference vector and the initial features, includes: The initial image features and initial text features are linearly transformed to obtain the query vector, key vector, and value vector; The difference vector is calculated based on the query vector and the key vector. Based on the difference vector, initial image features, and initial text features, layer-normalized features are obtained; The layer normalized features are input into the feedforward neural network to obtain feedforward features, which include image feedforward features, text feedforward features, and fused feedforward features.

7. The information entropy-guided de-biased multimodal fake news detection method as described in claim 1, characterized in that, The final features are input into the prediction classifier to obtain predicted values. Based on the predicted values ​​and the ground truth set, the prediction loss is obtained, expressed as: in, Indicates the true label, Represents the categories estimated from the training set. The prior probability, For predicted values, As a feature of fusion, For a set of categories, It is a set of categories Any category in the table indicates that the predictions for all categories are summed when calculating the loss function.

8. A de-biased multimodal fake news detection system based on information entropy guidance, characterized in that, include: The first acquisition module is used to acquire the training news dataset, which includes a news image set, a news text set, and a ground truth annotation set. The model building and training module is used to build a prediction model. The prediction model is trained using a training news dataset to obtain the prediction loss, total classification loss, and decoupling total loss. The prediction loss, total classification loss, and decoupling total loss are summed to obtain the total loss. The prediction model parameters that minimize the total loss are obtained to obtain the optimal parameters. The prediction model is updated according to the optimal parameters to obtain the optimal prediction model. The second acquisition module is used to acquire the actual news dataset; The prediction module is used to input the actual news dataset into the optimal prediction model to obtain the prediction result, which is whether the news is true or false. The process of training the prediction model using the training news dataset to obtain the prediction loss, total classification loss, and decoupling total loss is as follows: Extract features from the news image set to obtain image features; extract features from the news text set to obtain text features. The image features and text features are subjected to global feature extraction, concatenation, separation and mapping operations to obtain initial image features, initial text features and initial fused features; The decoupling total loss is obtained based on the initial fusion features, initial image features, and initial text features. Image embedding features and text embedding features are obtained based on the image features and text features. The image embedding features and text embedding features are then subjected to a nonlinear transformation to obtain nonlinear transformation features. The nonlinear transformation features are then weighted to obtain an attention score. Based on the attention score and the nonlinear transformation features, a weighted feature is obtained. The weighted feature includes image weighted features and text weighted features. The image weighted features, text weighted features, and initial fusion features are input into the classifier to obtain entropy features. Entropy weights are calculated based on the entropy features. The entropy weights include image entropy weights, text entropy weights, and fusion entropy weights. Based on the image weighted features, text weighted features, and fusion features, image classifier features, text classifier features, and fusion classifier features are obtained. Based on the image classifier features, text classifier features, fusion classifier features, and the ground truth set, image classification loss, text classification loss, and fusion classification loss are obtained. The total classification loss is obtained by considering the image classification loss, text classification loss, fusion classification loss, image entropy weight, text entropy weight, and fusion entropy weight. Based on the initial image features, the initial text features, and the initial fusion features, a difference vector is calculated, and based on the difference vector and the initial features, a feedforward feature is obtained. Based on the feedforward features, query vector, key vector, and value vector, attention is updated to obtain updated features. The updated features include image updated features, text updated features, and fused updated features. The query vector, key vector, and value vector are obtained through the feedforward features. The image update features, the text update features, and the fused update features are concatenated to obtain the final features; The final features are input into the prediction classifier to obtain the predicted value. Based on the predicted value and the real labeled set, the prediction loss is obtained.

9. A terminal device, comprising a memory and a processor, characterized in that, The memory stores a computer program that can run on a processor, and when the processor loads and executes the computer program, it employs the method described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs the method described in any one of claims 1 to 7.

Images