A multimodal method and apparatus for detecting fake news.

Abstract

This invention relates to a multimodal fake news detection method and apparatus, comprising: calculating cross-domain imbalance weights for each domain based on the number of news samples in each domain and the total number of news samples across all domains; calculating intra-domain imbalance weights for each domain based on the number of genuine news samples and the number of fake news samples in each domain; calculating global imbalance weights based on the total number of genuine news samples and the total number of fake news samples in the training set; training an initial news detection model based on the training set and a loss function embedding the cross-domain imbalance weights, intra-domain imbalance weights, and global imbalance weights; and inputting the news to be detected into the news detection model to obtain a detection result indicating whether the news to be detected is genuine or fake news. The method provided by this invention effectively guides the model to focus on samples with low data volume and low sample proportion, ensuring that the model can maintain good learning performance and detection capabilities even in scenarios with imbalanced data distribution.

Description

Technical Field

[0001] This invention belongs to the field of neural network and fake news detection technology, specifically relating to a multimodal fake news detection method and device. Background Technology

[0002] Misinformation on social media is increasingly appearing in multimodal forms (such as text and images) and spanning multiple domains, including politics, health, and entertainment. Fake news detection is evolving from traditional unimodal classification tasks to robust detection tasks that integrate multimodal reasoning, knowledge-enhanced verification, and cross-topic, cross-event, and cross-platform capabilities. In this context, multimodal information plays a crucial role in fake news detection because while individual texts or images may convey truthful content, their combination can produce misleading or even false information.

[0003] However, in practical applications, an effective detection model not only needs to integrate heterogeneous textual and visual evidence, but also must be able to learn from multiple source domains and further generalize to new or disseminated target domains. This multi-domain scenario is significantly challenging because different domains differ in writing style, topic distribution, and dissemination patterns, leading to a noticeable domain bias problem. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multimodal method and apparatus for detecting fake news.

[0005] The present invention includes a multimodal fake news detection method, comprising: acquiring an initial news detection model and a training set; wherein the training set includes news samples from multiple domains, the news samples include real news samples and fake news samples, and each news sample includes a text modality and an image modality; calculating cross-domain imbalance weights for each domain based on the number of news samples in each domain and the total number of news samples in all domains; calculating intra-domain imbalance weights for each domain based on the number of real news samples and the number of fake news samples in each domain; calculating global imbalance weights based on the total number of real news samples and the total number of fake news samples in the training set; training the initial news detection model based on the training set and a loss function embedded with the cross-domain imbalance weights, intra-domain imbalance weights, and global imbalance weights to obtain a trained news detection model; inputting news to be detected, including text modality and image modality, into the news detection model to obtain a detection result output by the news detection model indicating whether the news to be detected is real news or fake news.

[0006] Optionally, the calculation formula for the above cross-domain imbalance weights includes:

[0007]

[0008]

[0009]

[0010]

[0011] in, For the first Cross-domain unbalanced weights in each domain For the first Scarcity score in each field , For the first The number of news samples in each field , This refers to the number of news samples from the field with the largest number of news samples. , For global imbalance score, The total sample is discretely distributed. , , , The total number across all fields. The number of news samples in the first domain. For the first The number of news samples in each field For the first The number of news samples in each field Discrete distribution of the total sample The average entropy, Discrete distribution of the total sample The average Gini coefficient, For cross-domain unbalanced Gini weights.

[0012] Optionally, the formula for calculating the unbalanced weights within the above domain includes:

[0013]

[0014]

[0015]

[0016] in, For the first Intradomain imbalance weights for each domain For the first Intra-domain imbalance scores for each domain For the first Discrete distribution of samples in each domain , For the discrete distribution of the sample The average entropy, For the discrete distribution of the sample The average Gini coefficient, , , For the first The number of news samples in each field For the first The number of real news samples in each field No. The number of fake news samples in each field This represents the unbalanced Gini weights within the domain.

[0017] Optionally, the formula for calculating the above-mentioned global imbalance weights includes:

[0018] in, For the aforementioned globally unbalanced weights, For true and false labels, when , For the globally imbalanced weights of the real news samples, when , For the globally imbalanced weights of fake news samples; Let the number of all real news samples or the number of all fake news samples in the training set mentioned above be the number of samples in the training set. , Let be the number of all real news samples in the training set mentioned above, when , The number of all fake news samples in the training set mentioned above. The number of all news samples in the training set mentioned above. This represents the globally unbalanced Gini weight.

[0019] Optionally, the loss function mentioned above includes:

[0020] in, For weighted prototype clustering loss, This represents the number of news samples in the current training batch. For the first The news sample belongs to the first The aforementioned intra-domain imbalance weights for each domain For the first The projection vector of a news sample in the prototype space. For the additional introduction of the first The news sample belongs to the first The corresponding true / false labels for each field are: prototype vector, For the first The true and false labels of a news sample Indicates the first The news sample is a genuine news sample. Indicates the first One news sample is a fake news sample. Calculated for the L2 norm; the above loss function also includes:

[0021]

[0022] in, To monitor and compare losses, For the first Instance-supervised contrastive loss for a single news sample. For the first The news sample belongs to the first The aforementioned intra-domain imbalance weights for each domain For the first A positive sample set of news samples, The number of positive samples. , , Indicates except the first News samples other than the news sample, For the first The field to which each news sample belongs, For the first The field to which each news sample belongs; No. The true and false labels of a news sample Indicates the first The news sample is a genuine news sample. Indicates the first One news sample is a fake news sample; Denotes the first positive sample in the positive sample set. A news sample; For the first The projection vector of a news sample in the prototype space. For the first The projection vector of a news sample in the prototype space; For cosine similarity calculation, This is the temperature coefficient.

[0023] Optionally, the loss function mentioned above includes:

[0024] in, For the total loss function, For weighted cross-entropy calculation, The output of the initial news detection model mentioned above is the first... The probability that a news sample is fake news. For the first The true and false labels of a news sample Indicates the first The news sample is a genuine news sample. Indicates the first One news sample is a fake news sample; For the aforementioned globally unbalanced weights, when , For the globally imbalanced weights of the real news samples, when , For the globally imbalanced weights of fake news samples; The output of the initial news detection model mentioned above is the first... Modalities of a news sample The probability that it is fake news. For text modality, For image modality, It is a mixed mode. For the first The news sample belongs to the first Cross-domain unbalanced weights in each domain This represents the sample-level backbone network loss.

[0025] Optionally, the above-mentioned input of the news to be detected, including text modalities and image modalities, into the news detection model to obtain the detection result of whether the news to be detected is real or fake news, as output by the news detection model, includes: inputting the news to be detected into the news detection model to obtain the initial prediction probability output by the news detection model; obtaining the domain to which the news to be detected belongs, obtaining the initial prototype vector of the news to be detected based on the domain, and mixing the initial prototype vector with the global prototype vector to form a bridging prototype vector; wherein, the global prototype vector is calculated based on the cross-domain unbalanced weights and the prototype vectors of all domains; calculating the prototype prediction probability based on the projection vector of the news to be detected in the prototype space and the bridging prototype vector; and determining the detection result of whether the news to be detected is real or fake news based on the initial prediction probability and the prototype prediction probability.

[0026] Based on the same inventive concept, this invention also provides a multimodal fake news detection device, comprising: an acquisition module for acquiring an initial news detection model and a training set; wherein the training set includes news samples from multiple domains, the news samples include real news samples and fake news samples, and each news sample includes a text modality and an image modality; a weight calculation module for calculating cross-domain imbalance weights for each domain based on the number of news samples in each domain and the total number of news samples in all domains; calculating intra-domain imbalance weights for each domain based on the number of real news samples and the number of fake news samples in each domain; and calculating global imbalance weights based on the total number of real news samples and the total number of fake news samples in the training set; a training module for training the initial news detection model based on the training set and a loss function embedded with the cross-domain imbalance weights, intra-domain imbalance weights, and global imbalance weights, to obtain a trained news detection model; and a detection module for inputting news to be detected, including text modality and image modality, into the news detection model to obtain a detection result output by the news detection model indicating whether the news to be detected is real or fake news.

[0027] Based on the same inventive concept, the present invention also provides an electronic device, comprising: a processor; a memory for storing executable instructions; wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement any of the methods described above.

[0028] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor performs any of the above-described methods.

[0029] The beneficial effects of this invention are as follows: The method provided by this invention decomposes the data imbalance problem in multi-domain multimodal fake news detection into three levels: cross-domain sample imbalance, intra-domain label imbalance, and global label imbalance. It transforms the statistical information of the dataset into stable training weights, effectively guiding the model to focus on samples with low data volume and low sample ratio. This systematically alleviates the negative impact of multi-level data imbalance on model training and ensures that the model can still maintain good learning effect and detection performance in scenarios with uneven data distribution. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a multimodal-based method for detecting fake news provided in an embodiment of the present invention; Figure 2 A summary table of experimental results provided for embodiments of the present invention; Figure 3 Another summary table of experimental results provided for embodiments of the present invention; Figure 4 Another summary table of experimental results provided for embodiments of the present invention; Figure 5 A schematic diagram of a visual feature space provided for an embodiment of the present invention; Figure 6 Another schematic diagram of the visualized feature space provided in this embodiment of the invention; Figure 7 Another schematic diagram of the visualized feature space provided in this embodiment of the invention; Figure 8 Another schematic diagram of the visualized feature space provided in this embodiment of the invention; Figure 9 Another summary table of experimental results provided for embodiments of the present invention; Figure 10 A schematic diagram of a multimodal fake news detection device provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] To better understand the above-mentioned objectives, features, and advantages of the embodiments of the present invention, the solutions of the embodiments of the present invention will be further described below. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0034] Many specific details are set forth in the following description in order to provide a full understanding of the embodiments of the present invention, but the embodiments of the present invention may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the embodiments of the present invention, and not all embodiments.

[0035] One embodiment of the present invention provides a multimodal fake news detection method, such as... Figure 1As shown, it includes: S1. Obtain the initial news detection model and training set. The training set includes news samples from multiple domains, including real news samples and fake news samples. Each news sample includes both text and image modalities.

[0036] Specifically, multiple fields refer to the field to which the news sample belongs, such as the political field, health field, sports field, literature field, entertainment field, etc.; real news samples can be obtained from historical real news, while fake news samples can be obtained from historical fake news cases or generated randomly; text modality includes the text of the news sample, and image modality includes the accompanying image of the news sample.

[0037] S2. Calculate the cross-domain imbalance weight for each domain based on the number of news samples in each domain and the total number of news samples across all domains. The cross-domain imbalance weight is used to balance the impact of domains with a large number of samples versus those with a small number of samples on model training, thereby enabling the model to learn the sample features of each domain more evenly. Calculate the intra-domain imbalance weight for each domain based on the number of real news samples and fake news samples in each domain. The intra-domain imbalance weight is used to balance the impact of the ratio of real news samples to fake news samples in each domain on model training, ensuring that the model can learn the features of fake news in each domain without losing its ability to identify real news. Calculate the global imbalance weight based on the total number of real news samples and the total number of fake news samples in the training set. The global imbalance weight balances the impact of the ratio of real news samples to fake news samples on model training across all samples, avoiding the influence of prior distribution bias on the model's generalization ability.

[0038] S3. Train the initial news detection model based on the training set and the loss function that embeds cross-domain imbalance weights, intra-domain imbalance weights and global imbalance weights to obtain the trained news detection model.

[0039] In practice, the model uses the training set and loss function to calculate the error between the prediction and the true value, calculates the gradient through backpropagation, and the optimizer continuously updates the parameters, iterating repeatedly to reduce the error until the model training converges, thereby enabling the model to accurately fit data patterns and generalize predictions. During training, global imbalance weights are applied to the classification loss term of the overall samples to correct the uneven distribution of global classes; intra-domain imbalance weights are embedded in the loss calculation of single-domain samples to strengthen the model's learning focus on minority class samples within the same domain; cross-domain imbalance weights are used to balance the contribution of samples from different domains to gradient updates, avoiding model parameters being biased towards popular domains with large amounts of data. In a more specific embodiment, S3 above includes training the initial news detection model based on contrastive learning.

[0040] S4. Input the news to be detected, including text modality and image modality, into the news detection model, and obtain the detection result of the news detection model, which indicates whether the news to be detected is real news or fake news.

[0041] The method provided in this invention decomposes the data imbalance problem in multi-domain, multimodal fake news detection into three levels: cross-domain sample imbalance, intra-domain label imbalance, and global label imbalance. It transforms dataset statistics into stable training weights, effectively guiding the model to focus on samples with low data volume and low sample proportion. This systematically mitigates the negative impact of multi-level data imbalance on model training, ensuring that the model maintains good learning performance and detection capabilities even in scenarios with uneven data distribution. Furthermore, this weighting mechanism relies solely on label statistics, performs only one calculation, and remains unchanged throughout the training process. This avoids the drawback of traditional reweighting strategies that rely on model predictions and easily amplify early model biases, thus improving the stability and reliability of the training process.

[0042] In one specific embodiment, the training set described above is represented as a multi-domain, multimodal fake news dataset. :

[0043] in, and They represent the first The text and image modalities of the news sample Indicates the first The domain tags of the news sample (e.g., science, politics, health, etc.). Is it a true or false label (if the first) The news sample is a genuine news sample. If the first One news sample is a fake news sample. Therefore, each sample contains two modalities and domain information, and both the label distribution and the domain distribution may be imbalanced.

[0044] The goal of this invention is to learn a classifier (i.e., the news detection model described above):

[0045] in, This means that the classifier is fed news items to be detected, including both text and image modalities, along with the domain labels of the news items. The classifier then outputs the true / false labels for the news items accordingly. (If the news to be tested is a genuine news sample,) If the news to be tested is a fake news sample, ).

[0046] In its specific implementation, this invention employs a decoupled multimodal encoder as the backbone network. Specifically, this backbone network constructs three complementary views—a text view, an image view, and a multimodal view—based on a pre-trained encoder and a view-specific feature extractor. It then decouples features for each view to separate domain-related information from domain-invariant semantic information.

[0047] The embodiments of the present invention are built upon the output after filtering by the backbone network described above. Undisclosed content related to the backbone network can be obtained by those skilled in the art by consulting the corresponding technical documents, and will not be elaborated upon here. For each view... The encoder provides a filtered, domain-specific feature. A domain-invariant semantic feature These features serve as inputs to embodiments of the present invention, meaning that embodiments of the present invention extract these features from the training set or the news to be detected, and then use the aforementioned weights and training functions to train a model or use the trained model to perform news detection. Furthermore, each modality uses... and Predicting class probabilities, obtaining ,in, For text modality-based prediction probabilities, For image modality-based prediction probabilities, This is the prediction probability based on the mixed modality. The domain information decision layer further integrates the prediction results from these three views to obtain the final classifier probability. Classifier probability The label used to determine the authenticity of the news article being tested.

[0048] Because the number of cross-domain samples follows a long-tailed distribution, and the ratio of real to fake news may differ across domains, global class priors may also be biased. These imbalances affect different parts of the model: domain-related learning, intra-domain label-related learning, and global class imbalance calibration. If this invention uses only a single weight or a dynamic reweighting strategy, these effects may become confounded and difficult to interpret.

[0049] This invention first provides a method for calculating the information Gini coefficient to measure the degree of data unevenness. The aforementioned cross-domain imbalance weights and intra-domain imbalance weights are calculated based on this method. First, let... Let be the discrete distribution of the news samples, where the th is... A news sample , , Let be the total number of samples, and agree on the following: The present invention defines entropy in its embodiments. and Gini coefficient Normalize them to The information Gini coefficient is obtained by averaging the values ​​over a given interval. The formula is as follows:

[0050]

[0051]

[0052]

[0053]

[0054] To better characterize the multi-level imbalance in the training set, the above embodiments of the present invention introduce an Info-Gini weighting scheme combining entropy and the Gini coefficient. The advantage of this design lies in their complementary sensitivity to imbalance: entropy is more sensitive to highly skewed distributions, while the Gini coefficient contains more information in the moderately skewed range and is more closely related to classification error. The embodiments of the present invention define Info-Gini indices at the inter-domain, intra-domain, and global label levels, and further convert them into training weights. Since both normalized entropy and normalized Gini coefficient increase with increasing distribution uniformity, the above embodiments define the final Info-Gini score as 1 minus their average, making it positively correlated with label concentration. Therefore, a higher score indicates a more concentrated distribution, and thus a greater imbalance. Because these indices rely only on label statistics, they are more stable than reweighting strategies that rely on predictions and are less likely to amplify early model bias.

[0055] Based on the aforementioned information Gini coefficient, in some embodiments, the calculation formula for the aforementioned cross-domain imbalance weight includes:

[0056]

[0057]

[0058]

[0059]

[0060] in, For the first Cross-domain unbalanced weights are used to enhance data-scarce domains during domain-related learning, preventing the model from overfitting to data-rich domains. For the first Scarcity score in each field For the field with the largest sample size, For the domain with the smallest sample size, Approaching 0 avoids overly aggressive reweighting when the domain sizes differ by several orders of magnitude. For the first The number of news samples in each field , This refers to the number of news samples from the field with the largest number of news samples. , For global imbalance score, The total sample is discretely distributed. , , , The total number across all fields. The number of news samples in the first domain. For the first The number of news samples in each field For the first The number of news samples in each field Discrete distribution of the total sample The average entropy, Discrete distribution of the total sample The average Gini coefficient, These are cross-domain unbalanced Gini weights used to control the model's bias. The training intensity.

[0061] Within each domain, the proportion of real news and fake news may differ. Therefore, in some embodiments, the formula for calculating the domain imbalance weight includes:

[0062]

[0063]

[0064]

[0065] in, For the first Intra-domain imbalance weights are applied to domains with higher label imbalance and are used for label-related learning objectives, such as prototype clustering and instance-level contrastive learning. For the first Intra-domain imbalance scores for each domain For the first Discrete distribution of samples in each domain , For the discrete distribution of the sample The average entropy, For the discrete distribution of the sample The average Gini coefficient, , , For the first The number of news samples in each field For the first The number of real news samples in each field No. The number of fake news samples in each field The in-domain unbalanced Gini weights are used to control the model's bias towards the unbalanced Gini coefficients. The training intensity.

[0066] Even if each domain is balanced, the entire training dataset may still have biases in the prior distribution of the classes. Therefore, in some embodiments, the formula for calculating the globally imbalanced weights includes:

[0067] in, For globally unbalanced weights, For true and false labels, when , For the globally imbalanced weights of the real news samples, when , For the globally imbalanced weights of fake news samples; To determine the number of all real news samples or the number of all fake news samples in the training set, when , To determine the number of real news samples in the training set, when , To determine the number of fake news samples in the training set, To determine the number of all news samples in the training set, These are globally unbalanced Gini weights, used to control the model's bias towards... The training intensity.

[0068] The above embodiments first calculate the global category prior. And using a mild category-related weight. To calculate the weighted cross-entropy, this prior correction does not change the composition of mini-batch data compared to oversampling, and is easier to integrate with multi-objective functions.

[0069] Following decoupled multimodal encoders, the remaining problem is often geometric constraints: in data-scarce or label-skewed domains, minority class features become scattered and overlap with majority class features. Therefore, embodiments of the present invention impose explicit geometric constraints by introducing a domain-label prototype space and optimizing it at the instance and prototype levels using clustering, separation, and multi-level contrastive learning.

[0070] For each news sample In this embodiment of the invention, semantic features and filtered domain-specific features are connected to form Then, a multilayer perceptron (MLP) is used to project it onto... In the prototype space of dimension , the projection vector is obtained. The formula is expressed as:

[0071]

[0072] in, For semantic features based on text modality, For semantic features based on image modalities, For semantic features based on mixed modalities, For text-based modal domain features, For image-based modal domain features, For domain features based on hybrid modes, For the set of real numbers, This represents a multilayer perceptron.

[0073] For each domain - true / false label pairs This invention introduces a learnable prototype vector. Each prototype vector represents a "standard template" for each domain and its true / false news, and each prototype vector constitutes a prototype tensor:

[0074] in It is the number of fields. It refers to the number of categories (here, categories represent the truthfulness of news; true news is one category, and fake news is another). This indicates a full slice along the corresponding dimension, meaning all elements along that dimension are selected. This indicates that the domain label has been fixed. and category index true / false labels Then, from the prototype tensor Take the corresponding These prototype vectors are dimensional. They add a domain dimension to the class centers, enabling the model to learn the class centers within each domain, thus capturing the geometry within the domain more accurately.

[0075] In some embodiments, the loss function includes a weighted prototypical clustering loss, which directly reshapes the cluster structure in the representation space, making the classes in a highly imbalanced domain more compact, as shown in the following formula:

[0076] in, For weighted prototype clustering loss, This represents the number of news samples in the current training batch. For the first The news sample belongs to the first Intradomain imbalance weights for each domain For the first The projection vector of a news sample in the prototype space. For the additional introduction of the first The news sample belongs to the first The corresponding true / false labels for each field are: prototype vector, For the first The true and false labels of a news sample Indicates the first The news sample is a genuine news sample. Indicates the first One news sample is a fake news sample. Calculate the L2 norm.

[0077] Features in prototype space In this embodiment of the invention, supervised contrastive learning is used to group samples with the same domain and the same label together and separate all other samples. First, a positive sample set is defined:

[0078] in, For the first A positive sample set of news samples, Indicates except the first News samples other than the news sample, For the first The field to which each news sample belongs, For the first The field to which each news sample belongs; No. The true and false labels of a news sample Indicates the first The news sample is a genuine news sample. Indicates the first One news sample is a fake news sample.

[0079] And use cosine similarity:

[0080] in, and These are all calculation parameters used for substitution and have no practical significance.

[0081] In some embodiments, the instance-by-instance supervised contrast loss is defined as:

[0082] in, For the first Instance-supervised contrastive loss for a single news sample. The number of positive samples. ; Denotes the first positive sample in the positive sample set. A news sample; For the first The projection vector of a news sample in the prototype space. For the first The projection vector of a news sample in the prototype space. For the first The projection vector of a news sample in the prototype space; For cosine similarity calculation, This is the temperature coefficient.

[0083] Then, the batches are averaged, and the label weights within the domain are considered to obtain the supervised contrast loss:

[0084] in, To monitor and compare losses, For the first Supervisory comparison loss for a single news sample For the first The news sample belongs to the first Intradomain imbalance weights for each domain.

[0085] In some embodiments, the loss function provided by the present invention further includes:

[0086] in, For the total loss function, For weighted cross-entropy calculation, The first output of the initial news detection model The probability that a news sample is fake news. For the first The true and false labels of a news sample Indicates the first The news sample is a genuine news sample. Indicates the first One news sample is a fake news sample; For globally unbalanced weights, when , For the globally imbalanced weights of the real news samples, when , For the globally imbalanced weights of fake news samples; The first output of the initial news detection model Modalities of a news sample The probability that it is fake news. For text modality, For image modality, It is a mixed mode. For the first The news sample belongs to the first Cross-domain unbalanced weights in each domain This represents the sample-level backbone network loss.

[0087] To ensure good separation between the two prototype classes in each domain, in another embodiment of the invention, a prototype separation loss function based on interval is further introduced. :

[0088] in, This is the interval hyperparameter. This is the prototype vector of a real news sample. For the prototype vector of fake news samples, this loss increases the loss within each domain. and The distance between them complements the semantic decoupling of the domain and prevents prototype collapse in data-scarce domains.

[0089] In addition to instance-level comparison, in another embodiment, the present invention also imposes comparison constraints on domain-label prototypes to improve cross-domain consistency and class separability. For prototypes... The positive sample set contains samples with the same label. However, prototypes from different domains, and prototypes with different labels, are considered negative sets. Define the negative sample set. , Indicates the first For domains outside of the specified domain, the prototype-level contrastive loss is defined as:

[0090]

[0091] in, For the first Prototype vectors of news samples in each domain. For the first Prototype vectors of news samples in each domain. For the first In each field, and the true / false labels are related to the first... Different prototype vectors of news samples in each domain For temperature coefficients. The prototype-level contrastive loss function encourages alignment of prototypes with the same label across different domains, while maintaining separation of prototypes with different labels; therefore, it is related to the prototype separation loss function. Complementary.

[0092] In another embodiment, the present invention simultaneously preserves the sample-level backbone network loss. and domain-level backbone network losses Furthermore, the loss function is expanded based on all the above embodiments. The overall training objective combines the backbone network loss and the prototype learning loss. The overall training objective loss function is defined as follows:

[0093] In some embodiments, even if the prototype space structure on the training domain is good, the test distribution may still shift between different domains, which reduces the reliability of prototype distance and cross-domain prototype relationships. To improve transfer performance while keeping the training process simple, this invention introduces a lightweight bridging fusion mechanism that does not require adding an additional training loss term. This mechanism first constructs a global class prototype bridge from the learned domain-label prototypes, and then uses the bridge prototype for prototype-based prediction fusion.

[0094] In some embodiments, S4 includes: S41. Input the news to be detected into the news detection model to obtain the initial prediction probability output by the news detection model.

[0095] S42. Obtain the domain of the news to be detected, obtain the initial prototype vector of the news to be detected based on the domain, and mix the initial prototype vector with the global prototype vector to form a bridging prototype vector. The global prototype vector is calculated based on the cross-domain unbalanced weights and the prototype vectors of all domains.

[0096] Specifically, given a set of training domains This invention will use similar prototype vectors Merge into a shared global prototype vector :

[0097] in, The first one in the above embodiments The cross-domain unbalanced weights for each domain allow data-scarce domains to contribute more to the global class prototype. Then, each domain-specific prototype (i.e., the initial prototype vector mentioned above) is combined using convex combinations. This is combined with the corresponding global prototype to obtain the bridging prototype vector. :

[0098] in, The bridging strength coefficient is obtained from this. As a bridging prototype for cross-domain migration, it provides shared class-level references between different domains while preserving the domain-specific structures learned in the above embodiments.

[0099] S43. Calculate the prototype prediction probability based on the projection vector of the news to be detected in the prototype space and the bridging prototype vector.

[0100] Specifically, given a projection vector of a prototype space... This invention calculates the prototype prediction probability based on the squared distance to the bridging prototype vector. :

[0101] in, For the first The news sample belongs to the first The field has true / false labels. The bridging prototype vector, This indicates all true and false tags (i.e., all real news and fake news). For the first The news sample belongs to the first The field has true / false labels. The bridging prototype vector. This probability provides a geometrically perceptual view of the sample relative to the bridging prototype structure.

[0102] S44. The detection result of whether the news to be detected is real news or fake news is determined by combining the initial prediction probability and the prototype prediction probability.

[0103] Specifically, in this embodiment of the invention, the model prediction probability (i.e., the initial prediction probability output by the model) is used. Prototype-based prediction probability Linear fusion, for the first For a given news sample, the final predicted probability is:

[0104] in, It captured the discrimination signals learned by the backbone network, and Reflects the prototype geometry after cross-domain bridging, and integrates weights. Used to control the contribution of prototype-based signals to the final decision.

[0105] This invention primarily evaluates the methods provided in its embodiments on two widely used Chinese benchmark datasets, Weibo and Weibo-21 (hereinafter referred to as IGPC), which are also standard datasets used in recent multi-domain, multimodal fake news detection research. For completeness and for future expansion to a wider range of benchmark datasets, the experiments in this invention also summarize the recently released FineFake dataset.

[0106] Weibo serves as a benchmark for multi-domain, multimodal research. The training set contains 7,532 Weibo posts, of which 3,749 are real samples and 3,783 are fake samples. The test set contains 1,996 Weibo posts, of which 996 are real samples and 1,000 are fake samples. To support multi-domain evaluation, the dataset is further divided into nine domains: finance, health, military, science, politics, international, education, entertainment, and society.

[0107] Weibo-21 contains a total of 9127 Weibo posts, covering the nine domains mentioned above, of which 4640 are real samples and 4487 are fake samples. The experiments in this invention follow the benchmark partitioning and preprocessing protocols used in related multi-domain, multimodal work. Specifically, this invention uses the same cleaning process as in related technologies to remove duplicate and low-quality images to ensure fair comparison.

[0108] FineFake is a state-of-the-art knowledge-enhanced benchmark dataset for fine-grained multi-domain fake news detection. It contains 16,909 samples covering six semantic topics and eight platforms. Each sample includes multimodal content, underlying social context, verified shared knowledge, and fine-grained annotations that go beyond coarse binary labels. Compared to Weibo and Weibo-21, FineFake offers broader topic and platform coverage and richer evidentiary signals, making it a promising benchmark dataset for extending multi-domain, multimodal fake news evaluation beyond traditional binary settings.

[0109] To provide a comprehensive and impartial evaluation of IGPC, this invention compares it with representative methods from three categories: unimodal multidomain models, multimodal unidomain models, and multimodal multidomain models. The main baseline models used in the experiments of this invention include: (1) unimodal multidomain baseline models. MMoE shares a multi-gated hybrid expert architecture across relevant tasks and domains. MoSE extends the ideas of MoE using sequence experts. MDFEND is a robust multidomain model specifically designed for fake news, using domain-aware gating to aggregate expert outputs. 3FEND enriches the domain signal using a memory-guided multi-view framework. (2) Multimodal single-domain baseline. SpotFake combines image and text features for multimodal prediction. CAFE adaptively aggregates single-modal and cross-modal cues by modeling ambiguity. CMC emphasizes cross-modal relevance learning through knowledge distillation. BMR utilizes multi-view guidance and expert-based fusion. MRHFR enhances multimodal reasoning by modeling multiple reading habits. CMDMMD is a distribution alignment method that combines central moment difference and maximum mean difference. It optimizes the distribution consistency between true and false news and different modalities by simultaneously matching the higher-order statistical moments of the feature distribution with the kernel space mean embedding. (3) Multimodal multi-domain baseline model. KATMF models the domain-aware interaction between text, visual and knowledge signals. MMDFND is a cross-domain multimodal error information detection model that improves robustness to domain transformation and text-image inconsistency. DAMMFND is the strongest baseline model in the main setup of this invention. It explicitly decouples domain awareness and semantic multimodal features and can be regarded as the decoupled multimodal encoder mentioned in the above embodiments.

[0110] The specific experimental setup included: for text encoding, the input sequence length was set to 197, and pre-trained BERT (Bidirectional Encoder Representations from Transformers) and CLIP (Contrastive Language-Image Pretraining) text encoders were used for representation. For image encoding, each image was resized to 224×224 pixels before being input into the pre-trained MAE (Masked Autoencoders) and CLIP visual encoders to obtain visual features. The experiments used the Adam optimizer with an initial learning rate of 1e-4 and a weight decay of 5e-5.

[0111] Based on the model in the above embodiments, the main adjustable hyperparameters include: (1) Gini weight. , and They control the strength of the three training weights. (2) Prototype and contrast target: interval and temperature and (3) Migration enhancement: bridging strength and fusion weight Furthermore, the experiments were based on PyTorch and all were run on an NVIDIA RTX 4090 GPU. Model performance was measured by overall F1 score, accuracy (Acc), and area under the curve (AUC).

[0112] Figure 2 The table shows the comparison results of IGPC with representative multi-domain baseline models (including Sci, Mil, Edu, Soc, Pol, Hlth, Fin, Ent, and Dis / Int) on the Weibo, Weibo-21, and FineFake datasets. IGPC achieved the best overall performance on all three datasets. On the Weibo dataset, its F1 score, accuracy, and AUC were 0.954, 0.954, and 0.987, respectively; on the Weibo-21 dataset, they were 0.963, 0.963, and 0.990, respectively; and on the FineFake dataset, they were 0.802, 0.806, and 0.888, respectively.

[0113] Compared to the strongest baseline model, DAMMFND, IGPC improves the overall F1 score by 1.17% on the Weibo dataset, 1.69% on the Weibo-21 dataset, and 0.88% on the FineFake dataset. The corresponding relative improvements in accuracy are 1.06%, 1.69%, and 1.26%, respectively. The AUC improves from 0.983 to 0.987 on the Weibo dataset, from 0.985 to 0.990 on the Weibo-21 dataset, and from 0.874 to 0.888 on the FineFake dataset. These results demonstrate that IGPC consistently outperforms decoupled-based strong baseline methods in both overall classification quality and ranking performance across datasets of varying domain granularity and difficulty.

[0114] The domain-specific analysis further demonstrates that these improvements are not limited to a few categories but have broad applicability. On the Weibo dataset, IGPC achieved the best F1 score in 6 out of 9 categories, with significant improvements particularly in the science, military, education, and finance domains. On the Weibo-21 dataset, IGPC again led in 6 out of 9 categories, tied for first place in the politics category, and also achieved significant improvements in the science and education domains. On the FineFake dataset, the improvements across domains are not evenly distributed, and some categories are still better suited to DAMMFND or other baseline models; however, IGPC achieved the best overall results, particularly in the finance domain, achieving the highest category-level F1 score among all comparison methods. Furthermore, in most cases where IGPC was not ranked first, it still achieved a competitive second-best result. In summary, these observations indicate that the proposed design improves the overall robustness of the model, rather than just for a subset of specific domains.

[0115] IGPC also demonstrates significant advantages over previous baseline methods. Compared to MMoE, IGPC improves the overall F1 score on the Weibo dataset from 0.874 to 0.954, on the Weibo-21 dataset from 0.894 to 0.963, and on the FineFake dataset from 0.778 to 0.802, representing relative improvements of 9.15%, 7.72%, and 3.08%, respectively. This improvement indicates that the combined approach of multimodal modeling, imbalance-aware weighting, and prototype-based geometry learning is more suitable for multi-domain fake news detection than previous hybrid expert baseline methods, and still achieves improvements on more challenging fine-grained benchmark datasets such as FineFake.

[0116] In addition to comparisons with multi-domain baselines, the experiments in this embodiment further compare IGPC with recent representative multimodal single-domain fake news detection methods. The experimental results are in... Figure 3 As shown in the table.

[0117] Figure 4 The table illustrates an incremental ablation study that followed a construction path from a standard news detection model to the full IGPC, progressively introducing Module I (balancing weights), Module II (contrastive learning), and Module III (bridging prototype). Overall, the full model achieved state-of-the-art results on both datasets. Compared to DAMMFND, the full model improved accuracy from 0.944 to 0.954 on the Weibo dataset and from 0.947 to 0.963 on the Weibo-21 dataset, representing relative improvements of 1.06% and 1.69%, respectively.

[0118] The improvement across categories is also consistent. On the Weibo dataset, the F1 score for real news improved from 0.945 to 0.951, while the F1 score for fake news improved from 0.943 to 0.956. On the Weibo-21 dataset, the F1 score for both categories improved to 0.963. This pattern indicates that the proposed design improves the detection capability of fake news without weakening the recognition of real news, and achieves a more balanced decision boundary.

[0119] Analysis of the single-module variants further reveals the role of each component. On the Weibo dataset, Module II shows the most significant independent improvement, indicating that explicit geometric learning is particularly effective on this dataset. Module I primarily improves the F1 score for fake news, suggesting that imbalanced perceptual weighting is particularly effective for more difficult-to-identify deceptive samples. Module III also provides improvements when used alone, but its effect is more pronounced when combined with other modules. On the Weibo-21 dataset, the situation is slightly different: Module III becomes the most effective single-module component, while Modules I and II still provide steady improvements. This difference suggests that the relative contribution of each module depends on the data distribution, but all three modules are effective on different datasets.

[0120] The multi-module setup exhibits clear complementarity. On the Weibo dataset, the combination of Module I and Module II outperforms all single-module variants. On the Weibo-21 dataset, the combination of Module I and Module III achieves an accuracy of 0.961 with balanced F1 scores across classes. After adding the remaining modules, the complete model achieves the best overall performance on both datasets. These results demonstrate that the three modules are complementary, and their integration yields the most robust and balanced performance under the setup of this invention. Furthermore, the model including only Module I also outperforms the baseline model in accuracy and most F1 scores, further indicating that balanced weights provide a significant benefit to the model.

[0121] To further investigate the quality of the learned feature representations, this invention uses T-SNE (t The Distributed Stochastic Neighbor Embedding (DSO) algorithm provides a visual test of the method provided in this invention from both global and neighborhood perspectives. Figures 5 to 8 As shown. Figure 5 and Figure 6 The overall visualization shown provides a global view of the feature space across all domains, while Figure 7 and Figure 8 The domain visualizations shown further reveal the separability of real and fake news within each domain.

[0122] Specifically, such as Figure 5 and Figure 6 As shown, the learned feature representations clearly demonstrate a global separation between real and fake news samples on both Weibo and Weibo-21. The two classes of samples are mostly distributed in different regions, with only limited overlap near the decision boundary. Furthermore, visually, there are fewer obvious outliers, indicating that the proposed model has learned a more compact intra-class structure and a more easily separable inter-class distribution. This demonstrates that the model can capture discriminative multimodal features while maintaining a consistent global representation across domains.

[0123] Figure 7 and Figure 8 A more detailed analysis was conducted across multiple domains. In most domains, real news samples and fake news samples could still be distinguished into different clusters, suggesting that the model appears to preserve domain-specific discriminative structures rather than solely relying on global category patterns. Even in relatively challenging domains with more complex distributions, the learned features still exhibited reduced inter-category mixing and clearer separation trends. These observations are consistent with the quantitative results and further support the effectiveness of the methods described in this invention.

[0124] While visualizations show that IGPC learns clearer and more separable feature structures, it remains to be tested whether these advantages can generalize to unseen target domains. To this end, this invention further evaluates IGPC on the Weibo and Weibo-21 datasets using a standard hold-out cross-domain testing scheme. Specifically, in each test, a domain is selected as the target domain and completely excluded from the training process. Then, the model is retrained using only the remaining domains as source domains and tested directly on the held-out target domain. By sequentially treating all domains as target domains, this invention obtains a comprehensive evaluation of all domains. This evaluation scheme avoids using any target domain samples during training, thus providing a more rigorous assessment of cross-domain transferability. Furthermore, evaluating all domains, rather than just a small subset, reduces bias introduced by domain-specific partitioning, making the comparison results more reliable.

[0125] Figure 9The table shows the cross-domain test results with partial modules retained. Overall, across different target domains on both datasets, the full model consistently outperforms the variant model without Module III, demonstrating that Module III provides a stable improvement in cross-domain rumor detection. On the Weibo dataset, the absolute gain ranges from 0.043 to 0.166, with significant improvements particularly in the political, entertainment, and international domains. On the Weibo-21 dataset, the gain ranges from 0.029 to 0.089, with all domains without Module III still benefiting from it. These results support the effectiveness of Module III in improving the domain migration performance of models without Module III in domain migration scenarios.

[0126] In summary, the embodiments of this invention provide a simple and consistent method that connects imbalance-aware weighting, prototype-based representation learning, and cross-domain transfer. Experiments on the Weibo, Weibo-21, and FineFake datasets demonstrate that the method provided by these embodiments improves detection performance, learns clearer feature structures, and remains effective while preserving the target domain. These results indicate that explicitly modeling imbalance and prototype structures is beneficial for achieving robust multimodal fake news detection.

[0127] It should be noted that the method of this embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this embodiment, and the multiple devices will interact with each other to complete the above method.

[0128] It should be noted that some embodiments of the present invention have been described above. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0129] Based on the same inventive concept, corresponding to any of the methods in the above embodiments, the present invention also provides a multimodal fake news detection device, such as... Figure 10 As shown, it includes: The acquisition module is used to acquire the initial news detection model and training set. The training set includes news samples from multiple domains, including real news samples and fake news samples. Each news sample includes text modality and image modality.

[0130] The weight calculation module is used to calculate the cross-domain imbalance weight of each domain based on the number of news samples in each domain and the number of news samples in all domains; to calculate the intra-domain imbalance weight of each domain based on the number of real news samples and fake news samples in each domain; and to calculate the global imbalance weight based on the number of all real news samples and the number of all fake news samples in the training set.

[0131] The training module is used to train an initial news detection model based on the training set and a loss function that embeds cross-domain imbalance weights, intra-domain imbalance weights, and global imbalance weights, thus obtaining a trained news detection model.

[0132] The detection module is used to input the news to be detected, including text modalities and image modalities, into the news detection model, and obtain the detection result of the news to be detected as real news or fake news.

[0133] The apparatus provided in this invention decomposes the data imbalance problem in multi-domain, multimodal fake news detection into three levels: cross-domain sample imbalance, intra-domain label imbalance, and global label imbalance. It transforms dataset statistics into stable training weights, effectively guiding the model to focus on samples with low data volume and low sample proportion. This systematically mitigates the negative impact of multi-level data imbalance on model training, ensuring that the model maintains good learning performance and detection capabilities even in scenarios with uneven data distribution. Furthermore, this weighting mechanism relies solely on label statistics, performs only one calculation, and remains unchanged throughout the training process. This avoids the drawback of traditional reweighting strategies that rely on model predictions and easily amplify early model biases, thus improving the stability and reliability of the training process.

[0134] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this invention, the functions of each module can be implemented in one or more software and / or hardware components.

[0135] The apparatus described above is used to implement the corresponding multimodal fake news detection method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0136] Figure 11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown.

[0137] like Figure 11 As shown, the electronic device may include a processor and a memory storing computer program instructions.

[0138] Specifically, the processor may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0139] The memory may include a large-capacity storage device for information or instructions. For example, and not limitingly, the memory may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory may include removable or non-removable (or fixed) media. Where appropriate, the memory may be internal or external to the integrated gateway device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable ROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0140] The processor reads and executes computer program instructions stored in memory to perform the steps of the multimodal fake news detection method provided in this embodiment of the invention.

[0141] In one example, the electronic device may also include a transceiver and a bus. For example, Figure 11 As shown, the processor, memory, and transceiver are connected via a bus and communicate with each other.

[0142] A bus may be hardware, software, or both. For example, and not limitingly, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, a bus may include one or more buses. Although specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0143] The following are embodiments of a computer-readable storage medium provided in this invention. This computer-readable storage medium belongs to the same inventive concept as the multimodal fake news detection method in the above embodiments. For details not described in detail in the embodiments of the computer-readable storage medium, please refer to the embodiments of the multimodal fake news detection method described above.

[0144] This embodiment provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a multimodal fake news detection method.

[0145] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the above-described method operations, but can also perform related operations in the multimodal fake news detection method provided in any embodiment of the present invention.

[0146] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer cloud platform (which can be a personal computer, server, or network cloud platform, etc.) to execute the multimodal fake news detection method provided in the various embodiments of the present invention.

[0147] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0148] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments described above, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal fake news detection method, characterized in that, include: Obtain an initial news detection model and a training set; wherein, the training set includes news samples from multiple domains, the news samples include real news samples and fake news samples, and each news sample includes a text modality and an image modality; Calculate the cross-domain imbalance weight for each domain based on the number of news samples in each domain and the total number of news samples across all domains; calculate the intra-domain imbalance weight for each domain based on the number of real news samples and fake news samples in each domain; calculate the global imbalance weight based on the total number of real news samples and the total number of fake news samples in the training set. The initial news detection model is trained based on the training set and a loss function that embeds the cross-domain imbalance weights, the intra-domain imbalance weights, and the global imbalance weights, to obtain the trained news detection model. The news to be detected, including text and image modalities, is input into the news detection model to obtain the detection result of whether the news to be detected is real or fake news.

2. The multimodal fake news detection method according to claim 1, characterized in that, The formula for calculating the cross-domain imbalance weight includes: in, For the first Cross-domain unbalanced weights in each domain For the first Scarcity score in each field , For the first The number of news samples in each field , This refers to the number of news samples from the field with the largest number of news samples. , For global imbalance score, The total sample is discretely distributed. , , , The total number across all fields. The number of news samples in the first domain. For the first The number of news samples in each field For the first The number of news samples in each field Discrete distribution of the total sample The average entropy, Discrete distribution of the total sample The average Gini coefficient, For cross-domain unbalanced Gini weights.

3. The multimodal fake news detection method according to claim 1, characterized in that, The formula for calculating the unbalanced weights within the domain includes: in, For the first Intradomain imbalance weights for each domain For the first Intra-domain imbalance scores for each domain For the first Discrete distribution of samples in each domain , For the discrete distribution of the sample The average entropy, For the discrete distribution of the sample The average Gini coefficient, , , For the first The number of news samples in each field For the first The number of real news samples in each field No. The number of fake news samples in each field This represents the unbalanced Gini weights within the domain.

4. The multimodal fake news detection method according to claim 1, characterized in that, The formula for calculating the global imbalance weight includes: in, The global unbalanced weights, For true and false labels, when , For the globally imbalanced weights of the real news samples, when , For the globally imbalanced weights of fake news samples; For the number of all real news samples or the number of all fake news samples in the training set, when , Let be the number of all real news samples in the training set, when , The number of all fake news samples in the training set. The number of all news samples in the training set. This represents the globally unbalanced Gini weight.

5. The multimodal fake news detection method according to claim 1, characterized in that, The loss function includes: in, For weighted prototype clustering loss, This represents the number of news samples in the current training batch. For the first The news sample belongs to the first The domain-specific unbalanced weights of each domain For the first The projection vector of a news sample in the prototype space. For the additional introduction of the first The news sample belongs to the first The corresponding true / false labels for each field are: prototype vector, For the first The true and false labels of a news sample Indicates the first The news sample is a genuine news sample. Indicates the first One news sample is a fake news sample. Calculate the L2 norm; The loss function also includes: in, To monitor and compare losses, For the first Instance-supervised contrastive loss for a single news sample. For the first The news sample belongs to the first The domain-specific unbalanced weights of each domain For the first A positive sample set of news samples, The number of positive samples. , , Indicates except the first News samples other than the news sample, For the first The field to which each news sample belongs, For the first The field to which each news sample belongs; No. The true and false labels of a news sample Indicates the first The news sample is a genuine news sample. Indicates the first One news sample is a fake news sample; Denotes the first positive sample in the positive sample set. A news sample; For the first The projection vector of a news sample in the prototype space. For the first The projection vector of a news sample in the prototype space; For cosine similarity calculation, This is the temperature coefficient.

6. The multimodal fake news detection method according to claim 1, characterized in that, The loss function includes: in, For the total loss function, For weighted cross-entropy calculation, The first output of the initial news detection model The probability that a news sample is fake news. For the first The true and false labels of a news sample Indicates the first The news sample is a genuine news sample. Indicates the first One news sample is a fake news sample; For the globally unbalanced weights, when , For the globally imbalanced weights of the real news samples, when , For the globally imbalanced weights of fake news samples; The first output of the initial news detection model Modalities of a news sample The probability that it is fake news. For text modality, For image modality, It is a mixed mode. For the first The news sample belongs to the first Cross-domain unbalanced weights in each domain This represents the sample-level backbone network loss.

7. The multimodal fake news detection method according to claim 1, characterized in that, The process of inputting the news to be detected, including text and image modalities, into the news detection model and obtaining the detection result of whether the news to be detected is real or fake news, as output by the news detection model, includes: The news to be detected is input into the news detection model to obtain the initial predicted probability output by the news detection model; Obtain the domain of the news to be detected, obtain the initial prototype vector of the news to be detected based on the domain, and mix the initial prototype vector with the global prototype vector to form a bridging prototype vector; wherein, the global prototype vector is calculated based on the cross-domain unbalanced weight and the prototype vectors of all domains; The prototype prediction probability is calculated based on the projection vector of the news to be detected in the prototype space and the bridging prototype vector. The detection result of whether the news to be detected is real or fake news is determined by combining the initial prediction probability and the prototype prediction probability.

8. A multimodal fake news detection device, characterized in that, include: The acquisition module is used to acquire the initial news detection model and training set; wherein, the training set includes news samples from multiple domains, the news samples include real news samples and fake news samples, and each news sample includes a text modality and an image modality; The weight calculation module is used to calculate the cross-domain imbalance weight of each domain based on the number of news samples in each domain and the number of news samples in all domains; to calculate the intra-domain imbalance weight of each domain based on the number of real news samples and fake news samples in each domain; and to calculate the global imbalance weight based on the number of all real news samples and the number of all fake news samples in the training set. The training module is used to train the initial news detection model based on the training set and a loss function that embeds the cross-domain imbalance weights, the intra-domain imbalance weights and the global imbalance weights, so as to obtain the trained news detection model. The detection module is used to input the news to be detected, including text modalities and image modalities, into the news detection model, and obtain the detection result of the news to be detected, which is either real news or fake news, output by the news detection model.

9. An electronic device, characterized in that, include: processor; A memory for storing executable instructions; wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method described in any one of claims 1 to 7.

Images