Fake news identification method, device and electronic device based on uncertainty perception

Abstract

The present invention discloses a method for identifying fake news, specifically a method, device, and electronic device for identifying fake news based on uncertainty perception. The method comprises: inputting an image and text into a unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text; modeling the image representation and the text representation as Gaussian distributions to obtain the mean and variance of the image representation and the mean and variance of the text representation; obtaining a shared probability representation and an exclusive probability representation of the image and text, respectively, and then inputting these into an uncertainty-aware hybrid module for fusion, using uncertainty to guide the fusion process to obtain fused multimodal features; and classifying real news and fake news based on the fused multimodal features to obtain classification results. This method can better fuse multimodal information, thereby obtaining more accurate fake news identification results.

Description

Technical Field

[0001] The present invention relates to a false news identification method, and in particular to a false news identification method, device and electronic equipment based on uncertainty perception. Background Art

[0002] The rise of social media and instant messaging technologies has transformed the landscape of information dissemination, facilitating the rapid and widespread sharing of content. However, this advancement has also created conditions for the rapid spread of fake news—information deliberately fabricated to mislead, deceive, or manipulate public perception. Due to its captivating headlines and content, news can quickly garner widespread engagement and dissemination. The resulting impact on social stability is profound, necessitating an urgent need to investigate and implement effective fake news detection technologies.

[0003] Multimodal fake news detection technology integrates and processes information from various media forms to identify and exploit the interconnections and interactions between different media forms. However, existing methods mainly rely on deterministic representations and ignore the ambiguity and uncertainty inherent in multimodal data, which limits their ability to capture subtle relationships. This uncertainty exists both within a single modality (such as words or images may have multiple meanings in different contexts) and across modalities (for example, ambiguity may increase when text and images do not match). For example, an image may not completely match its corresponding text. This situation may be unintentional or may be part of deceptive content, increasing the complexity of interpretation. Handling unimodal and cross-modal uncertainty is crucial for models to flexibly and reliably analyze complex multimodal data. Therefore, commonly used methods in the existing technology cannot capture the uncertainty and ambiguity of multimodal data, resulting in low accuracy of fake news identification results. Summary of the Invention

[0004] The technical problem to be solved by the present invention is that the commonly used methods in the prior art cannot capture the uncertainty and ambiguity of multimodal data, resulting in low accuracy of fake news identification results. In order to solve the above problem, the present invention provides a fake news identification method, device and electronic device based on uncertainty perception.

[0005] The present invention includes:

[0006] In a first aspect, an embodiment of the present invention provides a method for identifying fake news based on uncertainty perception, comprising:

[0007] Inputting the image and text into a unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text;

[0008] Modeling the image representation and the text representation as Gaussian distributions, and obtaining a mean and variance of the image representation and a mean and variance of the text representation;

[0009] Inputting the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, inputting the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, inputting the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and inputting the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation;

[0010] The image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation are input into an uncertainty-aware hybrid module for fusion, and the uncertainty is used to guide the fusion process to obtain a fused multimodal feature;

[0011] Based on the fused multimodal features, real news and fake news are classified to obtain a classification result.

[0012] Optionally, the unimodal extraction module includes a Swin Transformer and BERT, and the step of inputting the image and text into the unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text includes:

[0013] Inputting the image into the Swin Transformer to extract image features and obtain an image representation corresponding to the image;

[0014] The text is input into the BERT to perform text feature extraction to obtain a text representation corresponding to the text.

[0015] Optionally, modeling the image representation and the text representation as Gaussian distributions to obtain the mean and variance of the image representation and the mean and variance of the text representation includes:

[0016] The target representation is input into the uncertain representation module, and the mean and variance of the Gaussian distribution of the target representation are obtained by using a multi-head attention mechanism and a feedforward neural network, combined with layer normalization and residual connection. The target representation is the image representation or the text representation.

[0017] Optionally, the step of inputting the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation into an uncertainty-aware hybrid module for fusing, and using uncertainty to guide the fusion process to obtain a fused multimodal feature includes:

[0018] Reparameterizing the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation to obtain a shared reparameterized vector and a specific reparameterized vector, wherein the specific reparameterized vector includes an image-specific reparameterized vector and a text-specific reparameterized vector;

[0019] Fusion is performed based on the shared reparameterized vector and the exclusive reparameterized vector, and uncertainty is used to guide the fusion process to obtain the fused multimodal feature.

[0020] Optionally, the fusing based on the shared reparameterized vector and the dedicated reparameterized vector, using uncertainty to guide the fusion process to obtain the fused multimodal feature, includes:

[0021] Based on the dedicated reparameterization vector and the shared reparameterization vector Determine the query Q i , key K i Sum V i :

[0022]

[0023] Calculate the attention weight for each attention head and get the fused representation x out :

[0024]

[0025] Among them, Q i , K i and V i The dimensions are all d k , represents the dimension of each attention head, H is the number of attention heads, and W is a learned projection matrix;

[0026] The fused multimodal feature is determined based on the fused representation. When the exclusive reparameterization vector includes the image-exclusive reparameterization vector, the fused representation includes the image fusion representation. When the exclusive reparameterization vector includes the text-exclusive reparameterization vector, the fused representation includes the text fusion representation.

[0027] Optionally, determining the fused multimodal feature based on the fused representation includes:

[0028] Scaling the fused representation based on a weighting factor to obtain a hybrid representation, wherein the weighting factor is determined based on KL divergence;

[0029] The mixed representation is optimized through residual connection, layer normalization and feed-forward network to obtain the fused multimodal features.

[0030] Optionally, before scaling the fused representation based on the weighting factor to obtain the hybrid representation, the method further includes:

[0031] determining an initial weight factor based on the KL divergence of the shared reparameterization vector and the dedicated reparameterization vector;

[0032] The initial weight factor is subjected to linear transformation and ReLU activation to obtain the weight factor.

[0033] In a second aspect, an embodiment of the present invention provides a false news identification device based on uncertainty perception, comprising:

[0034] A single-modal extraction module is used to extract features from images and texts to obtain an image representation corresponding to the image and a text representation corresponding to the text;

[0035] an uncertainty characterization module, configured to model the image representation and the text representation as Gaussian distributions, and obtain a mean and variance of the image representation and a mean and variance of the text representation;

[0036] a decoupled representation learning module, configured to input the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, input the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, input the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and input the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation;

[0037] an uncertainty-aware hybrid module for fusing the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation, using uncertainty to guide the fusion process to obtain a fused multimodal feature;

[0038] The classification module is used to classify real news and fake news based on the fused multimodal features to obtain a classification result.

[0039] In a third aspect, an embodiment of the present invention provides an electronic device comprising: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps of the false news identification method based on uncertainty perception as described in the first aspect.

[0040] In a fourth aspect, an embodiment of the present invention provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the false news identification method based on uncertainty perception as described in the first aspect.

[0041] The beneficial effects of the present invention are that, in the embodiments of the present application, on the one hand, Gaussian distribution is used to model the representation of information of text modality and information of image modality instead of using deterministic points, which can capture the uncertainty and ambiguity in multimodal data. On the other hand, a decoupled representation learning framework is introduced to obtain image shared probability representation, image exclusive probability representation, text shared probability representation and text exclusive probability representation, respectively, to decouple the shared and exclusive information between modalities, thereby learning a more robust and discriminative representation. On the third hand, the uncertainty of multimodal representation is used to guide the fusion process, thereby achieving better cross-modal interaction based on the uncertainty of modal representation. Through the method provided in the embodiments of the present application, multimodal information can be better fused, thereby obtaining a more accurate classification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Attachment Figure 1 A flowchart of a false news identification method based on uncertainty perception provided by an embodiment of the present invention;

[0043] Attachment Figure 2 A schematic diagram of a framework of a false news identification method based on uncertainty perception provided by an embodiment of the present invention;

[0044] Attachment Figure 3 For attachment Figure 2 Schematic diagram of the unimodal feature extraction module;

[0045] Attachment Figure 4 For attachment Figure 2 Schematic diagram of the decoupled representation learning module;

[0046] Attachment Figure 5 For attachment Figure 2 Schematic diagram of the uncertainty-aware hybrid module;

[0047] Attachment Figure 6 For attachment Figure 2 Schematic diagram of the probability representation module;

[0048] Attachment Figure 7 For attachment Figure 2 Schematic diagram of the uncertainty-aware hybrid module;

[0049] Attachment Figure 8 A schematic diagram of an uncertainty-aware hybrid module provided by an embodiment of the present invention;

[0050] Attachment Figure 9A schematic diagram of a false news identification device based on uncertainty perception provided by an embodiment of the present invention;

[0051] Attachment Figure 10 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In the embodiments of the present application, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In the embodiments of the present application, the term "multiple" refers to two or more, and other quantifiers are similar. The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0055] The embodiments of the present application provide a method and apparatus for identifying false news based on uncertainty perception, and an electronic device, aiming to improve the accuracy of false news identification.

[0056] See Figure 1 , Figure 1 This is a flow chart of a method for identifying fake news based on uncertainty perception provided by an embodiment of the present invention. The method specifically includes the following steps:

[0057] Step 101: Input the image and text into a unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text.

[0058] Step 102: Model the image representation and the text representation as Gaussian distributions to obtain the mean and variance of the image representation and the mean and variance of the text representation.

[0059] Step 103: input the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, input the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, input the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and input the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation.

[0060] Step 104: Input the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation into an uncertainty-aware hybrid module for fusion, using uncertainty to guide the fusion process to obtain a fused multimodal feature;

[0061] Step 105: Classify real news and fake news based on the fused multimodal features to obtain a classification result.

[0062] In an embodiment of the present application, on the first hand, Gaussian distribution is used to model the representation of multimodal information (including text information and image information) instead of using deterministic points, which can capture the uncertainty and ambiguity in multimodal data. On the second hand, a decoupled representation learning framework is introduced to obtain image shared probability representation, image exclusive probability representation, text shared probability representation and text exclusive probability representation, respectively, to decouple the shared and exclusive information between modalities, thereby learning a more robust and discriminative representation. On the third hand, the uncertainty of multimodal representation is used to guide the fusion process, so as to achieve better cross-modal interaction based on the uncertainty of modal representation. Through the method provided in the embodiment of the present application, multimodal information can be better fused, thereby obtaining a more accurate classification result.

[0063] See Figure 2 and Figure 3 Optionally, in some embodiments, the unimodal extraction module includes SwinTransformer and BERT, and step 101 includes:

[0064] Inputting the image into the Swin Transformer to extract image features and obtain an image representation corresponding to the image;

[0065] The text is input into the BERT to perform text feature extraction to obtain a text representation corresponding to the text.

[0066] As a specific example, for the input image V and text T, a pre-trained transformer-based model is used to extract the image representation X V and the text represents X T .

[0067] Specifically, for image modality data V, SwinTransformer pre-trained on a large-scale image dataset is used to extract image representations and convert them into sequence embeddings as the final image representation X V :

[0068]

[0069] Where n is the number of image blocks, is the representation of the i-th image block, d v is the hidden dimension of image representation, and SwinT is Swin Transformer.

[0070] For text modality data T, BERT pre-trained on large-scale text corpus is used to extract text representation X T By inputting the text T into the BERT model, we can get the text representation X T :

[0071]

[0072] Where m is the number of tokens in the text, is the representation of the i-th token, d t is the hidden dimension of the text representation.

[0073] Optionally, in some embodiments, step 102 includes:

[0074] The target representation is input into the uncertain representation module, and the mean and variance of the Gaussian distribution of the target representation are obtained by using a multi-head attention mechanism and a feedforward neural network, combined with layer normalization and residual connection. The target representation is the image representation or the text representation.

[0075] See Figure 2 and Figure 4 In order to represent the uncertainty and ambiguity of multimodal information, this embodiment provides an uncertainty representation module that models multimodal information as a Gaussian distribution. The Gaussian distribution has elegant mathematical properties, making modeling and analysis more convenient. In addition, the Gaussian distribution is widely used in various fields because it is common in natural phenomena and simplifies uncertainty modeling. In this embodiment, a multi-head attention mechanism and a feedforward neural network are used, combined with layer normalization and residual connections to obtain the mean and variance of the distribution. The specific process is as follows:

[0076] To calculate the mean of the Gaussian distribution of the image representation, first we represent the image X V Applying a feedforward neural network:

[0077]

[0078] Among them, FeedForward is a feedforward neural network, which is designed to promote feature-level interactions and enrich representations by capturing complex feature dependencies.

[0079] The result is then passed through a multi-head attention layer and added to the original input through a residual connection, and the mean μ is obtained after layer normalization:

[0080]

[0081] Among them, MultiHead is the multi-head attention layer and LN is the layer normalization.

[0082] To calculate the variance of the Gaussian distribution of the image representation, first transform the image representation X V The features of the exclusive variance are calculated through another feedforward neural network:

[0083]

[0084] The multi-head operation then captures the dependencies at the token level, thereby focusing on multiple aspects of the input sequence at the same time. These features are then passed through the multi-head attention layer, added to the original features through a residual connection and normalized to obtain the logarithmic variance logσ 2 :

[0085]

[0086] in, and are the mean and variance of the Gaussian distribution represented by the image.

[0087] In order to calculate the mean of the Gaussian distribution of the text representation, we first T Applying a feedforward neural network:

[0088]

[0089] The result is then passed through a multi-head attention layer and added to the original input through a residual connection, and the mean μ is obtained after layer normalization:

[0090]

[0091] To calculate the variance of the Gaussian distribution of the image representation, first transform the text representation X TThe features of the exclusive variance are calculated through another feedforward neural network:

[0092]

[0093] The multi-head operation then captures the dependencies at the token level, thereby focusing on multiple aspects of the input sequence at the same time. These features are then passed through the multi-head attention layer, added to the original features through a residual connection and normalized to obtain the logarithmic variance logσ 2 :

[0094]

[0095] in, and are the mean and variance of the Gaussian distribution of the image representation. The specific method for obtaining the mean and variance of the Gaussian distribution of the text representation can be found in the above image representation section and will not be repeated here.

[0096] In step 103, the shared information and exclusive information between modalities are decoupled through the decoupled representation learning framework to obtain image shared probability representation, image exclusive probability representation, text shared probability representation and text exclusive probability representation, so that the model can learn more robust and discriminative representations.

[0097] Specifically, the shared encoder is represented as E S , denote the image-specific encoder as E V , denote the text-specific encoder as E T . Shared Encoder E S It is used to learn the representation of modality sharing (or modality sharing features), which can capture the shared information between modalities; image-specific encoder E V Used to learn image modality-specific representations (or image modality-private features), text-specific encoder E T It is used to learn text modality-specific representations (or text modality-specific features), E V and E T Ability to capture modality-specific information.

[0098] As a specific embodiment, the shared encoder, the image-specific encoder and the text-specific encoder are all implemented as probability representation modules, specifically as follows Figure 4 and Figure 6 As shown. The shared encoder is a modality-shared probability representation module, the image-specific encoder is an image-private probability representation module, and the text-specific encoder is a text-private probability representation module. Image shared probability representation Text Sharing Probability Representation Image-specific probabilistic representation and text-specific probabilistic representation The mean and variance of can be obtained by the following formula:

[0099]

[0100] Where m∈{s,p} represents the shared encoder E S and dedicated encoder E V and E T , n∈{v,t} represents the image modality and text modality respectively.

[0101] Therefore, the distribution of the shared probability representation of image modalities can be obtained Distribution of shared probability representations across textual modalities Distribution of Image Modality-Specific Probabilistic Representations Distribution of text modality-specific probability representations as follows:

[0102]

[0103] Optionally, in some embodiments, step 104 includes:

[0104] Reparameterizing the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation to obtain a shared reparameterized vector and a specific reparameterized vector, wherein the specific reparameterized vector includes an image-specific reparameterized vector and a text-specific reparameterized vector;

[0105] Fusion is performed based on the shared reparameterized vector and the exclusive reparameterized vector, and uncertainty is used to guide the fusion process to obtain the fused multimodal feature.

[0106] See Figure 7 In this embodiment, the uncertainty-aware hybrid module first generates a modality-specific shared probability representation x p and the exclusive probability representation x s , where x p For the previous article or x s From the previous After concatenation, these features are modeled as Gaussian distributions to capture the uncertainty of each modality, represented by mean μ and covariance Σ.

[0107] In order to achieve gradient-based optimization, a reparameterization technique is applied in this embodiment to generate a dedicated reparameterization vector and the shared reparameterization vector The details are as follows:

[0108]

[0109] in, is a random variable sampled from a standard normal distribution, μ p and Σ p are the mean and covariance of the mode-specific probabilities, μ s and Σ s are the mean and covariance of the modal shared probability representation.

[0110] After reparameterization, a crisscross attention mechanism is applied where guide Specifically, and is converted into query, key and value, and then the attention weight is calculated for each attention head i. Optionally, the fusion is performed based on the shared reparameterized vector and the dedicated reparameterized vector, and uncertainty is used to guide the fusion process to obtain a fused multimodal feature, including:

[0111] Based on the dedicated reparameterization vector and the shared reparameterization vector Determine the query Q i , key K i Sum V i :

[0112]

[0113] in, Represents a linear layer that helps the model generate attention weights through feature transformation and projection, thereby focusing on the most relevant information in the input data.

[0114] Then, the attention weights are calculated for each attention head and the fused representation x is obtained out , the fusion is expressed as:

[0115]

[0116] Among them, Q i , K i and V i The dimensions are all d k , represents the dimension of each attention head, H is the number of attention heads, and W is a learned projection matrix;

[0117] A fused multimodal feature is determined based on the fused representation. When the exclusive reparameterization vector includes the image-exclusive reparameterization vector, the fused representation includes the image fusion representation. When the exclusive reparameterization vector includes the text-exclusive reparameterization vector, the fused representation includes the text fusion representation.

[0118] It should be understood that, in this embodiment, calculations need to be performed separately for the image modality and the text modality. When performing calculations for the image modality, x s Depend on When calculating for text mode, x s Depend on By calculating the image modality and text modality respectively in the above way, the image fusion representation corresponding to the image modality can be obtained. And the text fusion representation corresponding to the text modality

[0119] See Figure 8 In order to introduce uncertainty into the fusion process, the Kullback-Leibler (KL) divergence between the Gaussian distributions of private and shared features is used as a weighting factor. Although the reparameterization technique allows for differentiable sampling, it cannot fully capture the relative uncertainty between modalities. KL divergence solves this problem by directly measuring the distribution difference between private and shared features, thereby providing a detailed weighting for fusion. Optionally, in some embodiments, determining the fused multimodal features based on the fused representation includes:

[0120] Scaling the fused representation based on a weighting factor to obtain a hybrid representation, wherein the weighting factor is determined based on KL divergence;

[0121] The mixed representation is optimized through residual connection, layer normalization and feed-forward network to obtain the fused multimodal features.

[0122] Optionally, in some embodiments, before scaling the fused representation based on the weighting factor to obtain the hybrid representation, the method further includes:

[0123] determining an initial weight factor based on the KL divergence of the shared reparameterization vector and the dedicated reparameterization vector;

[0124] The initial weight factor is subjected to linear transformation and ReLU activation to obtain the weight factor.

[0125] Among them, D KL (x p ,x s ) provides an initial weight based on uncertainty, which is further transformed linearly and activated by ReLU to produce an optimized weighting factor The weighting factor Used to scale the fused representation to obtain a mixed representation:

[0126]

[0127] After obtaining the mixed representation, it is optimized through a series of residual connections, layer normalization (LN) and feed-forward network (FFN) to obtain fused multimodal features. The specific process can be expressed as:

[0128]

[0129] In this process, each residual connection combines the original input with the transformed representation, preserving key information while integrating new features. Layer normalization is applied after each residual connection to stabilize training by normalizing activations and reducing internal covariate shift. Feedforward networks are used to enrich the representation through nonlinear transformations, enabling the model to capture more complex interactions. In this embodiment, this sequence of residual connections, LNs, and FFNs is used to normalize, enhance, and stabilize the fused representation, generating a robust and informative output with cross-modal uncertainty.

[0130] Image fusion representation corresponding to image modality By performing the above processing, the image fusion features corresponding to the image modality can be obtained. And the text fusion representation corresponding to the text modality That is, in x out for In the case of for In x out for In the case of for In the specific implementation, by and The multimodal fusion features can be obtained by fusion.

[0131] After obtaining the fused multimodal features, in step 105, real news and fake news are classified based on the fused multimodal features to obtain classification results. Specifically, each modal feature [cls] token is extracted and a fully connected network is used to classify real and fake news.

[0132] like Figure 2 As shown, an embodiment of the present invention provides a decoupled representation learning model based on uncertainty perception, which specifically includes a unimodal feature extraction module, an uncertainty characterization module, a decoupled representation learning module, an uncertainty perception hybrid module and a classification module.

[0133] As a specific example, during the training of this model, the total loss of the model is as follows:

[0134]

[0135] Among them, λmi ,λ tri ,λ ctr ,λ rec Represent the coefficients of each loss respectively. is the classification loss, is the information redundancy constraint loss, is the inter-class triplet loss, is the intra-class center loss, is the reconstruction loss.

[0136] The following describes each loss separately.

[0137] Classification loss is the cross entropy loss, which is calculated as:

[0138]

[0139] Among them, N represents the number of training samples, p i represents the probability that the i-th training sample is fake news.

[0140] To ensure that the shared and dedicated representations are effectively decoupled and redundancy is minimized, an independence constraint is introduced to reduce mutual information, thereby reducing redundancy between the two representations. Due to its ability to capture geometric differences between distributions, this example uses the Wasserstein distance between the Gaussian distributions of the shared and dedicated representations as an effective measure of dependency, thereby encouraging minimal overlap and redundancy.

[0141] Taking the text mode as an example, for two Gaussian distributions and The closed-form expression of its 2-Wasserstein distance is:

[0142]

[0143] The goal in this embodiment is to reduce the redundancy between shared and exclusive representations by maximizing the approximation of mutual information. Therefore, the information redundancy constraint loss is defined as for:

[0144]

[0145] By setting the information redundancy constraint loss, the independence between shared and exclusive representations can be encouraged by reducing redundancy, thereby improving the decoupling effect and overall performance of the model.

[0146] To ensure alignment of shared representations across modalities, this example introduces a hybrid loss that combines an inter-class triplet loss and an intra-class center loss. The inter-class triplet loss encourages multimodal representations from the same category to be closer together while separating representations from different categories. The intra-class center loss aims to reduce the intra-class variance of the shared representations, further ensuring consistency within each category.

[0147] Specifically, the inter-class triplet loss is defined as follows:

[0148]

[0149] in, and Shared representations for samples from image and text modalities, respectively; and Represents positive samples from the same category; and represents negative samples from different categories; α is the boundary that encourages separation of inter-class representations; D KL (·||·) represents two Gaussian distributions and The KL divergence between .

[0150] However, relying solely on the inter-class triplet loss only considers the separation between categories, which may not be enough to ensure strong alignment within each category. Therefore, this embodiment also introduces the intra-class center loss to further minimize the intra-class variance of the shared representation. The intra-class center loss is defined as:

[0151]

[0152] Among them, C is the number of categories, N is the number of samples in each category, τ j is the weight of the jth distribution, given by Given, μ j is the mean of the jth shared representation, and μ weighted is the weighted mean of the categories, calculated as

[0153] Finally, the consistency constraint combines the inter-class triplet loss and the intra-class center loss and is expressed as:

[0154]

[0155] In order to distinguish shared and exclusive representations and reduce information loss in the process of learning decoupled representations, information reconstruction constraints are introduced. Specifically, Figure 7 As shown, the image decoder and text decoder are used to reconstruct the input representation, which further ensures that the information is well preserved during the decoupled representation learning process.

[0156] The reconstruction loss can be expressed as:

[0157]

[0158] Among them, Decoder V and Decoder T They are image decoder and text decoder. The image decoder and text decoder are implemented as feed-forward neural networks to reconstruct the input image and text representations.

[0159] As a specific example, first, a Gaussian distribution is used to model the representation of multimodal information, rather than using deterministic points. This can capture the uncertainty and ambiguity in multimodal data. The parameters of the Gaussian distribution can be learned using the probabilistic representation module proposed in this example. Second, a decoupled representation learning framework is introduced to decouple shared and specific information between modalities, enabling the model to learn more robust and discriminative representations. Specifically, the shared representations of modalities are learned by a shared encoder. To ensure the consistency of shared representations across modalities, a triplet learning component is introduced to bring multimodal representations of the same type closer together while pushing representations of different types further apart. Modality-specific representations are learned by a specific encoder to capture the unique information of the modality. In this process, independence constraints are introduced to minimize the mutual information between shared and specific representations. In addition, the use of an image decoder and a text decoder to reconstruct the input representation further ensures that information is well preserved during the decoupled representation learning process. Finally, this example proposes an uncertainty-aware fusion module that leverages the uncertainty of multimodal representations to guide the fusion process, thereby achieving better cross-modal interaction based on the uncertainty of modal representations. Specifically, a weighting factor is learned for each modality based on its uncertainty. This factor can dynamically adjust the importance of each modality during the fusion process. This method can improve the effect of multimodal data fusion and thus improve the accuracy of fake news identification.

[0160] See Figure 9 The embodiment of the present invention further provides a false news identification device 900 based on uncertainty perception, comprising:

[0161] A single-modality extraction module 901 is used to extract features from images and texts to obtain an image representation corresponding to the image and a text representation corresponding to the text;

[0162] An uncertainty characterization module 902 is configured to model the image representation and the text representation as Gaussian distributions to obtain a mean and variance of the image representation and a mean and variance of the text representation;

[0163] a decoupled representation learning module 903, configured to input the mean and variance of the image representation into a shared encoder to obtain an image-shared probability representation, input the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, input the mean and variance of the text representation into the shared encoder to obtain a text-shared probability representation, and input the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation;

[0164] an uncertainty-aware hybrid module 904 for fusing the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation, using uncertainty to guide the fusion process to obtain a fused multimodal feature;

[0165] The classification module 905 is used to classify real news and fake news based on the fused multimodal features to obtain a classification result.

[0166] Optionally, the unimodal extraction module includes Swin Transformer and BERT, and the unimodal extraction module is specifically used to:

[0167] Inputting the image into the Swin Transformer to extract image features and obtain an image representation corresponding to the image;

[0168] The text is input into the BERT to perform text feature extraction to obtain a text representation corresponding to the text.

[0169] Optionally, the uncertainty characterization module 902 is specifically configured to:

[0170] The target representation is input into the uncertain representation module, and the mean and variance of the Gaussian distribution of the target representation are obtained by using a multi-head attention mechanism and a feedforward neural network, combined with layer normalization and residual connection. The target representation is the image representation or the text representation.

[0171] Optionally, the uncertainty-aware hybrid module 904 includes:

[0172] a reparameterization unit, configured to reparameterize the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation to obtain a shared reparameterization vector and a specific reparameterization vector, wherein the specific reparameterization vector includes an image-specific reparameterization vector and a text-specific reparameterization vector;

[0173] A fusion unit is used to perform fusion based on the shared reparameterization vector and the exclusive reparameterization vector, and use uncertainty to guide the fusion process to obtain the fused multimodal feature.

[0174] Optionally, the fusion unit includes:

[0175] Determine a subunit for re-parameterizing the vector based on the exclusive and the shared reparameterization vector Determine the query Q i , key K i Sum V i :

[0176]

[0177]

[0178] The computation unit is used to calculate the attention weight for each attention head and obtain the fused representation x out :

[0179]

[0180] Among them, Q i , K i and V i The dimensions are all d k , represents the dimension of each attention head, H is the number of attention heads, and W is a learned projection matrix;

[0181] A determination unit is used to determine the fused multimodal feature based on the fused representation, when the exclusive reparameterization vector includes the image-exclusive reparameterization vector, the fused representation includes the image fusion representation, and when the exclusive reparameterization vector includes the text-exclusive reparameterization vector, the fused representation includes the text fusion representation.

[0182] Optionally, the determining unit is specifically configured to:

[0183] Scaling the fused representation based on a weighting factor to obtain a hybrid representation, wherein the weighting factor is determined based on KL divergence;

[0184] The mixed representation is optimized through residual connection, layer normalization and feed-forward network to obtain the fused multimodal features.

[0185] Optionally, the determining unit is further configured to:

[0186] determining an initial weight factor based on the KL divergence of the shared reparameterization vector and the dedicated reparameterization vector;

[0187] The initial weight factor is subjected to linear transformation and ReLU activation to obtain the weight factor.

[0188] The false news identification device 900 based on uncertainty perception provided in the embodiment of the present application can execute the above-mentioned method embodiment. Its implementation principle and technical effects are similar, and will not be repeated here in this embodiment.

[0189] It should be noted that the division of units in the embodiments of the present application is schematic and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0191] like Figure 10 As shown, an embodiment of the present application provides an electronic device 1000, including: a memory 1002, a processor 1001, and a program stored on the memory 1002 and executable on the processor 1001; the processor 1001 is configured to read the program in the memory 1002 to implement the steps in the false news identification method based on uncertainty perception as described above.

[0192] An embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by a processor, the various processes of the above-mentioned embodiment of the false news identification method based on uncertainty perception are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as compact disk (CD), digital video disc (DVD), Blu-ray disc (BD), high-definition versatile disc (HVD), etc.), and semiconductor memory (such as read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read only memory (EEPROM), non-volatile memory (NAND FLASH), solid state drive (SSD)), etc.

[0193] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0194] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD-ROM), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0195] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A false news identification method based on uncertainty perception, characterized by: include: Inputting the image and text into a unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text; Modeling the image representation and the text representation as Gaussian distributions, and obtaining a mean and variance of the image representation and a mean and variance of the text representation; Inputting the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, inputting the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, inputting the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and inputting the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation; The image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation are input into an uncertainty-aware hybrid module for fusion, and the uncertainty is used to guide the fusion process to obtain a fused multimodal feature; Classifying true news and false news based on the fused multimodal features to obtain a classification result; The step of inputting the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation into an uncertainty-aware hybrid module for fusion, and using uncertainty to guide the fusion process to obtain a fused multimodal feature, includes: Reparameterizing the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation to obtain a shared reparameterized vector and a specific reparameterized vector, wherein the specific reparameterized vector includes an image-specific reparameterized vector and a text-specific reparameterized vector; performing fusion based on the shared reparameterized vector and the exclusive reparameterized vector, and utilizing uncertainty to guide the fusion process to obtain the fused multimodal feature; The step of fusing the shared reparameterized vector and the exclusive reparameterized vector, and using uncertainty to guide the fusion process to obtain the fused multimodal features, includes: Based on the dedicated reparameterization vector and the shared reparameterization vector Determine the query Q i , key K i Sum V i : Calculate the attention weight for each attention head and get the fused representation x out : Among them, Q i , K i and V i The dimensions are all d k , represents the dimension of each attention head, H is the number of attention heads, and W is a learned projection matrix; The fused multimodal feature is determined based on the fused representation. When the exclusive reparameterization vector includes the image-exclusive reparameterization vector, the fused representation includes an image fusion representation. When the exclusive reparameterization vector includes the text-exclusive reparameterization vector, the fused representation includes a text fusion representation.

2. The method according to claim 1, wherein: The unimodal extraction module includes a Swin Transformer and BERT, and the image and text are input into the unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text, including: Inputting the image into the Swin Transformer to extract image features and obtain an image representation corresponding to the image; The text is input into the BERT to perform text feature extraction to obtain a text representation corresponding to the text.

3. The method according to claim 1, wherein: Modeling the image representation and the text representation as Gaussian distributions to obtain the mean and variance of the image representation and the mean and variance of the text representation includes: The target representation is input into the uncertain representation module, and the mean and variance of the Gaussian distribution of the target representation are obtained by using a multi-head attention mechanism and a feedforward neural network, combined with layer normalization and residual connection. The target representation is the image representation or the text representation.

4. The method according to claim 1, wherein: The determining the fused multimodal feature based on the fused representation includes: Scaling the fused representation based on a weighting factor to obtain a hybrid representation, wherein the weighting factor is determined based on KL divergence; The mixed representation is optimized through residual connection, layer normalization and feed-forward network to obtain the fused multimodal features.

5. The method according to claim 4, wherein: Before scaling the fused representation based on the weighting factor to obtain the hybrid representation, the method further includes: determining an initial weight factor based on the KL divergence of the shared reparameterization vector and the dedicated reparameterization vector; The initial weight factor is subjected to linear transformation and ReLU activation to obtain the weight factor.

6. A false news identification device based on uncertainty perception, characterized by: include: A single-modal extraction module is used to extract features from images and texts to obtain an image representation corresponding to the image and a text representation corresponding to the text; an uncertainty characterization module, configured to model the image representation and the text representation as Gaussian distributions, and obtain a mean and variance of the image representation and a mean and variance of the text representation; a decoupled representation learning module, configured to input the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, input the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, input the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and input the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation; an uncertainty-aware hybrid module for fusing the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation, using uncertainty to guide the fusion process to obtain a fused multimodal feature; A classification module, configured to classify real news and fake news based on the fused multimodal features to obtain a classification result; The uncertainty-aware hybrid module includes: a reparameterization unit, configured to reparameterize the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation to obtain a shared reparameterization vector and a specific reparameterization vector, wherein the specific reparameterization vector includes an image-specific reparameterization vector and a text-specific reparameterization vector; a fusion unit, configured to perform fusion based on the shared reparameterized vector and the dedicated reparameterized vector, and to obtain the fused multimodal feature by using uncertainty to guide the fusion process; Wherein, the fusion unit includes: A first determining subunit is configured to determine a specific reparameterization vector based on the specific reparameterization vector. and the shared reparameterization vector Determine the query Q i , key K i Sum V i : The calculation subunit is used to calculate the attention weight for each attention head and obtain the fused representation x out : Among them, Q i , K i and V i The dimensions are all d k , represents the dimension of each attention head, H is the number of attention heads, and W is a learned projection matrix; The second determination subunit is used to determine the fused multimodal feature based on the fused representation, when the exclusive reparameterization vector includes the image-exclusive reparameterization vector, the fused representation includes the image fusion representation, and when the exclusive reparameterization vector includes the text-exclusive reparameterization vector, the fused representation includes the text fusion representation.

7. An electronic device comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is configured to read the program in the memory to implement the steps of the method for identifying false news based on uncertainty perception as described in any one of claims 1 to 5.

8. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the false news identification method based on uncertainty perception are implemented as described in any one of claims 1 to 5.

Images