False news identification method and device based on uncertainty perception and electronic equipment

Abstract

The invention discloses a false news recognition method, and particularly relates to a false news recognition method and device based on uncertainty perception and electronic equipment, and the method comprises the steps: inputting an image and a text into a single-mode extraction module for feature extraction, and obtaining an image representation corresponding to the image and a text representation corresponding to the text; modeling the image representation and the text representation into Gaussian distribution to obtain a mean value and a variance of the image representation and a mean value and a variance of the text representation; the shared probability representation and the exclusive probability representation of the image and the text are obtained respectively and then input into an uncertainty perception hybrid module for fusion, and a fusion process is guided by using uncertainty to obtain a fused multi-modal feature; and classifying the real news and the false news based on the fused multi-modal features to obtain a classification result. According to the method, multi-modal information can be better fused, so that a more accurate false news identification result is obtained.

Description

Technical Field

[0001] The present invention relates to a false news identification method, and in particular to a false news identification method, device and electronic equipment based on uncertainty perception. Background Art

[0002] The rise of social media and instant messaging technologies has changed the landscape of information dissemination, facilitating the rapid and widespread sharing of content. However, this advancement has also created conditions for the rapid spread of fake news, which refers to deliberately fabricated information to mislead, deceive, or manipulate public perception. Due to its intriguing headlines and content, news can quickly gain widespread interaction and spread. The resulting impact on social stability is profound, and there is an urgent need to investigate and implement effective fake news detection technology.

[0003] Multimodal fake news detection techniques integrate and process information in various media forms to identify and exploit the interconnections and interactions between different media forms, but existing methods mainly rely on deterministic representations and ignore the ambiguity and uncertainty inherent in multimodal data, which limits their ability to capture subtle relationships. This uncertainty exists both within a single modality (such as words or images may have multiple meanings in different contexts) and across modalities (for example, ambiguity may increase when text and images do not match). For example, an image may not completely match its corresponding text, which may be unintentional or part of deceptive content, increasing the complexity of interpretation. Dealing with unimodal and cross-modal uncertainty is crucial for models to flexibly and reliably analyze complex multimodal data. Therefore, the commonly used methods in the prior art cannot capture the uncertainty and ambiguity of multimodal data, resulting in low accuracy of fake news identification results. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the commonly used methods in the prior art cannot capture the uncertainty and ambiguity of multimodal data, resulting in low accuracy of false news identification results. In order to solve the above problem, the present invention provides a false news identification method, device and electronic device based on uncertainty perception.

[0005] The content of the present invention includes:

[0006] In a first aspect, an embodiment of the present invention provides a false news identification method based on uncertainty perception, comprising:

[0007] Inputting the image and the text into a unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text;

[0008] Modeling the image representation and the text representation as Gaussian distributions to obtain a mean and variance of the image representation and a mean and variance of the text representation;

[0009] Inputting the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, inputting the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, inputting the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and inputting the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation;

[0010] The image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation are input into an uncertainty-aware hybrid module for fusion, and the fusion process is guided by uncertainty to obtain a fused multimodal feature;

[0011] Based on the fused multimodal features, real news and fake news are classified to obtain classification results.

[0012] Optionally, the unimodal extraction module includes Swin Transformer and BERT, and the step of inputting the image and the text into the unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text includes:

[0013] Input the image into the Swin Transformer to extract image features, and obtain an image representation corresponding to the image;

[0014] The text is input into the BERT to extract text features and obtain a text representation corresponding to the text.

[0015] Optionally, modeling the image representation and the text representation as Gaussian distributions to obtain a mean and a variance of the image representation and a mean and a variance of the text representation includes:

[0016] The target representation is input into the uncertain representation module, and the mean and variance of the Gaussian distribution of the target representation are obtained by using a multi-head attention mechanism and a feedforward neural network in combination with layer normalization and residual connection. The target representation is the image representation or the text representation.

[0017] Optionally, the step of inputting the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation into an uncertainty-aware hybrid module for fusion, and using uncertainty to guide the fusion process to obtain a fused multimodal feature includes:

[0018] Reparameterizing the image shared probability representation, the image-specific probability representation, the text shared probability representation, and the text-specific probability representation to obtain a shared reparameterization vector and a specific reparameterization vector, wherein the specific reparameterization vector includes an image-specific reparameterization vector and a text-specific reparameterization vector;

[0019] The shared re-parameterized vector and the exclusive re-parameterized vector are fused, and the fusion process is guided by uncertainty to obtain the fused multi-modal feature.

[0020] Optionally, the fusing based on the shared reparameterization vector and the exclusive reparameterization vector, using uncertainty to guide the fusion process, to obtain the fused multimodal feature, includes:

[0021] Based on the dedicated reparameterization vector and the shared reparameterization vector Determine the query Q i , key K i Sum value V i :

[0022]

[0023] Calculate the attention weight for each attention head and get the fused representation x out :

[0024]

[0025] Among them, Q i , K i and V i The dimensions are all d k , represents the dimension of each attention head, H is the number of attention heads, and W is a learned projection matrix;

[0026] The fused multimodal feature is determined based on the fused representation. When the exclusive reparameterization vector includes the image-exclusive reparameterization vector, the fused representation includes the image fusion representation. When the exclusive reparameterization vector includes the text-exclusive reparameterization vector, the fused representation includes the text fusion representation.

[0027] Optionally, determining the fused multimodal feature based on the fused representation includes:

[0028] Scaling the fused representation based on a weighting factor to obtain a hybrid representation, wherein the weighting factor is determined based on a KL divergence;

[0029] The mixed representation is optimized through residual connection, layer normalization and feed-forward network to obtain the fused multimodal features.

[0030] Optionally, before scaling the fused representation based on the weighting factor to obtain the mixed representation, the method further includes:

[0031] Determining an initial weight factor based on the KL divergence of the shared reparameterization vector and the dedicated reparameterization vector;

[0032] The initial weight factor is subjected to linear transformation and ReLU activation to obtain the weight factor.

[0033] In a second aspect, an embodiment of the present invention provides a false news identification device based on uncertainty perception, comprising:

[0034] A single-mode extraction module is used to extract features from images and texts to obtain an image representation corresponding to the image and a text representation corresponding to the text;

[0035] An uncertainty characterization module, used for modeling the image representation and the text representation as Gaussian distributions to obtain a mean and variance of the image representation and a mean and variance of the text representation;

[0036] A decoupled representation learning module, used for inputting the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, inputting the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, inputting the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and inputting the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation;

[0037] An uncertainty-aware hybrid module, for fusing the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation, and using uncertainty to guide the fusion process to obtain a fused multimodal feature;

[0038] The classification module is used to classify real news and fake news based on the fused multimodal features to obtain a classification result.

[0039] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps in the false news identification method based on uncertainty perception as described in the first aspect.

[0040] In a fourth aspect, an embodiment of the present invention provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the false news identification method based on uncertainty perception as described in the first aspect.

[0041] The beneficial effects of the present invention are that, in the embodiments of the present application, on the one hand, Gaussian distribution is used to model the representation of information in the text modality and information in the image modality, rather than using deterministic points, which can capture the uncertainty and ambiguity in multimodal data. On the second hand, a decoupled representation learning framework is introduced to obtain image shared probability representation, image exclusive probability representation, text shared probability representation, and text exclusive probability representation, respectively, to decouple the shared and exclusive information between modalities, thereby learning a more robust and discriminative representation. On the third hand, the uncertainty of multimodal representation is used to guide the fusion process, thereby achieving better cross-modal interaction based on the uncertainty of modal representation. Through the method provided in the embodiments of the present application, multimodal information can be better fused, thereby obtaining a more accurate classification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Attached Figure 1 A flowchart of a false news identification method based on uncertainty perception provided by an embodiment of the present invention;

[0043] Attached Figure 2 A schematic diagram of a framework of a false news identification method based on uncertainty perception provided by an embodiment of the present invention;

[0044] Attached Figure 3 For attachment Figure 2 Schematic diagram of the unimodal feature extraction module;

[0045] Attached Figure 4 For attachment Figure 2 Schematic diagram of the decoupled representation learning module;

[0046] Attached Figure 5 For attachment Figure 2 Schematic diagram of the uncertainty-aware hybrid module;

[0047] Attached Figure 6 For attachment Figure 2 Schematic diagram of the probability representation module;

[0048] Attached Figure 7 For attachment Figure 2 Schematic diagram of the uncertainty-aware hybrid module;

[0049] Attached Figure 8 A schematic diagram of an uncertainty-aware hybrid module provided by an embodiment of the present invention;

[0050] Attached Fig. 9A schematic diagram of a false news identification device based on uncertainty perception provided by an embodiment of the present invention;

[0051] Attached Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In the embodiments of the present application, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In the embodiments of the present application, the term "multiple" refers to two or more, and other quantifiers are similar. The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] The embodiments of the present application provide a false news identification method and device and an electronic device based on uncertainty perception, aiming to improve the accuracy of false news identification.

[0056] See also Figure 1 , Figure 1 The present invention provides a flowchart of a method for identifying false news based on uncertainty perception, which specifically includes the following steps:

[0057] Step 101: input the image and text into a unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text.

[0058] Step 102: Model the image representation and the text representation as Gaussian distributions to obtain the mean and variance of the image representation and the mean and variance of the text representation.

[0059] Step 103, input the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, input the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, input the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and input the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation.

[0060] Step 104, inputting the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation into an uncertainty-aware hybrid module for fusion, using uncertainty to guide the fusion process, and obtaining a fused multimodal feature;

[0061] Step 105: classify real news and fake news based on the fused multimodal features to obtain a classification result.

[0062] In an embodiment of the present application, on the first hand, a Gaussian distribution is used to model the representation of multimodal information (including text information and image information) instead of using deterministic points, which can capture the uncertainty and ambiguity in multimodal data. On the second hand, a decoupled representation learning framework is introduced to obtain image shared probability representation, image exclusive probability representation, text shared probability representation and text exclusive probability representation, respectively, to decouple the shared and exclusive information between modalities, thereby learning a more robust and discriminative representation. On the third hand, the uncertainty of multimodal representation is used to guide the fusion process, so as to achieve better cross-modal interaction according to the uncertainty of modal representation. Through the method provided in the embodiment of the present application, multimodal information can be better fused, thereby obtaining a more accurate classification result.

[0063] See also Figure 2 and Figure 3 Optionally, in some embodiments, the unimodal extraction module includes SwinTransformer and BERT, and step 101 includes:

[0064] Input the image into the Swin Transformer to extract image features, and obtain an image representation corresponding to the image;

[0065] The text is input into the BERT to extract text features and obtain a text representation corresponding to the text.

[0066] As a specific example, for an input image V and text T, a pre-trained transformer-based model is used to extract an image representation X V and and text representation X T .

[0067] Specifically, for image modality data V, SwinTransformer pre-trained on a large-scale image dataset is used to extract image representations and convert them into sequence embeddings as the final image representation X V :

[0068]

[0069] Where n is the number of image blocks, is the representation of the i-th image block, d v is the hidden dimension of image representation, and SwinT is the Swin Transformer.

[0070] For text modality data T, BERT pre-trained on a large-scale text corpus is used to extract text representation X T By inputting the text T into the BERT model, we get the text representation X T :

[0071]

[0072] Among them, m is the number of tokens in the text, is the representation of the i-th token, d t is the hidden dimension of the text representation.

[0073] Optionally, in some embodiments, step 102 includes:

[0074] The target representation is input into the uncertain representation module, and the mean and variance of the Gaussian distribution of the target representation are obtained by using a multi-head attention mechanism and a feedforward neural network in combination with layer normalization and residual connection. The target representation is the image representation or the text representation.

[0075] See also Figure 2 and Figure 4 In order to represent the uncertainty and ambiguity of multimodal information, this embodiment provides an uncertainty representation module to model multimodal information as a Gaussian distribution. Gaussian distribution has elegant mathematical properties, which makes modeling and analysis more convenient, and Gaussian distribution is widely used in various fields because it is common in natural phenomena and simplifies uncertainty modeling. In this embodiment, a multi-head attention mechanism and a feedforward neural network are used, combined with layer normalization and residual connections to obtain the mean and variance of the distribution. The specific process is as follows:

[0076] To calculate the mean of the Gaussian distribution of the image representation, first we represent the image X V Applying a feedforward neural network:

[0077]

[0078] Among them, FeedForward is a feedforward neural network, which is designed to promote feature-level interactions and enrich representations by capturing complex feature dependencies.

[0079] The result is then passed through a multi-head attention layer and added to the original input through a residual connection, and the mean μ is obtained after layer normalization:

[0080]

[0081] Among them, MultiHead is a multi-head attention layer and LN is layer normalization.

[0082] To calculate the variance of the Gaussian distribution of the image representation, first transform the image representation X V The variance-specific features are computed through another feed-forward neural network:

[0083]

[0084] Then the multi-head operation captures the dependencies at the token level, thereby focusing on multiple aspects of the input sequence at the same time. These features are then added to the original features through a residual connection and normalized to obtain the logarithmic variance logσ 2 :

[0085]

[0086] in, and are the mean and variance of the Gaussian distribution represented by the image.

[0087] In order to calculate the mean of the Gaussian distribution of the text representation, we first T Applying a feedforward neural network:

[0088]

[0089] The result is then passed through a multi-head attention layer and added to the original input through a residual connection, and the mean μ is obtained after layer normalization:

[0090]

[0091] To calculate the variance of the Gaussian distribution of the image representation, first transform the text representation X TThe variance-specific features are computed through another feed-forward neural network:

[0092]

[0093] Then the multi-head operation captures the dependencies at the token level, thereby focusing on multiple aspects of the input sequence at the same time. These features are then added to the original features through a residual connection and normalized to obtain the logarithmic variance logσ 2 :

[0094]

[0095] in, and are the mean and variance of the Gaussian distribution of the image representation. The specific method of obtaining the mean and variance of the Gaussian distribution of the text representation can be found in the above-mentioned image representation part, which will not be repeated here.

[0096] In step 103, the shared information and exclusive information between modalities are decoupled through a decoupled representation learning framework to obtain image shared probability representation, image exclusive probability representation, text shared probability representation and text exclusive probability representation, so that the model can learn more robust and discriminative representations.

[0097] Specifically, the shared encoder is represented as E S , denote the image-specific encoder as E V , denote the text-specific encoder as E T . Shared Encoder E S It is used to learn the representation of modality sharing (or modality sharing features), which can capture the shared information between modalities; image-specific encoder E V Used to learn image modality-specific representations (or image modality-private features), text-specific encoder E T It is used to learn text modality-specific representations (or text modality-specific features), E V and E T Capable of capturing modality-specific information.

[0098] As a specific embodiment, the shared encoder, the image-specific encoder and the text-specific encoder are all implemented as probability representation modules, as shown in Figure 4 and Figure 6 As shown in Figure 2, the shared encoder is a modality-shared probability representation module, the image-specific encoder is an image-private probability representation module, and the text-specific encoder is a text-private probability representation module. Image shared probability representation Text Sharing Probability Representation Image-specific probabilistic representation and text-specific probability representation The mean and variance of can be obtained by the following formula:

[0099]

[0100] Where m∈{s,p} represents the shared encoder E S and dedicated encoder E V and E T , n∈{v,t} represents the image modality and text modality respectively.

[0101] Therefore, the distribution of the shared probability representation of image modalities can be obtained Distribution of shared probability representations across textual modalities Distribution of Image Modality-Specific Probability Representations Distribution of text modality-specific probability representations as follows:

[0102]

[0103] Optionally, in some embodiments, step 104 includes:

[0104] Reparameterizing the image shared probability representation, the image-specific probability representation, the text shared probability representation, and the text-specific probability representation to obtain a shared reparameterization vector and a specific reparameterization vector, wherein the specific reparameterization vector includes an image-specific reparameterization vector and a text-specific reparameterization vector;

[0105] The shared re-parameterized vector and the exclusive re-parameterized vector are fused, and the fusion process is guided by uncertainty to obtain the fused multi-modal feature.

[0106] See also Figure 7 In this embodiment, the uncertainty-aware hybrid module first generates a modality-specific shared probability representation x p and the exclusive probability representation x s , where x p For the previous or x s By the previous Concatenated, these features are modeled as Gaussian distributions to capture the uncertainty of each modality, represented by mean μ and covariance Σ.

[0107] In order to achieve gradient-based optimization, a reparameterization technique is applied in this embodiment to generate a dedicated reparameterization vector and the shared reparameterization vector The details are as follows:

[0108]

[0109] in, is a random variable sampled from a standard normal distribution, μ p and Σ p are the mean and covariance of the mode-specific probabilities, μ s and Σ s are the mean and covariance of the modal shared probability representation.

[0110] After reparameterization, a criss-cross attention mechanism is applied, where guide Specifically, and is converted into query, key and value, and then the attention weight is calculated for each attention head i. Optionally, the fusion is performed based on the shared reparameterization vector and the exclusive reparameterization vector, and uncertainty is used to guide the fusion process to obtain a fused multimodal feature, including:

[0111] Based on the dedicated reparameterization vector and the shared reparameterization vector Determine the query Q i , key K i Sum value V i :

[0112]

[0113] in, Represents a linear layer that helps the model generate attention weights through feature transformation and projection, thereby focusing on the most relevant information in the input data.

[0114] Then, the attention weights are calculated for each attention head and the fused representation x is obtained. out , the fusion is expressed as:

[0115]

[0116] Among them, Q i , K i and V i The dimensions are all d k , represents the dimension of each attention head, H is the number of attention heads, and W is a learned projection matrix;

[0117] A fused multimodal feature is determined based on the fused representation. When the exclusive reparameterization vector includes the image-exclusive reparameterization vector, the fused representation includes the image fusion representation. When the exclusive reparameterization vector includes the text-exclusive reparameterization vector, the fused representation includes the text fusion representation.

[0118] It should be understood that in this embodiment, calculations need to be performed separately for the image modality and the text modality. When calculating for the image modality, x s Depend on When calculating for text mode, x s Depend on By calculating the image modality and text modality respectively in the above way, the image fusion representation corresponding to the image modality can be obtained. And the text fusion representation corresponding to the text modality

[0119] See also Figure 8 In order to introduce uncertainty in the fusion process, the Kullback-Leibler (KL) divergence between the Gaussian distributions of private and shared features is used as a weighting factor. Although the reparameterization technique allows for differentiable sampling, it cannot fully capture the relative uncertainty between modalities. KL divergence solves this problem by directly measuring the distribution difference between private and shared features, thereby providing a detailed weighting for fusion. Optionally, in some embodiments, determining the fused multimodal features based on the fused representation includes:

[0120] Scaling the fused representation based on a weighting factor to obtain a hybrid representation, wherein the weighting factor is determined based on a KL divergence;

[0121] The mixed representation is optimized through residual connection, layer normalization and feed-forward network to obtain the fused multimodal features.

[0122] Optionally, in some embodiments, before scaling the fused representation based on the weighting factor to obtain the mixed representation, the method further includes:

[0123] Determining an initial weight factor based on the KL divergence of the shared reparameterization vector and the dedicated reparameterization vector;

[0124] The initial weight factor is subjected to linear transformation and ReLU activation to obtain the weight factor.

[0125] Among them, D KL (x p ,x s ) provides an initial weight based on uncertainty, which is further transformed linearly and activated by ReLU to produce an optimized weighting factor The weighting factor Used to scale the fused representation to obtain a mixed representation:

[0126]

[0127] After obtaining the mixed representation, the mixed representation is optimized through a series of residual connections, layer normalization (LN) and feed-forward network (FFN) to obtain fused multimodal features. The specific process can be expressed as:

[0128]

[0129] In this process, each residual connection combines the original input with the transformed representation, preserving key information while integrating new features. Layer normalization is applied after each residual connection to stabilize training by normalizing activations and reducing internal covariate shift. Feedforward networks are used to enrich representations through nonlinear transformations, allowing the model to capture more complex interactions. In this embodiment, this sequence of residual connections, LNs, and FFNs is used to normalize, enhance, and stabilize the fused representation, generating a robust and informative output with cross-modal uncertainty.

[0130] Image fusion representation corresponding to image modality By performing the above processing, the image fusion features corresponding to the image modality can be obtained. And the text fusion representation corresponding to the text modality That is, in x out for In the case of for In x out for In the case of for In the specific implementation, by and The multimodal fusion features can be obtained by fusion.

[0131] After obtaining the fused multimodal features, in step 105, real news and fake news are classified based on the fused multimodal features to obtain classification results. Specifically, each modal feature [cls] token is extracted, and the real and fake news are classified using a fully connected network.

[0132] like Figure 2 As shown, an embodiment of the present invention provides a decoupled representation learning model based on uncertainty perception, which specifically includes a unimodal feature extraction module, an uncertainty characterization module, a decoupled representation learning module, an uncertainty-aware hybrid module and a classification module.

[0133] As a specific example, during the training of the model, the total loss of the model is as follows:

[0134]

[0135] Among them, λmi , tri , ctr , rec Represent the coefficients of each loss respectively. is the classification loss, is the information redundancy constraint loss, is the inter-class triplet loss, is the intra-class center loss, is the reconstruction loss.

[0136] The following is an explanation of each loss.

[0137] Classification Loss is the cross entropy loss, calculated as:

[0138]

[0139] Where N represents the number of training samples, p i represents the probability that the i-th training sample is fake news.

[0140] In order to ensure that the shared representation and the exclusive representation can be effectively decoupled and the redundancy is minimized, an independence constraint is introduced to reduce the mutual information, thereby reducing the redundancy between the two representations. Due to its ability to capture geometric differences between distributions, the Wasserstein distance between the Gaussian distributions of the shared and exclusive representations is used as an effective measure of dependency in this embodiment, thereby encouraging minimized overlap and redundancy.

[0141] Taking the text mode as an example, for two Gaussian distributions and The closed-form expression of its 2-Wasserstein distance is:

[0142]

[0143] The goal in this example is to reduce the redundancy between shared and exclusive representations by maximizing the approximation of mutual information. Therefore, the information redundancy constraint loss is defined as for:

[0144]

[0145] By setting the information redundancy constraint loss, the independence between shared and exclusive representations can be encouraged by reducing redundancy, thereby improving the decoupling effect and overall performance of the model.

[0146] To ensure the alignment of shared representations across modalities, this embodiment introduces a hybrid loss that combines the inter-class triplet loss and the intra-class center loss. The inter-class triplet loss encourages multimodal representations from the same category to be closer to each other while separating representations of different categories. The intra-class center loss aims to reduce the intra-class variance of the shared representations and further ensure consistency within each category.

[0147] Specifically, the inter-class triplet loss is defined as follows:

[0148]

[0149] in, and A shared representation of samples from image and text modalities, respectively; and Represents positive samples from the same category; and represents negative samples from different categories; α is the boundary that encourages separation of representations between classes; D KL (·||·) represents two Gaussian distributions and The KL divergence between .

[0150] However, relying solely on the inter-class triplet loss only considers the separation between categories, which may not be enough to ensure strong alignment within each category. Therefore, this embodiment also introduces the intra-class center loss to further minimize the intra-class variance of the shared representation. The intra-class center loss is defined as:

[0151]

[0152] Among them, C is the number of categories, N is the number of samples in each category, and τ j is the weight of the jth distribution, given by Given, μ j is the mean of the jth shared representation, and μ weighted is the weighted mean of the categories, calculated as

[0153] Finally, the consistency constraint combines the inter-class triplet loss and the intra-class center loss, expressed as:

[0154]

[0155] In order to distinguish shared and exclusive representations and reduce information loss in the process of learning disentangled representations, information reconstruction constraints are introduced. Specifically, Figure 7 As shown, the image decoder and text decoder are used to reconstruct the input representation, which further ensures that the information is well preserved during the disentangled representation learning process.

[0156] The reconstruction loss can be expressed as:

[0157]

[0158] Among them, Decoder V and Decoder T They are image decoder and text decoder respectively. The image decoder and text decoder are implemented as feed-forward neural networks to reconstruct the input image and text representations.

[0159] As a specific embodiment, first, a Gaussian distribution is used to model the representation of multimodal information instead of using deterministic points, which can capture the uncertainty and ambiguity in multimodal data. The parameters of the Gaussian distribution can be learned by the probabilistic representation module proposed in this embodiment. Secondly, a decoupled representation learning framework is introduced to decouple the shared and exclusive information between modalities, so that the model can learn more robust and discriminative representations. Specifically, the representation shared by the modality is learned by a shared encoder. In order to ensure the consistency of the cross-modal shared representation, a triple learning component is introduced to bring the multimodal representations of the same type closer while pushing the representations of different classes away, while the modality-specific representation is learned by a dedicated encoder to capture the unique information of the modality. In this process, independence constraints are introduced to ensure that the mutual information between shared and exclusive representations is minimized. In addition, the use of image decoders and text decoders to reconstruct the input representation can further ensure that the information is well preserved during the decoupled representation learning process. Finally, this embodiment proposes an uncertainty-aware fusion module that uses the uncertainty of multimodal representations to guide the fusion process, thereby achieving better cross-modal interaction based on the uncertainty of modal representations. Specifically, a weight factor is learned for each modality based on its uncertainty, which can dynamically adjust the importance of each modality during the fusion process. Through the above method, the effect of multimodal data fusion can be improved, thereby improving the accuracy of fake news identification.

[0160] See also Fig. 9 The embodiment of the present invention further provides a false news identification device 900 based on uncertainty perception, comprising:

[0161] The single-mode extraction module 901 is used to extract features from images and texts to obtain image representations corresponding to the images and text representations corresponding to the texts;

[0162] An uncertainty characterization module 902, configured to model the image representation and the text representation as Gaussian distributions, and obtain a mean and variance of the image representation and a mean and variance of the text representation;

[0163] A decoupled representation learning module 903, configured to input the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, input the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, input the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and input the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation;

[0164] An uncertainty-aware hybrid module 904, for fusing the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation, and using uncertainty to guide the fusion process to obtain a fused multimodal feature;

[0165] The classification module 905 is used to classify real news and fake news based on the fused multimodal features to obtain a classification result.

[0166] Optionally, the unimodal extraction module includes Swin Transformer and BERT, and the unimodal extraction module is specifically used for:

[0167] Input the image into the Swin Transformer to extract image features, and obtain an image representation corresponding to the image;

[0168] The text is input into the BERT to extract text features and obtain a text representation corresponding to the text.

[0169] Optionally, the uncertainty characterization module 902 is specifically used for:

[0170] The target representation is input into the uncertain representation module, and the mean and variance of the Gaussian distribution of the target representation are obtained by using a multi-head attention mechanism and a feedforward neural network in combination with layer normalization and residual connection. The target representation is the image representation or the text representation.

[0171] Optionally, the uncertainty-aware mixing module 904 includes:

[0172] A reparameterization unit, used for reparameterizing the image shared probability representation, the image-specific probability representation, the text shared probability representation, and the text-specific probability representation to obtain a shared reparameterization vector and a specific reparameterization vector, wherein the specific reparameterization vector includes an image-specific reparameterization vector and a text-specific reparameterization vector;

[0173] A fusion unit is used to perform fusion based on the shared re-parameterization vector and the exclusive re-parameterization vector, and use uncertainty to guide the fusion process to obtain the fused multi-modal feature.

[0174] Optionally, the fusion unit comprises:

[0175] Determine a subunit for re-parameterizing the vector based on the exclusive and the shared reparameterization vector Determine the query Q i , key K i Sum value V i :

[0176]

[0177]

[0178] The computation unit is used to calculate the attention weight for each attention head and obtain the fused representation x out :

[0179]

[0180] Among them, Q i , K i and V i The dimensions are all d k , represents the dimension of each attention head, H is the number of attention heads, and W is a learned projection matrix;

[0181] A determination unit is used to determine the fused multimodal feature based on the fused representation, when the exclusive reparameterization vector includes the image-exclusive reparameterization vector, the fused representation includes the image fusion representation, when the exclusive reparameterization vector includes the text-exclusive reparameterization vector, the fused representation includes the text fusion representation.

[0182] Optionally, the determining unit is specifically configured to:

[0183] Scaling the fused representation based on a weighting factor to obtain a hybrid representation, wherein the weighting factor is determined based on a KL divergence;

[0184] The mixed representation is optimized through residual connection, layer normalization and feed-forward network to obtain the fused multimodal features.

[0185] Optionally, the determining unit is further configured to:

[0186] Determining an initial weight factor based on the KL divergence of the shared reparameterization vector and the dedicated reparameterization vector;

[0187] The initial weight factor is subjected to linear transformation and ReLU activation to obtain the weight factor.

[0188] The false news identification device 900 based on uncertainty perception provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be repeated in this embodiment.

[0189] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0191] like Fig.10 As shown, an embodiment of the present application provides an electronic device 1000, including: a memory 1002, a processor 1001, and a program stored in the memory 1002 and executable on the processor 1001; the processor 1001 is used to read the program in the memory 1002 to implement the steps in the false news identification method based on uncertainty perception as described above.

[0192] The embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by a processor, the various processes of the above-mentioned false news identification method embodiment based on uncertainty perception are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as compact disk (CD), digital video disc (DVD), Blu-ray Disc (BD), high-definition versatile disc (HVD), etc.), and semiconductor memory (such as read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read only memory (EEPROM), non-volatile memory (NAND FLASH), solid-state drive (Solid State Disk or Solid State Drive, SSD)), etc.

[0193] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0195] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A false news identification method based on uncertainty perception, characterized in that: include: Inputting the image and the text into a unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text; Modeling the image representation and the text representation as Gaussian distributions to obtain a mean and variance of the image representation and a mean and variance of the text representation; Inputting the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, inputting the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, inputting the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and inputting the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation; The image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation are input into an uncertainty-aware hybrid module for fusion, and the fusion process is guided by uncertainty to obtain a fused multimodal feature; Based on the fused multimodal features, real news and fake news are classified to obtain classification results.

2. The method according to claim 1, characterized in that: The unimodal extraction module includes Swin Transformer and BERT, and the image and text are input into the unimodal extraction module for feature extraction to obtain an image representation corresponding to the image and a text representation corresponding to the text, including: Input the image into the Swin Transformer to extract image features, and obtain an image representation corresponding to the image; The text is input into the BERT to extract text features and obtain a text representation corresponding to the text.

3. The method according to claim 1, characterized in that: The step of modeling the image representation and the text representation as Gaussian distributions to obtain a mean and a variance of the image representation and a mean and a variance of the text representation includes: The target representation is input into the uncertain representation module, and the mean and variance of the Gaussian distribution of the target representation are obtained by using a multi-head attention mechanism and a feedforward neural network in combination with layer normalization and residual connection. The target representation is the image representation or the text representation.

4. The method according to any one of claims 1 to 3, characterized in that: The step of inputting the image shared probability representation, the image exclusive probability representation, the text shared probability representation and the text exclusive probability representation into an uncertainty-aware hybrid module for fusion, and using uncertainty to guide the fusion process to obtain a fused multimodal feature includes: Reparameterizing the image shared probability representation, the image-specific probability representation, the text shared probability representation, and the text-specific probability representation to obtain a shared reparameterization vector and a specific reparameterization vector, wherein the specific reparameterization vector includes an image-specific reparameterization vector and a text-specific reparameterization vector; The shared re-parameterized vector and the exclusive re-parameterized vector are fused, and the fusion process is guided by uncertainty to obtain the fused multi-modal feature.

5. The method according to claim 4, characterized in that: The fusing based on the shared re-parameterized vector and the exclusive re-parameterized vector, and using uncertainty to guide the fusion process to obtain the fused multi-modal feature, includes: Based on the dedicated reparameterization vector and the shared reparameterization vector Determine the query Q i , key K i Sum value V i : Calculate the attention weight for each attention head and get the fused representation x out : Among them, Q i , K i and V i The dimensions are all d k , represents the dimension of each attention head, H is the number of attention heads, and W is a learned projection matrix; The fused multimodal feature is determined based on the fused representation. When the exclusive reparameterization vector includes the image-exclusive reparameterization vector, the fused representation includes the image fusion representation. When the exclusive reparameterization vector includes the text-exclusive reparameterization vector, the fused representation includes the text fusion representation.

6. The method according to claim 5, characterized in that: The determining the fused multimodal feature based on the fused representation includes: Scaling the fused representation based on a weighting factor to obtain a hybrid representation, wherein the weighting factor is determined based on a KL divergence; The mixed representation is optimized through residual connection, layer normalization and feed-forward network to obtain the fused multimodal features.

7. The method according to claim 6, characterized in that: Before scaling the fused representation based on the weighting factor to obtain the mixed representation, the method further includes: Determining an initial weight factor based on the KL divergence of the shared reparameterization vector and the dedicated reparameterization vector; The initial weight factor is subjected to linear transformation and ReLU activation to obtain the weight factor.

8. A false news identification device based on uncertainty perception, characterized in that: include: A single-mode extraction module is used to extract features from images and texts to obtain an image representation corresponding to the image and a text representation corresponding to the text; An uncertainty characterization module, used for modeling the image representation and the text representation as Gaussian distributions to obtain a mean and variance of the image representation and a mean and variance of the text representation; A decoupled representation learning module, used for inputting the mean and variance of the image representation into a shared encoder to obtain an image shared probability representation, inputting the mean and variance of the image representation into an image-specific encoder to obtain an image-specific probability representation, inputting the mean and variance of the text representation into the shared encoder to obtain a text shared probability representation, and inputting the mean and variance of the text representation into a text-specific encoder to obtain a text-specific probability representation; An uncertainty-aware hybrid module, for fusing the image-shared probability representation, the image-specific probability representation, the text-shared probability representation, and the text-specific probability representation, and using uncertainty to guide the fusion process to obtain a fused multimodal feature; The classification module is used to classify real news and fake news based on the fused multimodal features to obtain a classification result.

9. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the false news identification method based on uncertainty perception as described in any one of claims 1 to 7.

10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps in the false news identification method based on uncertainty perception as described in any one of claims 1 to 7 are implemented.

Images