False news detection method and device based on dual consistency verification module

Abstract

This invention discloses a method, apparatus, storage medium, and electronic device for detecting fake news based on a dual consistency verification module. Belonging to the fields of artificial intelligence and fake information detection, the technical problem this invention aims to solve is how to jointly verify the authenticity of news content from two perspectives: internal text-image consistency and external retrieval consistency, thereby obtaining a discriminative news encoding representation and improving the accuracy of fake news detection. The technical solution adopted is as follows: S1, Obtaining a fake news detection dataset: Downloading a publicly available fake news detection dataset from the internet; S2, Constructing a fake news detection model: Constructing a fake news detection model based on the dual consistency verification module; S3, Training the fake news detection model: Training the fake news detection model constructed in step S2 on the fake news detection dataset obtained in step S1.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and fake information detection technology, specifically to a method and apparatus for detecting fake news based on a dual consistency verification module. Background Technology

[0002] Fake news detection is a technology that analyzes news content and related information to determine the authenticity of news. This news can include, but is not limited to, various types such as social events, breaking news, health information, and entertainment gossip, as well as more specific forms such as multimodal news content like photo reports, social media tweets, and short video captions. Fake news detection typically relies on multimodal features extracted from news text and images, such as semantic information in the text, visual information in the images, and semantic relationships between text and images. As a cutting-edge technological application, the value and importance of fake news detection technology are reflected in various social needs and technological advancements. For example, in the field of social media governance, fake news detection technology can identify and intercept misleading information in a timely manner by analyzing massive amounts of published news content in real time, curbing the spread of rumors, and thus maintaining a healthy online public opinion environment. This is of great significance for protecting the public's right to know and maintaining social stability. In the news media industry, the application of fake news detection is reflected in assisting editors and fact-checkers by automating the verification of the authenticity of news content. This includes automatically marking suspicious content and recommending reference evidence. This not only enhances the credibility of news releases but also promotes the healthy development of the media ecosystem and drives news dissemination towards a more standardized and credible direction.

[0003] A core challenge in fake news detection lies in the concealment of false information and the diversity of discernible clues. News content typically consists of multiple modalities, including text and images, which collectively convey the information about the events described in the news. However, due to the diversity and deception of fake news fabrication methods, even seemingly authentic news reports can have highly concealed falsehoods. First, relying solely on internal text-image consistency checks is insufficient to identify all fake news. For example, in meticulously crafted fake news with consistent text and images, the text and images may be semantically highly matched, but the event itself is fabricated. If the detection model only considers internal text-image consistency, it cannot detect such fabricated content. Furthermore, in authentic news using diagrams or reference images, there may be semantic differences between the text and images, easily leading to misjudgments. Second, relying solely on external evidence also has limitations. For breaking news or recently occurring events, external evidence databases may not yet contain relevant information, resulting in a lack of effective reference evidence. For fake reports deliberately mimicking the style of authentic news, a single external check may also lead to misjudgments due to the retrieval of only partially relevant content. If the detection model cannot jointly verify the news content from the perspectives of internal image and text consistency and external evidence retrieval consistency, it may lead to incorrect judgment results.

[0004] To address the aforementioned problems, this invention introduces a method and apparatus for detecting fake news based on a dual consistency verification module. This method and apparatus, based on the dual consistency verification module, enables the model to jointly verify the authenticity of news content from two perspectives: internal text-image consistency and external retrieval consistency. This results in a discriminative news encoding representation, thereby improving the accuracy of fake news detection. Summary of the Invention

[0005] The technical objective of this invention is to provide a method, apparatus, storage medium, and electronic device for detecting fake news based on a dual consistency verification module, which can jointly verify the authenticity of news content from two perspectives: internal image and text consistency and external retrieval consistency, thereby improving the accuracy of fake news detection.

[0006] The present invention achieves its objective by employing the following technical means:

[0007] The technical objective of this invention is achieved as follows: a method for detecting fake news based on a dual consistency verification module, comprising the following steps:

[0008] S1. Obtain the fake news detection dataset: Download the publicly available fake news detection dataset from the internet;

[0009] S2. Constructing a fake news detection model: Constructing a fake news detection model based on a dual consistency verification module;

[0010] S3. Train the fake news detection model: Train the fake news detection model constructed in step S2 on the fake news detection dataset obtained in step S1.

[0011] As a further limitation of this technical solution, step S2 is used to construct an input module, a dual encoding module, a dual consistency verification module, and a fusion classification module, thereby constructing a fake news detection model; the aforementioned modules correspond to sub-steps, denoted as S201, S202, S203, and S204, respectively.

[0012] As a further limitation of this technical solution, step S201 constructs an input module, which extracts one data point from the fake news detection dataset obtained in step S1. This data point contains a news text, a news image, and a corresponding true / false label, thus obtaining one input data point, denoted as . ,in Represents news text data, Represents news image data, The labels indicate whether the news is real or fake; 1 represents real news and 0 represents fake news.

[0013] S202 constructs a dual-encoding module, using a pre-trained CLIP model as the backbone network, to encode the news text data and news image data in the input data constructed in S201, thereby obtaining text features and image features, denoted as... and The specific implementation is shown in the following formula:

[0014] ;

[0015] ;

[0016] in, This indicates news text data, sourced from S201. This indicates news image data, sourced from S201. The text encoder representing CLIP. This refers to the CLIP image encoder.

[0017] As a further limitation of this technical solution, S203 constructs a dual consistency verification module, which includes an internal consistency verification submodule and an external consistency verification submodule; wherein, the internal consistency verification submodule receives text features. and image features The semantic consistency of the text and images within the news is verified to obtain the intrinsic interaction features. Consistency score ;

[0018] Specifically, the implementation process of the intrinsic consistency verification submodule is as follows:

[0019] S20301, Generation of text perturbation view and image perturbation view:

[0020] Text features are processed using a text perturbation mapping method, and the processing result is then mapped using the Gumbel-Softmax function to obtain a text semantic mask. The text semantic mask is then multiplied element-wise by the text features, and the result is multiplied by a perturbation coefficient and added element-wise to the text features to obtain a text perturbation view. The specific implementation is shown in the following formula:

[0021] ;

[0022] ;

[0023] in, Representing text features, GumbelSoftmax represents the learnable parameters in the text perturbation map, and GumbelSoftmax represents the Gumbel-Softmax function. Represents a semantic mask for text. This represents the disturbance coefficient. This indicates element-wise multiplication. This represents a text perturbation view;

[0024] Image features are processed using image perturbation mapping, and the processing result is mapped using the Gumbel-Softmax function to obtain an image semantic mask. The image semantic mask is then multiplied element-wise by the image features, and the result is multiplied by a perturbation coefficient and added element-wise to the image features to obtain a perturbed image view. The specific implementation is shown in the following formula:

[0025] ;

[0026] ;

[0027] in, Representing image features, GumbelSoftmax represents the learnable parameters in the image perturbation map, and GumbelSoftmax represents the Gumbel-Softmax function. Represents an image semantic mask. This represents the disturbance coefficient. This indicates element-wise multiplication. Indicates a perturbed view of the image;

[0028] S20302. Generation of original interaction features and disturbance response features:

[0029] Using text features as queries and image features as keys, a cross-attention mechanism is used to obtain the original interaction features. Using a text-perturbed view as a query and an image-perturbed view as a key, the same cross-attention mechanism is used to obtain the perturbed interaction features. The perturbed response features are obtained by subtracting the original interaction features from the perturbed interaction features element-wise. See the following formula for details:

[0030] ;

[0031] ;

[0032] ;

[0033] in, Representing text features, Representing image features, This represents a text perturbation view. This represents the perturbation view of the image, all from S20301. CrossAttn represents the cross-attention mechanism. Represents the original interaction features. This indicates the characteristics of perturbation interaction. Indicates the characteristics of the disturbance response;

[0034] S20303, Bilinear Matching Feature Generation:

[0035] Text features are processed using a low-rank text projection matrix, and the result is mapped using the ReLU function to obtain the low-rank text projection result. Image features are processed using an image low-rank matrix projection, and the result is mapped using the ReLU function to obtain the low-rank image projection result. The low-rank text projection result and the low-rank image projection result are then multiplied element-wise to obtain the bilinear matching feature. See the following formula for details:

[0036] ;

[0037] in, Representing text features, Representing image features, This represents the learnable parameters in the low-rank projection matrix of the text. This represents the learnable parameters in the low-rank matrix projection of the image. Represents the ReLU function. This indicates element-wise multiplication. This represents the bilinear matching feature;

[0038] S20304, Construction of Intrinsic Interaction Features and Consistency Scoring:

[0039] The original interaction features, perturbation response features, and bilinear matching features are concatenated to obtain the intrinsic interaction features. These intrinsic interaction features are then mapped using a scoring head network and a sigmoid function to obtain the intrinsic consistency score. The specific implementation is shown in the following formula:

[0040] ;

[0041] ;

[0042] in, This represents the original interaction features, derived from S20302. This indicates the disturbance response characteristics, originating from S20302. This represents a bilinear matching feature, derived from S20303. Indicates a join operation. Indicates intrinsic interaction characteristics, This represents the learnable parameters in the scoring head network. This represents the Sigmoid function. This represents the intrinsic consistency score.

[0043] As a further limitation of this technical solution, S203 constructs a dual consistency verification module, which includes an internal consistency verification submodule and an external consistency verification submodule; wherein, the external consistency verification submodule receives news text data. News image data Text features and image features The consistency between news content and external search evidence is verified to obtain the external evidence interaction characteristics. Consistency score with foreign evidence ;

[0044] Specifically, the implementation process of the external certificate consistency verification submodule is as follows:

[0045] S20401. Generation of text-based and image-based evidence features:

[0046] News text data is input into a large language model for processing to obtain text-perspective query statements. These queries are then processed using Tavily search to obtain text-perspective retrieval results. These results are then encoded using an encoding network to obtain text-perspective evidence features. The specific implementation is illustrated in the following formula:

[0047] ;

[0048] ;

[0049] in, Represents news text data, Representing a large language model, This indicates a query statement from a textual perspective. This indicates a Tavily search. Indicates the number of search results. This represents an encoding network that concatenates and uniformly encodes the k retrieved text fragments. Indicates the characteristics of evidence from a textual perspective;

[0050] News image data is input into a large language model for processing to obtain image perspective query statements; the image perspective query statements are then used to obtain image perspective retrieval results through Tavily search, and these results are encoded using an encoding network to obtain image perspective evidence features; the specific implementation is shown in the following formula:

[0051] ;

[0052] ;

[0053] in, Represents news image data, Representing a large language model, This represents a query statement indicating the viewpoint of an image. This indicates a Tavily search. Indicates the number of search results. This represents an encoding network that concatenates and uniformly encodes the k retrieved text fragments. Indicates the characteristics of image perspective evidence;

[0054] S20402, News Display Generation:

[0055] The text features and image features are concatenated, and the concatenation result is mapped using a news representation projection matrix to obtain the news representation; the specific implementation is shown in the following formula:

[0056] ;

[0057] in, Representing text features, Representing image features, Indicates a join operation. The news represents the projection matrix. This indicates news reporting;

[0058] S20403. Generation of textual perspective stance features and image perspective stance features:

[0059] The news representation is concatenated with text-based evidence features, and the concatenation result is mapped using a position-discrimination projection matrix and a Tanh function to obtain text-based position features. Similarly, the news representation is concatenated with image-based evidence features, and the concatenation result is mapped using a position-discrimination projection matrix and a Tanh function to obtain image-based position features. The specific implementation is shown in the following formula:

[0060] ;

[0061] ;

[0062] in, The news indicates that it originated from S20402. This represents textual evidence features, derived from S20401. This represents the features of the image perspective evidence, derived from S20401. Represents the position discrimination projection matrix. This represents the Tanh function. Indicates the perspective and stance of the text. Indicates the perspective and positional characteristics of an image;

[0063] S20404, Construction of External Evidence Interaction Features and Consistency Scoring:

[0064] The textual perspective stance features and textual perspective evidence features are multiplied element-wise, and the image perspective stance features and image perspective evidence features are multiplied element-wise. The results of these two multiplications are then summed element-wise to obtain the external evidence interaction features. These external evidence interaction features are then mapped using a scoring head network and a sigmoid function to obtain the external evidence consistency score. The specific implementation is shown in the following formula:

[0065] ;

[0066] ;

[0067] in, This indicates the text's perspective and stance characteristics, derived from S20403. This represents the image's perspective and position characteristics, derived from S20403. This represents textual evidence features, derived from S20401. This represents the features of the image perspective evidence, derived from S20401. This indicates element-wise multiplication. Indicates the interaction characteristics of external evidence. This represents the learnable parameters in the scoring head network. This represents the Sigmoid function. This indicates the consistency score of the external certificate.

[0068] As a further limitation of this technical solution, S204 constructs a fusion classification module, which receives the intrinsic interaction features output by the dual consistency verification module. Intrinsic consistency score External evidence interaction characteristics Consistency score with foreign evidence The data is then fused and categorized to determine the probability that the news item is fake news.

[0069] Specifically, the implementation process of the fusion classification module is as follows:

[0070] First, a dual-source score fusion mapping process is performed on the intrinsic consistency score and the extrinsic consistency score to obtain the dual-source consistency score; the specific implementation is shown in the following formula:

[0071] ;

[0072] in, Indicates the intrinsic consistency score. This indicates the score for consistency with external documentation. This represents the dual-source consistency fusion matrix. This represents the Sigmoid function. Indicates the two-source consistency score;

[0073] Then, a dual-source feature fusion process is performed on the intrinsic interaction features and the extrinsic interaction features to obtain the dual-source interaction features; the specific implementation is shown in the following formula:

[0074] ;

[0075] in, Indicates intrinsic interaction characteristics, Indicates the interaction characteristics of external evidence. Indicates a join operation. This represents a dual-source interactive fusion matrix. Indicates dual-source interaction characteristics;

[0076] Finally, the two-source consistency score and the two-source interaction features are multiplied by a scalar-vector multiplication, and the result is mapped through a classification head network and a sigmoid function to obtain the probability that the news data belongs to fake news, denoted as . The specific implementation is shown in the following formula:

[0077] ;

[0078] in, Indicates the two-source consistency score. Indicates dual-source interaction characteristics, Represents the classification head network, This represents the Sigmoid function. This indicates the probability that the news item is fake news.

[0079] As a further limitation of this technical solution, step S3 is used to construct a classification loss function, an intrinsic consistency loss function, an extrinsic consistency loss function, and an optimization function, thereby training the fake news detection model; the aforementioned modules correspond to sub-steps, denoted as S301, S302, S303, and S304, and are described in sequence as follows:

[0080] S301. Construct the classification loss function:

[0081] This invention uses binary cross-entropy as the classification loss function, as shown in the following formula:

[0082] ;

[0083] in, For genuine and fake labels, The false probability of the model's output;

[0084] S302. Construct the intrinsic consistency loss function:

[0085] This invention employs binary cross-entropy as the intrinsic consistency loss function and introduces an intrinsic contrast regularization term with random permutation within a batch to enhance the discriminative power of the intrinsic consistency score. The formula is shown below:

[0086] ;

[0087] ;

[0088] in, For genuine and fake labels, The intrinsic consistency score, The batch size, This is a random permutation function within a batch. This means combining the text of the i-th sample with the text of the permuted sample. The intrinsic consistency score is calculated after pairing images of a sample.

[0089] S303. Constructing the external consistency loss function:

[0090] This invention uses binary cross-entropy as the external consistency loss function, as shown in the following formula:

[0091] ;

[0092] in, For genuine and fake labels, The score is the consistency score with external evidence.

[0093] The total loss function is obtained by weighted summing of the classification loss, intrinsic consistency loss, intrinsic contrast regularization term, and extrinsic consistency loss, as shown in the following formula:

[0094] ;

[0095] in, , , This is the loss weighting balancing coefficient;

[0096] S304. Construct the optimization function:

[0097] After testing various optimization functions, the Adam optimization function was ultimately selected. Except for its learning rate, which was set to 3e-5, all other hyperparameters of Adam were set to their default values ​​in PyTorch. Once the model was trained, predictions were performed on the test set. News text and images were input into the model to obtain the probability that a news item was fake, and the authenticity of the news was determined accordingly.

[0098] A fake news detection device based on a dual consistency verification module, the device comprising:

[0099] The fake news detection dataset acquisition unit is used to download publicly available fake news detection datasets from the internet;

[0100] The fake news detection model building unit is used to build an input module, a dual encoding module, a dual consistency verification module, and a fusion classification module, thereby building a fake news detection model.

[0101] The fake news detection model training unit is used to construct classification loss functions, intrinsic consistency loss functions, extrinsic consistency loss functions, and optimization functions, thereby completing the training and prediction of the model.

[0102] An electronic device includes: a memory and at least one processor;

[0103] The memory contains computer programs;

[0104] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the fake news detection method based on the double consistency verification module as described above.

[0105] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the fake news detection method based on a dual consistency verification module as described above.

[0106] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0107] (1) This invention can efficiently extract semantic features of news text and news images through a dual encoding module, while maintaining the computational efficiency of the model, and realize the rapid processing and analysis of multimodal news content;

[0108] (2) The present invention can extract clues for identifying fake news from the perspective of semantic matching of text and images within the news through the internal consistency verification submodule, thereby obtaining a news encoding representation that can distinguish fake news of text and image mismatch, thereby improving the accuracy and reliability of the model in detecting fake news;

[0109] (3) The present invention can extract clues for identifying fake news from the perspective of matching news content with external search evidence through the external evidence consistency verification submodule, thereby enhancing the model’s ability to identify fabricated fake news and improving the accuracy and reliability of the model in detecting fake news.

[0110] (4) By integrating the classification module, this invention can adaptively integrate the internal news coding representation and the external news coding representation, thereby improving the generalization ability and applicability of the model;

[0111] (5) The method and apparatus proposed in this invention, combined with the dual consistency verification module, can jointly verify the authenticity of news content from two perspectives: internal image and text consistency and external evidence retrieval consistency, thereby improving the accuracy of fake news detection. Attached Figure Description

[0112] Figure 1 This is a flowchart of a fake news detection method based on a double consistency verification module;

[0113] Figure 2 A flowchart for building a fake news detection model;

[0114] Figure 3 A flowchart for training a fake news detection model;

[0115] Figure 4 A flowchart of a fake news detection device based on a dual consistency verification module;

[0116] Figure 5 This is a schematic diagram of the internal consistency verification submodule;

[0117] Figure 6This is a structural diagram of the external certificate consistency verification submodule;

[0118] Figure 7 This is a schematic diagram of the framework of a fake news detection model based on a double consistency verification module. Detailed Implementation

[0119] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0120] Example 1:

[0121] The overall model framework structure of this invention is as follows: Figure 7 As shown. By Figure 7 As can be seen, the main framework of this invention comprises three modules: a dual-encoding module, a dual-consistency verification module, and a fusion classification module. The dual-consistency verification module further includes an internal consistency verification submodule and an external consistency verification submodule. The dual-encoding module receives news text data and news image data. It encodes the news text data using a text encoder to obtain text features, and encodes the news image data using an image encoder to obtain image features. The news text data, news image data, text features, and image features are then fed into the dual-consistency verification module. The dual-consistency verification module receives the news text data, news image data, text features, and image features, and processes them respectively through the internal consistency verification submodule and the external consistency verification submodule, thereby obtaining internal interaction features, internal consistency scores, external interaction features, and external consistency scores. The results are then sent to the fusion classification module. The fusion classification module receives internal interaction features, internal consistency scores, external evidence interaction features, and external evidence consistency scores. First, it performs a dual-source score fusion mapping process on the internal consistency scores and external evidence consistency scores to obtain dual-source consistency scores. Then, it performs a dual-source feature interaction fusion process on the internal interaction features and external evidence interaction features to obtain dual-source interaction features. Finally, it performs a scalar-vector multiplication on the dual-source consistency scores and dual-source interaction features and maps them through the classification head network to obtain the predicted probability that the news data belongs to fake news. This result is then fed back to relevant personnel as the prediction result.

[0122] Example 2:

[0123] Internal consistency verification submodule, such as Figure 5As shown, the intrinsic consistency verification submodule receives text features and image features. First, it maps the text features and image features using a perturbation mapping matrix and the Gumbel-Softmax function to obtain text semantic masks and image semantic masks, respectively. Then, it combines these with perturbation coefficients to obtain text perturbation views and image perturbation views. Next, it processes the text features and image features through a cross-attention mechanism to obtain the original interaction features, and processes the text perturbation views and image perturbation views through the same mechanism to obtain perturbation interaction features. The two are then subtracted to obtain perturbation response features. Simultaneously, the text features and image features are processed through low-rank projection and the ReLU function, and then multiplied element-wise to obtain bilinear matching features. Finally, the original interaction features, perturbation response features, and bilinear matching features are concatenated to obtain intrinsic interaction features. Finally, the intrinsic interaction features are mapped through a scoring head network and the Sigmoid function to obtain the intrinsic consistency score.

[0124] Specifically, the implementation process of this internal consistency verification submodule is as follows:

[0125] S20301, Generation of text perturbation view and image perturbation view:

[0126] Text features are processed using a text perturbation mapping method, and the processing result is then mapped using the Gumbel-Softmax function to obtain a text semantic mask. The text semantic mask is then multiplied element-wise by the text features, and the result is multiplied by a perturbation coefficient and added element-wise to the text features to obtain a text perturbation view. The specific implementation is shown in the following formula:

[0127] ;

[0128] ;

[0129] in, Representing text features, GumbelSoftmax represents the learnable parameters in the text perturbation map, and GumbelSoftmax represents the Gumbel-Softmax function. Represents a semantic mask for text. This represents the disturbance coefficient. This indicates element-wise multiplication. This represents a text perturbation view;

[0130] Image features are processed using image perturbation mapping, and the processing result is mapped using the Gumbel-Softmax function to obtain an image semantic mask. The image semantic mask is then multiplied element-wise by the image features, and the result is multiplied by a perturbation coefficient and added element-wise to the image features to obtain a perturbed image view. The specific implementation is shown in the following formula:

[0131] ;

[0132] ;

[0133] in, Representing image features, GumbelSoftmax represents the learnable parameters in the image perturbation map, and GumbelSoftmax represents the Gumbel-Softmax function. Represents an image semantic mask. This represents the disturbance coefficient. This indicates element-wise multiplication. Indicates a perturbed view of the image;

[0134] S20302. Generation of original interaction features and disturbance response features:

[0135] Using text features as queries and image features as keys, a cross-attention mechanism is used to obtain the original interaction features. Using a text-perturbed view as a query and an image-perturbed view as a key, the same cross-attention mechanism is used to obtain the perturbed interaction features. The perturbed response features are obtained by subtracting the original interaction features from the perturbed interaction features element-wise. See the following formula for details:

[0136] ;

[0137] ;

[0138] ;

[0139] in, Representing text features, Representing image features, This represents a text perturbation view. This represents the perturbation view of the image, all from S20301. CrossAttn represents the cross-attention mechanism. Represents the original interaction features. This indicates the characteristics of perturbation interaction. Indicates the characteristics of the disturbance response;

[0140] S20303, Bilinear Matching Feature Generation:

[0141] Text features are processed using a low-rank text projection matrix, and the result is mapped using the ReLU function to obtain the low-rank text projection result. Image features are processed using an image low-rank matrix projection, and the result is mapped using the ReLU function to obtain the low-rank image projection result. The low-rank text projection result and the low-rank image projection result are then multiplied element-wise to obtain the bilinear matching feature. See the following formula for details:

[0142] ;

[0143] in, Representing text features, Representing image features, This represents the learnable parameters in the low-rank projection matrix of the text. This represents the learnable parameters in the low-rank matrix projection of the image. Represents the ReLU function. This indicates element-wise multiplication. This represents the bilinear matching feature;

[0144] S20304, Construction of Intrinsic Interaction Features and Consistency Scoring:

[0145] The original interaction features, perturbation response features, and bilinear matching features are concatenated to obtain the intrinsic interaction features. These intrinsic interaction features are then mapped using a scoring head network and a sigmoid function to obtain the intrinsic consistency score. The specific implementation is shown in the following formula:

[0146] ;

[0147] ;

[0148] in, This represents the original interaction features, derived from S20302. This indicates the disturbance response characteristics, originating from S20302. This represents a bilinear matching feature, derived from S20303. Indicates a join operation. Indicates intrinsic interaction characteristics, This represents the learnable parameters in the scoring head network. This represents the Sigmoid function. This represents the intrinsic consistency score.

[0149] Example: Text perturbation mapping matrix Image perturbation mapping matrix Text low-rank projection matrix Image low-rank matrix projection matrix Disturbance coefficient and the learnable parameters in the scoring head network All of these are learnable parameters. In PyTorch, the code implementation described above is as follows:

[0150] import torch

[0151] from torch import nn

[0152] import torch.nn.functional as F

[0153] class IntraConsistencyVerificationModule(nn.Module):

[0154] def __init__(self, embed_dim: int = 512, num_heads: int = 8,

[0155] low_rank: int = 256, hidden_dim: int = 512):

[0156] super().__init__()

[0157] # S20301: Generation of Text Perturbation View and Image Perturbation View

[0158] self.text_perturb_proj = nn.Linear(embed_dim, embed_dim)

[0159] self.image_perturb_proj = nn.Linear(embed_dim, embed_dim)

[0160] self.alpha = nn.Parameter(torch.tensor(0.1))

[0161] # S20302: Generation of Original Interaction Features and Perturbation Response Features

[0162] self.cross_attn = nn.MultiheadAttention(embed_dim, num_heads,batch_first=True)

[0163] # S20303: Bilinear Matching Feature Generation

[0164] self.text_low_rank_proj = nn.Linear(embed_dim, low_rank)

[0165] self.image_low_rank_proj = nn.Linear(embed_dim, low_rank)

[0166] # S20304: Construction of Intrinsic Interaction Features and Consistency Scoring

[0167] intra_dim = embed_dim * 2 + low_rank

[0168] self.score_head = nn.Sequential(

[0169] nn.Linear(intra_dim, hidden_dim),

[0170] nn.ReLU(),

[0171] nn.Dropout(0.1),

[0172] nn.Linear(hidden_dim, 1) )

[0174] def cross_attention(self, q, k):

[0175] h, _ = self.cross_attn(q.unsqueeze(0).unsqueeze(0),

[0176] k.unsqueeze(0).unsqueeze(0),

[0177] k.unsqueeze(0).unsqueeze(0))

[0178] return h.squeeze(0).squeeze(0)

[0179] def forward(self, f_T, f_I):

[0180] # S20301: Generation of Text Perturbation View and Image Perturbation View

[0181] m_T = F.gumbel_softmax(self.text_perturb_proj(f_T), tau=0.5,hard=False)

[0182] m_I = F.gumbel_softmax(self.image_perturb_proj(f_I), tau=0.5,hard=False)

[0183] tilde_T = f_T + self.alpha * (m_T * f_T)

[0184] tilde_I = f_I + self.alpha * (m_I * f_I)

[0185] # S20302: Generation of Original Interaction Features and Perturbation Response Features

[0186] h_orig = self.cross_attention(f_T, f_I)

[0187] h_pert = self.cross_attention(tilde_T, tilde_I)

[0188] r = h_orig - h_pert

[0189] # S20303: Bilinear Matching Feature Generation

[0190] b = F.relu(self.text_low_rank_proj(f_T)) * F.relu(self.image_low_rank_proj(f_I))

[0191] # S20304: Construction of Intrinsic Interaction Features and Consistency Scoring

[0192] h_intra = torch.cat([h_orig, r, b], dim=-1)

[0193] s_intra = torch.sigmoid(self.score_head(h_intra)).squeeze(-1)

[0194] return h_intra, s_intra

[0195] intra_module = IntraConsistencyVerificationModule(embed_dim=512, num_heads=8,

[0196] low_rank=256,hidden_dim=512).to(device)

[0197] h_intra, s_intra = intra_module(f_T, f_I);

[0198] Where f_T represents text features, f_I represents image features, h_intra represents intrinsic interaction features, and s_intra represents consistency score.

[0199] Example 3:

[0200] External certificate consistency verification submodule, such as Figure 6 As shown, the external evidence consistency verification submodule receives news text data, news image data, text features, and image features. First, it inputs the news text data and news image data into a large language model to obtain text-perspective query statements and image-perspective query statements, respectively. Then, it processes the two query statements through a Tavily retrieval and encoding network to obtain text-perspective evidence features and image-perspective evidence features, respectively. Simultaneously, it connects and projects the text features and image features to obtain a news representation. Then, it connects the news representation with the text-perspective evidence features and image-perspective evidence features, respectively, and processes them through positional discrimination projection and the Tanh function to obtain text-perspective positional features and image-perspective positional features. Next, it performs element-wise multiplication of the text-perspective positional features with the text-perspective evidence features and the image-perspective positional features with the image-perspective evidence features, and adds the two multiplication results element-wise to obtain the external evidence interaction features. Finally, it maps the external evidence interaction features through a scoring head network and a Sigmoid function to obtain an external evidence consistency score, and then sends the external evidence interaction features and external evidence consistency score to the fusion classification module.

[0201] Specifically, the implementation process of this external certificate consistency verification submodule is as follows:

[0202] S20401. Generation of text-based and image-based evidence features:

[0203] News text data is input into a large language model for processing to obtain text-perspective query statements. These queries are then processed using Tavily search to obtain text-perspective retrieval results. These results are then encoded using an encoding network to obtain text-perspective evidence features. The specific implementation is illustrated in the following formula:

[0204] ;

[0205] ;

[0206] in, Represents news text data, Representing a large language model, This indicates a query statement from a textual perspective. This indicates a Tavily search. Indicates the number of search results. This represents an encoding network that concatenates and uniformly encodes the k retrieved text fragments. Indicates the characteristics of evidence from a textual perspective;

[0207] News image data is input into a large language model for processing to obtain image perspective query statements; the image perspective query statements are then used to obtain image perspective retrieval results through Tavily search, and these results are encoded using an encoding network to obtain image perspective evidence features; the specific implementation is shown in the following formula:

[0208] ;

[0209] ;

[0210] in, Represents news image data, Representing a large language model, This represents a query statement indicating the viewpoint of an image. This indicates a Tavily search. Indicates the number of search results. This represents an encoding network that concatenates and uniformly encodes the k retrieved text fragments. This indicates the characteristics of image perspective evidence.

[0211] S20402, News Display Generation:

[0212] The text features and image features are concatenated, and the concatenation result is mapped using a news representation projection matrix to obtain the news representation; the specific implementation is shown in the following formula:

[0213] ;

[0214] in, Representing text features, Representing image features, Indicates a join operation. The news represents the projection matrix. This indicates a news report.

[0215] S20403. Generation of textual perspective stance features and image perspective stance features:

[0216] The news representation is concatenated with text-based evidence features, and the concatenation result is mapped using a position-discrimination projection matrix and a Tanh function to obtain text-based position features. Similarly, the news representation is concatenated with image-based evidence features, and the concatenation result is mapped using a position-discrimination projection matrix and a Tanh function to obtain image-based position features. The specific implementation is shown in the following formula:

[0217] ;

[0218] ;

[0219] in, The news indicates that it originated from S20402. This represents textual evidence features, derived from S20401. This represents the features of the image perspective evidence, derived from S20401. Represents the position discrimination projection matrix. This represents the Tanh function. Indicates the perspective and stance of the text. Indicates the perspective and positional characteristics of an image;

[0220] S20404, Construction of External Evidence Interaction Features and Consistency Scoring:

[0221] The textual perspective stance features and textual perspective evidence features are multiplied element-wise, and the image perspective stance features and image perspective evidence features are multiplied element-wise. The results of these two multiplications are then summed element-wise to obtain the external evidence interaction features. These external evidence interaction features are then mapped using a scoring head network and a sigmoid function to obtain the external evidence consistency score. The specific implementation is shown in the following formula:

[0222] ;

[0223] ;

[0224] in, This indicates the text's perspective and stance characteristics, derived from S20403. This represents the image's perspective and position characteristics, derived from S20403. This represents textual evidence features, derived from S20401. This represents the features of the image perspective evidence, derived from S20401. This indicates element-wise multiplication. Indicates the interaction characteristics of external evidence. This represents the learnable parameters in the scoring head network. This represents the Sigmoid function. This indicates the consistency score of the external certificate.

[0225] For example: The large language model used in this invention is GPT-4o, and the news representation projection matrix... Position discrimination projection matrix and the learnable parameters in the scoring head network All of these are learnable parameters. In PyTorch, the code implementation described above is as follows:

[0226] import torch

[0227] from torch import nn

[0228] import torch.nn.functional as F

[0229] from tavily import TavilyClient

[0230] class InterConsistencyVerificationModule(nn.Module):

[0231] def __init__(self, tavily_api_key, llm, evidence_encoder,

[0232] embed_dim: int = 512, top_k: int = 5, hidden_dim:int = 512):

[0233] super().__init__()

[0234] # S20401: Generation of Textual and Image-Based Evidence Features

[0235] self.tavily_client = TavilyClient(api_key=tavily_api_key)

[0236] self.llm = llm

[0237] self.evidence_encoder = evidence_encoder

[0238] self.top_k = top_k

[0239] # S20402: News Display Generation

[0240] self.W_N = nn.Linear(embed_dim * 2, embed_dim)

[0241] # S20403: Generation of textual and image-based perspective stance features

[0242] self.W_S = nn.Linear(embed_dim * 2, 1)

[0243] # S20404: Construction of External Evidence Interaction Features and Consistency Scoring

[0244] self.score_head = nn.Sequential(

[0245] nn.Linear(embed_dim, hidden_dim),

[0246] nn.ReLU(),

[0247] nn.Dropout(0.1),

[0248] nn.Linear(hidden_dim, 1) )

[0250] def retrieve_view(self, news_input):

[0251] q = self.llm(news_input)

[0252] res = self.tavily_client.search(query=q, max_results=self.top_k)

[0253] ev_text = " ".join([r['title'] + '. ' + r['content'] for r inres['results']])

[0254] f_E = self.evidence_encoder(ev_text) # [d]

[0255] return f_E

[0256] def forward(self, news_text, news_image, f_T, f_I):

[0257] # S20401: Generation of Textual and Image-Based Evidence Features

[0258] f_E_T = self.retrieve_view(news_text) # [d]

[0259] f_E_I = self.retrieve_view(news_image) # [d]

[0260] # S20402: News Display Generation

[0261] f_N = self.W_N(torch.cat([f_T, f_I], dim=-1)) # [d]

[0262] # S20403: Generation of textual and image-based perspective stance features

[0263] p_T = torch.tanh(self.W_S(torch.cat([f_N, f_E_T], dim=-1))).squeeze(-1)

[0264] p_I = torch.tanh(self.W_S(torch.cat([f_N, f_E_I], dim=-1))).squeeze(-1)

[0265] # S20404: Construction of External Evidence Interaction Features and Consistency Scoring

[0266] h_inter = p_T * f_E_T + p_I * f_E_I # [d]

[0267] s_inter = torch.sigmoid(self.score_head(h_inter)).squeeze(-1)

[0268] return h_inter, s_inter

[0269] inter_module = InterConsistencyVerificationModule(tavily_api_key="YOUR_KEY",

[0270] llm=llm_model,

[0271] evidence_encoder=clip_text_encoder,

[0272] embed_dim=512, top_k=5,

[0273] hidden_dim=512).to(device)

[0274] h_inter, s_inter = inter_module(news_text, news_image, f_T, f_I);

[0275] Wherein, news_text represents news text data, news_image represents news image data, f_T represents text features, f_I represents image features, h_inter represents foreign evidence interaction features, and s_inter represents foreign evidence consistency score.

[0276] Example 4:

[0277] As attached Figure 1 As shown, this invention provides a method for detecting fake news based on a dual consistency verification module. The method includes the following steps:

[0278] S1. Obtain the fake news detection dataset: Download the publicly available fake news detection dataset from the internet;

[0279] S2. Constructing a fake news detection model: Constructing a fake news detection model based on a dual consistency verification module;

[0280] S3. Train the fake news detection model: Train the fake news detection model constructed in step S2 on the fake news detection dataset obtained in step S1.

[0281] S1. Obtain the fake news detection dataset:

[0282] Download publicly available fake news detection datasets from the internet.

[0283] For example, there are many publicly available fake news detection datasets online, such as the Weibo dataset. The Weibo dataset is based on real and fake news collected from the Sina Weibo platform, containing thousands of news samples with paired images and text, covering various topics such as society, breaking news, health information, and entertainment gossip. The fake news samples have been verified by Sina Weibo's official debunking system, while the real news samples come from reports published by authoritative media. Each sample includes news text, a news image, and corresponding real / fake labels. The Weibo dataset is frequently used in the development of multimodal fake news detection algorithms. The Weibo dataset contains the following types of data and information:

[0284] News text data: Each sample contains a piece of news text describing the specific event content;

[0285] News image data: Images that accompany the news text, which may include on-site photos, photos of people, illustrations, etc.

[0286] Real / Fake Labels: Binary labels (real news / fake news) corresponding to each news item, used to train and validate the fake news detection model.

[0287] S2. Construct a fake news detection model:

[0288] The process of building a fake news detection model is as follows: Figure 2 As shown, the main operations are to build an input module, a dual encoding module, a dual consistency verification module, and a fusion classification module.

[0289] S201. Constructing the input module:

[0290] Extract one data point from the fake news detection dataset obtained in step S1. This data point contains a news text, a news image, and a corresponding true / false label, thus obtaining one input data point, denoted as . ,in Represents news text data, Represents news image data, The labels indicate whether the news is real or fake; 1 represents real news and 0 represents fake news.

[0291] S202, Constructing a dual-encoding module:

[0292] The dual-encoding module uses a pre-trained CLIP model as the backbone network to encode the news text data and news image data in the input data constructed by S201, thereby obtaining text features and image features, denoted as... and The specific implementation is shown in the following formula:

[0293] ;

[0294] ;

[0295] in, This indicates news text data, sourced from S201. This indicates news image data, sourced from S201. The text encoder representing CLIP. This refers to the CLIP image encoder.

[0296] For example, in the implementation of this invention, the dual-encoding module uses a pre-trained CLIP-ViT-B / 32 model, freezes the backbone, and only fine-tunes the last projection layer. In PyTorch, the code implementation described above is as follows:

[0297] import torch

[0298] from torch import nn

[0299] import clip

[0300] class DualModalEncoder(nn.Module):

[0301] def __init__(self, clip_model_name="ViT-B / 32", embed_dim=512, freeze_backbone=True):

[0302] super(DualModalEncoder, self).__init__()

[0303] # Load the pre-trained CLIP model

[0304] self.clip_model, self.preprocess = clip.load(clip_model_name, device='cuda')

[0305] # Freeze CLIP backbone network

[0306] if freeze_backbone:

[0307] for param in self.clip_model.parameters():

[0308] param.requires_grad = False

[0309] # Learnable text projection layer

[0310] self.text_proj = nn.Linear(embed_dim, embed_dim)

[0311] # Learnable image projection layer

[0312] self.image_proj = nn.Linear(embed_dim, embed_dim)

[0313] def forward(self, text_tokens, image_tensor):

[0314] # Text Encoding

[0315] with torch.no_grad():

[0316] text_raw = self.clip_model.encode_text(text_tokens).float()

[0317] image_raw = self.clip_model.encode_image(image_tensor).float()

[0318] # Projected into a unified space

[0319] f_T = self.text_proj(text_raw)

[0320] f_I = self.image_proj(image_raw)

[0321] return f_T, f_I

[0322] dual_encoder = DualModalEncoder(clip_model_name="ViT-B / 32", embed_dim=512).to(device)

[0323] f_T, f_I = dual_encoder(text_tokens, image_tensor);

[0324] Here, text_tokens represents news text data that has been segmented, image_tensor represents news image data that has been preprocessed, f_T represents text features, f_I represents image features, and dual_encoder represents the dual encoding module.

[0325] S203. Construct a dual consistency verification module:

[0326] The dual consistency verification module receives news text data, news image data, text features, and image features, and processes them through the internal consistency verification submodule and the external consistency verification submodule, respectively, to obtain internal interaction features, internal consistency scores, external interaction features, and external consistency scores. The specific implementation process of the internal consistency verification submodule has been described in detail in Example 2, and the specific implementation process of the external consistency verification submodule has been described in detail in Example 3.

[0327] S204. Constructing the fusion classification module:

[0328] The fusion classification module receives the internal interaction features, internal consistency scores, external evidence interaction features, and external evidence consistency scores output by the S203 dual consistency verification module. First, it performs dual-source score fusion mapping processing on the internal consistency scores and external evidence consistency scores to obtain the dual-source consistency score. The specific implementation is shown in the following formula:

[0329] ;

[0330] in, Indicates the intrinsic consistency score. This indicates the score for consistency with external documentation. This represents the dual-source consistency fusion matrix. This represents the Sigmoid function. Indicates the two-source consistency score;

[0331] Then, a dual-source feature fusion process is performed on the intrinsic interaction features and the extrinsic interaction features to obtain the dual-source interaction features; the specific implementation is shown in the following formula:

[0332] ;

[0333] in, Indicates intrinsic interaction characteristics, Indicates the interaction characteristics of external evidence. Indicates a join operation. This represents a dual-source interactive fusion matrix. Indicates dual-source interaction characteristics;

[0334] Finally, the two-source consistency score and the two-source interaction features are multiplied by a scalar-vector multiplication, and the result is mapped through a classification head network and a sigmoid function to obtain the probability that the news data belongs to fake news, denoted as . The specific implementation is shown in the following formula:

[0335] ;

[0336] in, Indicates the two-source consistency score. Indicates dual-source interaction characteristics, Represents the classification head network, This represents the Sigmoid function. This indicates the probability that the news item is fake news.

[0337] Example: Dual-source consistency fusion matrix Dual-source interactive fusion matrix Furthermore, the learnable parameters in the classification head network are all learnable linear transformation matrices. In PyTorch, the code implementation described above is as follows:

[0338] import torch

[0339] from torch import nn

[0340] class FusionClassifier(nn.Module):

[0341] def __init__(self, intra_dim: int, inter_dim: int,

[0342] fusion_dim: int = 512, hidden_dim: int = 512):

[0343] super().__init__()

[0344] # Dual-source consistency fusion matrix

[0345] self.W_s2 = nn.Linear(2, 1)

[0346] # Dual-source interactive fusion matrix

[0347] self.W_h = nn.Linear(intra_dim + inter_dim, fusion_dim)

[0348] # Classification Head Network

[0349] self.cls_head = nn.Sequential(

[0350] nn.Linear(fusion_dim, hidden_dim),

[0351] nn.ReLU(),

[0352] nn.Dropout(0.1),

[0353] nn.Linear(hidden_dim, hidden_dim / / 2),

[0354] nn.ReLU(),

[0355] nn.Linear(hidden_dim / / 2, 1) )

[0357] self.sigmoid = nn.Sigmoid()

[0358] def forward(self, h_intra, h_inter, s_intra, s_inter):

[0359] # Dual-source consistency score

[0360] s_concat = torch.stack([s_intra, s_inter], dim=-1) # [2]

[0361] s_fusion = torch.sigmoid(self.W_s(s_concat)).squeeze(-1)

[0362] # Dual-source interaction features

[0363] h_concat = torch.cat([h_intra, h_inter], dim=-1)

[0364] h_fusion = self.W_h(h_concat)

[0365] # Scalar-Vector Multiplication and Classification

[0366] weighted = s_fusion * h_fusion

[0367] logits = self.cls_head(weighted)

[0368] y_hat = self.sigmoid(logits).squeeze(-1)

[0369] return y_hat

[0370] fusion_cls = FusionClassifier(intra_dim=h_intra.shape[-1],

[0371] inter_dim=h_inter.shape[-1],

[0372] fusion_dim=512,

[0373] hidden_dim=512).to(device)

[0374] y_hat = fusion_cls(h_intra, h_inter, s_intra, s_inter);

[0375] Where h_intra represents the internal interaction feature, h_inter represents the external interaction feature, s_intra represents the internal consistency score, s_inter represents the external consistency score, and y_hat represents the probability that the news data belongs to fake news.

[0376] S3. Training the fake news detection model:

[0377] The fake news detection model constructed in step S2 is trained on the fake news detection dataset obtained in step S1. The process is as follows: Figure 3 As shown:

[0378] S301. Construct the classification loss function:

[0379] This invention uses binary cross-entropy as the classification loss function, as shown in the following formula:

[0380] ;

[0381] in, For genuine and fake labels, This represents the false prediction probability output by the model.

[0382] For example, in PyTorch, the code implementation described above would be as follows:

[0383] # Calculate the error between the predicted value and the label using the binary cross-entropy loss function.

[0384] loss_fct_cls = nn.BCELoss()

[0385] loss_cls = loss_fct_cls(y_hat, y_true.float());

[0386] Where y_true represents the true / false label, and y_hat represents the false prediction probability output by the model.

[0387] S302. Construct the intrinsic consistency loss function:

[0388] This invention employs binary cross-entropy as the intrinsic consistency loss function and introduces an intrinsic contrast regularization term with random permutation within a batch to enhance the discriminative power of the intrinsic consistency score. The formula is shown below:

[0389] ;

[0390] ;

[0391] in, For genuine and fake labels, The intrinsic consistency score, The batch size, This is a random permutation function within a batch. This means combining the text of the i-th sample with the text of the permuted sample. The intrinsic consistency score is calculated after pairing the images of the samples.

[0392] For example, in PyTorch, the code implementation described above would be as follows:

[0393] # Intrinsic consistency loss

[0394] loss_fct_intra = nn.BCELoss()

[0395] loss_intra = loss_fct_intra(s_intra, y_true.float())

[0396] # Hard negative sample comparison regularization

[0397] B = f_T.shape[0]

[0398] perm = torch.randperm(B).to(device)

[0399] f_I_perm = f_I[perm]

[0400] # Recalculate the intrinsic consistency score (using the permuted image features)

[0401] s_intra_neg = compute_intra_score(f_T, f_I_perm) # Call the intrinsic consistency verification submodule

[0402] loss_intra_cont = -torch.log(1 - s_intra_neg + 1e-8).mean();

[0403] Where s_intra is the intrinsic consistency score, y_true is the true / false label, and s_intra_neg is the intrinsic consistency score of the hard negative sample.

[0404] S303. Constructing the external consistency loss function:

[0405] This invention uses binary cross-entropy as the external consistency loss function, as shown in the following formula:

[0406] ;

[0407] in, For genuine and fake labels, The score represents the consistency score between external evidence and external evidence.

[0408] For example, in PyTorch, the code implementation described above would be as follows:

[0409] # Loss of consistency with external evidence

[0410] loss_fct_inter = nn.BCELoss()

[0411] loss_inter = loss_fct_inter(s_inter, y_true.float());

[0412] Where s_inter is the external consistency score, and y_true is the true / false label.

[0413] The total loss function is obtained by weighted summing of the classification loss, intrinsic consistency loss, intrinsic contrast regularization term, and extrinsic consistency loss, as shown in the following formula:

[0414] ;

[0415] in, , , This is the loss weighting balancing coefficient.

[0416] For example, in PyTorch, the code implementation described above would be as follows:

[0417] # Weighted sum of total losses

[0418] lambda_1, lambda_2, lambda_3 = 0.5, 0.3, 0.5

[0419] loss_total = loss_cls + lambda_1 * loss_intra + lambda_2 * loss_intra_cont + lambda_3 * loss_inter

[0420] loss_total.backward();

[0421] Where loss_cls is the classification loss, loss_intra is the intrinsic consistency loss, loss_intra_cont is the intrinsic contrast regularization loss, loss_inter is the extrinsic consistency loss, and loss_total is the weighted total loss.

[0422] S304. Construct the optimization function:

[0423] After testing various optimization functions, the model was finally selected to use the Adam optimization function. Except for its learning rate, which was set to 3e-5, all other hyperparameters of Adam were set to their default values ​​in PyTorch.

[0424] For example, in PyTorch, the code implementation described above would be as follows:

[0425] # Optimize model parameters using the Adam optimizer

[0426] optimizer = torch.optim.Adam(optimizer_grouped_parameters, lr=3e-5)

[0427] optimizer.zero_grad()

[0428] loss_total.backward()

[0429] optimizer.step();

[0430] Here, optimizer_grouped_parameters are the parameters to be optimized, which by default are all trainable parameters in the fake news detection method.

[0431] Once the model is trained, predictions can be performed on the test set. By inputting news text and images into the model, the probability that the news is fake news can be obtained, and the authenticity of the news can be determined accordingly.

[0432] Example 5:

[0433] This device mainly consists of three units: a fake news detection dataset acquisition unit, a fake news detection model construction unit, and a fake news detection model training unit. Its process is as follows: Figure 4 As shown, the specific functions of each unit are as follows:

[0434] The fake news detection dataset acquisition unit is used to download publicly available fake news detection datasets from the internet.

[0435] The fake news detection model building unit is used to build an input module, a dual encoding module, a dual consistency verification module, and a fusion classification module, thereby building a fake news detection model.

[0436] The fake news detection model training unit is used to construct classification loss functions, intrinsic consistency loss functions, extrinsic consistency loss functions, and optimization functions, thereby completing the training and prediction of the model.

[0437] Furthermore, the fake news detection model building unit also includes:

[0438] The input module unit is responsible for constructing the news text data, news image data, and true / false labels required for model training, thereby obtaining the input data.

[0439] A dual-encoding module unit is constructed to encode news text and news images, thereby obtaining text features and image features;

[0440] A dual consistency verification module is constructed, which is responsible for processing news text data, news image data, text features and image features through the internal consistency verification submodule and the external consistency verification submodule, thereby obtaining internal interaction features, internal consistency scores, external interaction features and external consistency scores;

[0441] A fusion classification module is constructed to receive internal interaction features, internal consistency scores, external evidence interaction features, and external evidence consistency scores. It performs feature concatenation and mapping on these features to obtain the probability that the news data belongs to fake news, and then feeds the prediction result back to relevant personnel.

[0442] Furthermore, the fake news detection model training unit also includes:

[0443] Construct a classification loss function unit, which is responsible for using the binary cross-entropy loss function to calculate the error between the predicted false probability and the true label;

[0444] Construct an intrinsic consistency loss function unit, which is responsible for calculating the error between the intrinsic consistency score and the true label using the binary cross-entropy loss function, and using an intrinsic contrast regularization term with random permutation within the batch to enhance the discriminative ability of the intrinsic consistency score;

[0445] Construct an external consistency loss function unit, which is responsible for calculating the error between the external consistency score and the true label using the binary cross-entropy loss function;

[0446] An optimization function unit is constructed to adjust the parameters during model training and reduce the total loss error obtained by weighted summation of classification loss, intrinsic consistency loss, intrinsic contrast regularization term and extrinsic consistency loss.

[0447] Example 6:

[0448] This disclosure provides an electronic device, including: a memory and at least one processor;

[0449] The memory contains computer programs;

[0450] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the fake news detection method based on the double consistency verification module as described above.

[0451] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0452] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0453] Example 6:

[0454] This disclosure provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the fake news detection method based on a dual consistency verification module as described above. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer (or CPU or MPU) of the system or apparatus to read and execute the program code stored in the storage medium.

[0455] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0456] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD+ROM, CD+R, CD+RW, DVD+ROM, DVD+RYM, DVD+RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0457] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0458] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting fake news based on a dual-consistency verification module, characterized in that: The method includes the following steps: S1. Obtain the fake news detection dataset: Download the publicly available fake news detection dataset from the internet; S2. Constructing a fake news detection model: Constructing a fake news detection model based on a dual consistency verification module; S3. Train the fake news detection model: Train the fake news detection model constructed in step S2 on the fake news detection dataset obtained in step S1.

2. The method for detecting fake news based on dual-consistency verification module according to claim 1, characterized in that: Step S2 is used to construct an input module, a dual encoding module, a dual consistency verification module, and a fusion classification module, thereby constructing a fake news detection model; the aforementioned modules correspond to sub-steps, denoted as S201, S202, S203, and S204, respectively.

3. The fake news detection method based on a dual consistency verification module according to claim 2, characterized in that: The S201 constructs an input module, extracts a piece of data from the fake news detection data set obtained in the S1 step, wherein the data contains a piece of news text, a news image, and a corresponding true or false label, thereby obtaining a piece of input data, denoted as wherein represents news text data, represents news image data, represents a true or false label, that is, 1 represents real news and 0 represents fake news. The S202 constructs a double coding module, adopts a pre-trained CLIP model as a backbone network, and respectively encodes news text data and news image data in input data constructed by the S201, so as to obtain text features and image features, denoted as and ; specific implementation is shown in the following formula: ； ； in, This indicates news text data, sourced from S201. This indicates news image data, sourced from S201. The text encoder representing CLIP. This refers to the CLIP image encoder.

4. The method for detecting fake news based on a dual consistency verification module according to claim 2, characterized in that: S203 constructs a dual consistency verification module, which includes an internal consistency verification submodule and an external consistency verification submodule; wherein, the internal consistency verification submodule receives text features. and image features The semantic consistency of the text and images within the news is verified to obtain the intrinsic interaction features. Consistency score ; Specifically, the implementation process of the intrinsic consistency verification submodule is as follows: S20301, Generation of text perturbation view and image perturbation view: Text features are processed using a text perturbation mapping method, and the processing result is then mapped using the Gumbel-Softmax function to obtain a text semantic mask. The text semantic mask is then multiplied element-wise by the text features, and the result is multiplied by a perturbation coefficient and added element-wise to the text features to obtain a text perturbation view. The specific implementation is shown in the following formula: ； ； in, Representing text features, GumbelSoftmax represents the learnable parameters in the text perturbation map, and GumbelSoftmax represents the Gumbel-Softmax function. Represents a semantic mask for text. This represents the disturbance coefficient. This indicates element-wise multiplication. This represents a text perturbation view; Image features are processed using image perturbation mapping, and the processing result is mapped using the Gumbel-Softmax function to obtain an image semantic mask. The image semantic mask is then multiplied element-wise by the image features, and the result is multiplied by a perturbation coefficient and added element-wise to the image features to obtain a perturbed image view. The specific implementation is shown in the following formula: ； ； in, Representing image features, GumbelSoftmax represents the learnable parameters in the image perturbation map, and GumbelSoftmax represents the Gumbel-Softmax function. Represents an image semantic mask. This represents the disturbance coefficient. This indicates element-wise multiplication. Indicates a perturbed view of the image; S20302. Generation of original interaction features and disturbance response features: Using text features as queries and image features as keys, a cross-attention mechanism is used to obtain the original interaction features. Using a text-perturbed view as a query and an image-perturbed view as a key, the same cross-attention mechanism is used to obtain the perturbed interaction features. The perturbed response features are obtained by subtracting the original interaction features from the perturbed interaction features element-wise. See the following formula for details: ； ； ； in, Representing text features, Representing image features, This represents a text perturbation view. This represents the perturbation view of the image, all from S20301. CrossAttn represents the cross-attention mechanism. Represents the original interaction features. This indicates the characteristics of perturbation interaction. Indicates the characteristics of the disturbance response; S20303, Bilinear Matching Feature Generation: Text features are processed using a low-rank text projection matrix, and the result is mapped using the ReLU function to obtain the low-rank text projection result. Image features are processed using an image low-rank matrix projection, and the result is mapped using the ReLU function to obtain the low-rank image projection result. The low-rank text projection result and the low-rank image projection result are then multiplied element-wise to obtain the bilinear matching feature. See the following formula for details: ； in, Representing text features, Representing image features, This represents the learnable parameters in the low-rank projection matrix of the text. This represents the learnable parameters in the low-rank matrix projection of the image. Represents the ReLU function. This indicates element-wise multiplication. This represents the bilinear matching feature; S20304, Construction of Intrinsic Interaction Features and Consistency Scoring: The original interaction features, perturbation response features, and bilinear matching features are concatenated to obtain the intrinsic interaction features. These intrinsic interaction features are then mapped using a scoring head network and a sigmoid function to obtain the intrinsic consistency score. The specific implementation is shown in the following formula: ； ； in, This represents the original interaction features, derived from S20302. This indicates the disturbance response characteristics, originating from S20302. This indicates a bilinear matching feature, derived from S20303. Indicates a join operation. Indicates intrinsic interaction characteristics, This represents the learnable parameters in the scoring head network. This represents the Sigmoid function. This represents the intrinsic consistency score.

5. The fake news detection method based on a dual consistency verification module according to claim 4, characterized in that: S203 constructs a dual consistency verification module, which includes an internal consistency verification submodule and an external consistency verification submodule; wherein, the external consistency verification submodule receives news text data. News image data Text features and image features The consistency between news content and external search evidence is verified to obtain the external evidence interaction characteristics. Consistency score with foreign evidence ; Specifically, the implementation process of the external certificate consistency verification submodule is as follows: S20401. Generation of text-based and image-based evidence features: News text data is input into a large language model for processing to obtain text-perspective query statements. These queries are then processed using Tavily to obtain text-perspective search results, which are further encoded using an encoding network to obtain text-perspective evidence features. The specific implementation is illustrated in the following formula: ； ； in, Represents news text data, Representing a large language model, This indicates a query statement from a textual perspective. This indicates a Tavily search. Indicates the number of search results. This represents an encoding network that concatenates and uniformly encodes the k retrieved text fragments. Indicates the characteristics of evidence from a textual perspective; News image data is input into a large language model for processing to obtain image perspective query statements; the image perspective query statements are then used to obtain image perspective retrieval results through Tavily search, and these results are encoded using an encoding network to obtain image perspective evidence features; the specific implementation is shown in the following formula: ； ； in, Represents news image data, Representing a large language model, This represents a query statement indicating the viewpoint of an image. This indicates a Tavily search. Indicates the number of search results. This represents an encoding network that concatenates and uniformly encodes the k retrieved text fragments. Indicates the characteristics of image perspective evidence; S20402, News Display Generation: The text features and image features are concatenated, and the concatenation result is mapped using a news representation projection matrix to obtain the news representation; the specific implementation is shown in the following formula: ； in, Representing text features, Representing image features, Indicates a join operation. The news represents the projection matrix. This indicates news reporting; S20403. Generation of textual perspective stance features and image perspective stance features: The news representation is concatenated with text-based evidence features, and the concatenation result is mapped using a position-discrimination projection matrix and a Tanh function to obtain text-based position features. Similarly, the news representation is concatenated with image-based evidence features, and the concatenation result is mapped using a position-discrimination projection matrix and a Tanh function to obtain image-based position features. The specific implementation is shown in the following formula: ； ； in, The news indicates that it originated from S20402. This represents textual evidence features, derived from S20401. This represents the features of the image perspective evidence, derived from S20401. Represents the position discrimination projection matrix. This represents the Tanh function. Indicates the perspective and stance of the text. Indicates the perspective and positional characteristics of an image; S20404, Construction of External Evidence Interaction Features and Consistency Scoring: The textual perspective stance features and textual perspective evidence features are multiplied element-wise, and the image perspective stance features and image perspective evidence features are multiplied element-wise. The results of these two multiplications are then summed element-wise to obtain the external evidence interaction features. These external evidence interaction features are then mapped using a scoring head network and a sigmoid function to obtain the external evidence consistency score. The specific implementation is shown in the following formula: ； ； in, This indicates the text's perspective and stance characteristics, derived from S20403. This represents the image's perspective and position characteristics, derived from S20403. This represents textual evidence features, derived from S20401. This represents the features of the image perspective evidence, derived from S20401. This indicates element-wise multiplication. Indicates the interaction characteristics of external evidence. This represents the learnable parameters in the scoring head network. This represents the Sigmoid function. This indicates the consistency score of the external certificate.

6. The fake news detection method based on a dual consistency verification module according to claim 2, characterized in that: S204 constructs a fusion classification module, which receives the intrinsic interaction features output by the dual consistency verification module. Intrinsic consistency score External evidence interaction characteristics Consistency score with foreign evidence The data is then fused and categorized to determine the probability that the news item is fake news. Specifically, the implementation process of the fusion classification module is as follows: First, a dual-source score fusion mapping process is performed on the intrinsic consistency score and the extrinsic consistency score to obtain the dual-source consistency score; the specific implementation is shown in the following formula: ； in, Indicates the intrinsic consistency score. This indicates the consistency score of the external certificate. This represents the dual-source consistency fusion matrix. This represents the Sigmoid function. Indicates the two-source consistency score; Then, a dual-source feature fusion process is performed on the intrinsic interaction features and the extrinsic interaction features to obtain the dual-source interaction features; the specific implementation is shown in the following formula: ； in, Indicates intrinsic interaction characteristics, Indicates the interaction characteristics of external evidence. Indicates a join operation. This represents a dual-source interactive fusion matrix. Indicates dual-source interaction characteristics; Finally, the two-source consistency score and the two-source interaction features are multiplied by a scalar-vector multiplication, and the result is mapped through a classification head network and a sigmoid function to obtain the probability that the news data belongs to fake news, denoted as . The specific implementation is shown in the following formula: ； in, Indicates the two-source consistency score. Indicates dual-source interaction characteristics, Represents the classification head network, This represents the Sigmoid function. This indicates the probability that the news item is fake news.

7. The method for detecting fake news based on a dual consistency verification module according to claim 1, characterized in that: Step S3 is used to construct a classification loss function, an intrinsic consistency loss function, an extrinsic consistency loss function, and an optimization function, thereby training the fake news detection model. The aforementioned modules correspond to sub-steps, denoted as S301, S302, S303, and S304, which are explained in sequence as follows: S301. Construct the classification loss function: This invention uses binary cross-entropy as the classification loss function, as shown in the following formula: ； in, For genuine and fake labels, The false probability of the model's output; S302. Construct the intrinsic consistency loss function: This invention employs binary cross-entropy as the intrinsic consistency loss function and introduces an intrinsic contrast regularization term with random permutations within a batch to enhance the discriminative power of the intrinsic consistency score. The formula is shown below: ； ； in, For genuine and fake labels, The intrinsic consistency score, The batch size, This is a random permutation function within a batch. This means combining the text of the i-th sample with the text of the permuted sample. The intrinsic consistency score is calculated after pairing images of a sample. S303. Constructing the external consistency loss function: This invention uses binary cross-entropy as the external consistency loss function, as shown in the following formula: ； in, For genuine and fake labels, The score is the consistency score with external evidence. The total loss function is obtained by weighted summing of the classification loss, intrinsic consistency loss, intrinsic contrast regularization term, and extrinsic consistency loss, as shown in the following formula: ； in, , , This is the loss weighting balancing coefficient; S304. Construct the optimization function: After testing various optimization functions, the Adam optimization function was finally selected as the optimization function. Except for its learning rate being set to 3e-5, all other hyperparameters of Adam were set to the default values ​​in PyTorch. When the model is trained, predictions can be performed on the test set. By inputting news text elements and news image data into the model, the probability that the news is fake news can be obtained, and the authenticity of the news can be judged accordingly.

8. A fake news detection device based on a dual consistency verification module as described in claim 1, characterized in that: The device includes: The fake news detection dataset acquisition unit is used to download publicly available fake news detection datasets from the internet; The fake news detection model building unit is used to build an input module, a dual encoding module, a dual consistency verification module, and a fusion classification module, thereby building a fake news detection model; The fake news detection model training unit is used to construct classification loss functions, intrinsic consistency loss functions, extrinsic consistency loss functions, and optimization functions, thereby completing the training and prediction of the model.

9. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the fake news detection method based on a dual consistency verification module as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the fake news detection method based on a dual consistency verification module as described in any one of claims 1 to 7.

Images