False news detection system and method of graph attention network based on multi-modal similarity perception

Abstract

The invention belongs to the technical field of false news detection, and discloses a false news detection system and method of a graph attention network based on multi-modal similarity perception, and the system comprises a multi-modal data preprocessing module which is responsible for cleaning and formatting the original input text and image data, and recognizing key entity information from the original input text and image data; the multi-channel feature extraction module is responsible for extracting multi-channel features such as semantic features, emotion features, spatial domain features and frequency domain features from the preprocessed texts and images, and performing preliminary intra-modal feature fusion through a gating mechanism; the multi-modal similarity perception heterogeneous graph construction module is responsible for constructing a heterogeneous graph by utilizing the extracted entity information and the fused modal features; and the false information detection module is responsible for performing information aggregation and feature learning on the constructed heterogeneous graph by using a graph attention network, performing joint optimization through an adaptive weighted loss function in combination with an analysis result of a single-mode detection branch, and finally outputting authenticity prediction of news.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the technical field of fake news detection, and in particular relates to a fake news detection system and method based on a graph attention network with multimodal similarity perception. Background Art

[0002] In the current wave of digitalization, information is produced and disseminated at an unprecedented speed and breadth. Social media platforms, with their immediacy and openness, have dramatically transformed the news ecosystem and the way the public accesses information. However, this has also facilitated the proliferation and spread of fake news. Fake news generally refers to false content deliberately fabricated and disseminated for deceptive purposes, often motivated by economic, political, or social interference. The spread of such news can have widespread and profound negative impacts on society: in public health, it can distort scientific understanding and hinder effective disease prevention and control; in the political arena, it can mislead public opinion and erode the credibility of the democratic process; and it can also exacerbate social conflicts and undermine social trust. Therefore, developing effective fake news detection technologies has become an urgent need to maintain a healthy information ecosystem and safeguard social stability.

[0003] Early research on fake news detection focused primarily on analyzing a single information modality, such as semantic and sentiment analysis based on textual content, or visual feature analysis based on image provenance and traces of manipulation. However, fake news is increasingly being disseminated through multimodal formats, combining text, images, and videos. This exploits the strong appeal of visual information to human cognition, making false content more deceptive and inflammatory, posing a significant challenge to detection methods that rely on single-modality analysis. For example, information manipulators may combine real, unrelated images with misleading text, or maliciously edit and splice video clips. In these cases, simply analyzing any one modality makes it difficult to determine the overall authenticity of the information. Therefore, how to effectively align, correlate, and fuse this heterogeneous information to accurately identify possible inconsistencies or manipulation traces across modalities has become a core challenge in this field.

[0004] Given the inherent limitations of single-modal detection methods, academia and industry have begun actively exploring fake news detection technologies based on multimodal fusion, and have made significant progress. These methods aim to comprehensively utilize information clues provided by different modalities, such as text and images, to make judgments. For example, some methods (such as the "Multimodal False Information Detection Method Based on Semantic Consistency") focus on the content itself, extracting global and local features from text and images and deeply analyzing the semantic consistency between these features to determine whether the information is false. The core of this approach is to capture the internal connections and potential contradictions between multimodal content. Other methods (such as the "Fake News Detection Method that Fusion of Propagation Features and Multimodal Content Features") recognize that the dissemination behavior of news also contains important clues. These methods not only integrate multimodal content features, but also extract news dissemination characteristics by constructing social network maps. They combine content and dissemination information for comprehensive judgment, attempting to examine fake news from a broader perspective.

[0005] However, despite the breakthroughs in multimodal fusion and feature mining, the following technical issues still exist in deep information interaction and the collaborative use of key features:

[0006] (1) Inherent limitations of single-modal detection methods: Although analysis methods targeting a single information modality (such as pure text or pure image) are effective in specific scenarios, when faced with the increasing number of fake news that uses a combination of multiple modalities such as text and images to spread, such methods are unable to effectively utilize the inconsistencies or manipulation traces that may exist in cross-modal information. Therefore, the comprehensiveness and accuracy of their detection are limited.

[0007] (2) Existing mainstream multimodal fusion strategies fail to fully capture the deep correlations and potential conflicts between modalities: Current mainstream fusion strategies, such as simple feature splicing or weighted fusion based on attention mechanism, can integrate features from different modalities. However, when performing cross-modal fusion, they rarely take into full consideration the distribution differences of different modal features in the vector space, and fail to effectively establish fine-grained correspondences of cross-modal semantics. It is difficult to fully capture the deep intrinsic correlations and potential conflicts between modalities, which restricts the effective fusion and utilization of multimodal information and limits the detection performance of the model.

[0008] (3) Insufficient exploration of the potential of specific key information dimensions in fake news detection: Existing research has not fully and systematically explored and verified the discriminative potential of the emotional characteristics contained in text content in fake news detection. At the same time, the application of frequency domain features of image content (such as signal changes caused by compression, editing, or generation processes) in fake news detection is relatively limited, failing to fully utilize these important underlying visual discriminative cues. Summary of the Invention

[0009] To address the problems of the existing technology, the present invention provides a fake news detection system and method based on a multimodal similarity-aware graph attention network. This system aims to address the difficulty of existing technologies in effectively capturing and integrating fine-grained semantic associations between images and text content when processing multimodal fake news, thereby improving the accuracy, robustness, and interpretability of detection. This system is called the Multimodal Similarity-aware Graph Attention Network (MS-GAT). The core of this framework is to construct and process a heterogeneous graph structure that can explicitly capture fine-grained semantic associations between images and text content, and combine it with unimodal analysis based on multi-channel features to form a joint learning framework.

[0010] The present invention is implemented as follows: a fake news detection system based on a multimodal similarity-aware graph attention network, comprising:

[0011] Multimodal data preprocessing module: responsible for cleaning and formatting the original input text and image data, and identifying key entity information from it.

[0012] Multi-channel feature extraction module: responsible for extracting multi-channel features such as semantics, emotions, spatial domains and frequency domains from preprocessed text and images, and performing preliminary intra-modal feature fusion through a gating mechanism.

[0013] Multimodal similarity-aware heterogeneous graph construction module: responsible for using the extracted entity information and fused modal features to construct a heterogeneous graph that can explicitly express the fine-grained semantic associations between multimodal content.

[0014] False information detection module: This module is responsible for using the graph attention network to perform information aggregation and feature learning on the constructed heterogeneous graph, combining the analysis results of the unimodal detection branch, and jointly optimizing them through an adaptive weighted loss function to ultimately output a prediction of the authenticity of the news.

[0015] Furthermore, the multimodal data preprocessing module specifically includes:

[0016] Responsible for cleaning, formatting and entity identification of raw multimodal data.

[0017] For the input text content T, first perform standard text preprocessing such as word segmentation and cleaning, and then apply named entity recognition tools to identify text entities from the preprocessed text. Let the set of identified text entities be E T .

[0018] For the input image content I, standard image preprocessing operations are first performed, including size and pixel normalization. At the same time, optical character recognition is performed to extract the embedded text in the image, and subject recognition is performed to determine the main visual focus and area of ​​the image. Subsequently, the NER tool is applied to the text results extracted by OCR to identify the text entities in the image. Finally, the subject recognition results are combined with the text entities in the image to form the final image entity E. I .

[0019] After this stage, standardized multimodal data is obtained, and the text entity set E identified from it is used to construct the graph structure. T and image entity set E I .

[0020] Furthermore, the multi-channel feature extraction module specifically includes:

[0021] Multi-channel information useful for fake news detection is extracted from text and images and initially fused through a gate control mechanism to provide rich and integrated node features for the subsequent construction of the graph structure.

[0022] For text modalities, this paper extracts semantic and sentiment features. Semantic features are obtained by encoding the original text using a pre-trained Bert-based model, capturing the meaning and context of the text in a high-dimensional semantic space. Sentiment features are obtained using a Bert model fine-tuned on a sentiment annotation dataset, aiming to capture the emotional tendencies expressed in the text. These two features constitute a multi-channel representation of text.

[0023] For image modalities, this paper extracts spatial and frequency domain features. Spatial domain features are obtained by encoding images using a pre-trained ViT-base model, focusing on capturing the visual patterns and content of the image in pixel space. Frequency domain features are obtained by processing the image using a fast Fourier transform (FFT), analyzing the image's characteristics in frequency distribution, aiming to capture compression artifacts or potential editing operations. These features constitute a multi-channel representation of the image.

[0024] In order to effectively integrate the multi-channel features extracted from each modality, a gate control mechanism is introduced for preliminary fusion. A gate control unit based on MLP and Sigmoid activation is designed, which receives semantic features and sentiment features as input, learns to generate dynamic weights to balance the contributions of the two, and outputs a fused text feature vector. Formally, let the semantic feature be T sem , the emotional feature is T senti , the output of the gate unit is g∈[0,1] D (D is the feature dimension), fusion feature Tf It can be expressed as

[0025] T f =g⊙T sem +(1-g)⊙T senti

[0026] A similar gating mechanism is used to fuse the spatial domain features of the image. spa and frequency domain features I freq , get the final fused image feature I f Through this gated fusion, the model is able to adaptively adjust the importance of different feature channels based on the input content. These fused text and image feature vectors are not only used as the initial attributes of the modal hub nodes when constructing the graph structure, but are also directly input into subsequent unimodal detection for reasoning.

[0027] Furthermore, the multimodal similarity-aware heterogeneous graph construction module specifically includes:

[0028] It is responsible for converting multimodal information into a graph structure that can explicitly capture the fine-grained cross-modal connections between text and image content. The constructed graph G = (V, E) will serve as the input of the graph attention mechanism in the subsequent detection model, guiding the propagation and aggregation of information between different nodes.

[0029] First, define the node set V of the graph. Nodes are the basic units of the graph structure and represent key information points in multimodal information. Based on the entity recognition process, we divide the nodes into the following three categories: text entity nodes V T , image entity node V I and the modal hub node V M ={T f ,I f The total number of nodes in the graph |V| = |V T |+|V I The initial feature representations of text entity nodes and image entity nodes are extracted by the pre-trained CLIP model, and the initial feature representation of modal hub nodes comes from the multi-channel feature vector fused in the previous section.

[0030] Next, define the edge set E of the graph. The edge connects the nodes in the graph, reflecting the strength of the association between the nodes, and its weight w ij In GAT, it is used to guide information aggregation. The core of edge construction is to capture multimodal similarity associations. For any two entity nodes i, j∈V T ∪V I , their initial features will be calculated and The cosine similarity between them is used as the basis for measuring their semantic association:

[0031]

[0032] This similarity calculation is performed in a shared embedding space constructed by the CLIP cross-modal pre-trained model, which can effectively measure the multimodal semantic similarity between text entities, image entities, and cross-modal entities.

[0033] Based on the calculated entity similarity, an entity similarity edge is constructed. For any two text entities i, j∈V T , if Similarity(i,j)>θ t , then add an undirected edge between i and j with a weight of w ij =Similarity(i, j). The intramodal edges between image entities are constructed in a similar way, using a threshold θ i Cross-modal entity similarity edges are the key to capturing multimodal associations. They connect entity nodes with high similarity between different modalities. For any text entity i∈V T and image entity j∈V I , if Similarity(i, j)>θ m , then add an undirected edge between i and j with a weight of w ij =Similarity(i, j).

[0034] In addition to the similarity edges between entities, the graph structure also contains entity-modality hub edges that connect entities to modality hubs. This type of edge is designed to aggregate entity information within the modality to the hub node. For each text entity i∈V T , add an undirected edge. For each image entity j∈V I , add an undirected edge. The weights of these edges are w ij Set to a fixed value w m .

[0035] After the construction is completed, the initial features of the node are obtained. Edge index and edge weight W = {w ij |(i, j)∈E}. This graph uses multimodal entities as basic units, builds connections by capturing fine-grained semantic similarity, and strengthens cross-modal information aggregation through modal hubs, providing rich structural and attribute information for subsequent graph neural network processing.

[0036] Furthermore, the false information detection module specifically includes:

[0037] Information processing and multimodal fusion detection are performed on the constructed multimodal similarity-aware heterogeneous graph G, and the model is jointly learned in combination with the single-modal detection branch.

[0038] On the heterogeneous graph G, a multi-layer graph attention network (GAT) is used to learn node representations. The GAT layer uses the attributes of the edges to perform similarity-aware message passing and aggregation, adaptively fusing neighbor information, and obtaining the final representation of all nodes in the graph. Subsequently, the global feature h of the graph is obtained by performing a global pooling operation on these final node embeddings G This feature is input into the multimodal graph detection branch, which outputs a graph-based

[0039] In addition to the detection branch based on graph information, a text detection branch and an image detection branch are also introduced, which run in parallel to capture fake news clues from different perspectives. The text detection branch receives the extracted multi-channel features of the original text and outputs them after processing. The image detection branch receives the extracted multi-channel features of the original image and outputs them after processing

[0040] For the output of each detection branch, the present invention calculates the cross entropy loss between it and the true label y: Total training loss is the weighted sum of these three losses, where the weights are automatically learned by the model. The loss function is defined as:

[0041]

[0042] where λ j is related to each loss term The associated learnable parameter represents the logarithmic variance of the output of the corresponding task. Therefore, exp(-λ j ) as The weight of λ is proportional to the estimated accuracy of the task: j The larger the exp(-λ j ) is smaller, the lower the weight of the loss term. j λ j The term plays a regularization role. By minimizing The model can automatically evaluate and adjust the importance of each branch in joint training, thereby optimizing the overall multimodal fake news detection performance and avoiding the tediousness and limitations of manual parameter adjustment.

[0043] Another object of the present invention is to provide a method for detecting fake news based on a multimodal similarity-aware graph attention network, which uses the fake news detection system based on a multimodal similarity-aware graph attention network, comprising:

[0044] Step 1: Multimodal Data Preprocessing and Entity Recognition

[0045] The text content T is preprocessed and key text entities are identified by NER tools to form a text entity set. The image content I is preprocessed and image entities are identified by OCR and subject recognition technology to form an image entity set.

[0046] Step 2: Multi-channel feature extraction and node initialization

[0047] The initial feature representations of text entities and image entities are encoded using the pre-trained CLIP model. These encoded feature vectors serve as the initial attributes of the corresponding nodes in the heterogeneous graph. At the same time, the overall features of the text modality and the overall features of the image modality are also generated through corresponding multi-channel feature extraction and fusion mechanisms to initialize the text hub node and image hub node.

[0048] Step 3: Multimodal Similarity-Aware Heterogeneous Graph Construction

[0049] Based on the entity features encoded by CLIP, the semantic similarity between cross-modal entities is calculated.

[0050] Step 4: False Information Detection

[0051] The constructed heterogeneous graph is fed into the Graph Attention Network (GAT). GAT uses the edges in the graph structure to pass messages and aggregate node information. Ultimately, based on the learned graph representation that incorporates this inconsistency signal, the classifier in the multimodal graph detection branch can accurately classify news samples as fake news.

[0052] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the fake news detection method based on the multimodal similarity-perceived graph attention network.

[0053] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the fake news detection method based on the multimodal similarity-aware graph attention network.

[0054] Another object of the present invention is to provide an information data processing terminal, which includes the fake news detection system based on the graph attention network with multimodal similarity perception.

[0055] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0056] First, this paper addresses the existing issues in multimodal fake news detection, such as insufficient cross-modal semantic association modeling, insufficient multimodal information fusion, and difficulty in effectively balancing the contributions of various branches in multi-task learning. We design a fake news detection framework based on a multimodal similarity-aware graph attention network (MS-GAT). Its main advantages are as follows:

[0057] (1) Comprehensive and robust multi-channel feature representation and preliminary fusion: This paper extracts multi-channel features from the semantic and emotional dimensions of text and the spatial and frequency domains of images, and performs preliminary fusion of the multi-channel features within each modality through a gating mechanism. This design can capture richer and more detailed single-modal information, providing high-quality and more robust feature input for subsequent cross-modal association modeling and detection, overcoming the limitations of traditional methods that may rely on single or shallow features.

[0058] (2) Explicit and fine-grained cross-modal semantic association modeling: This paper innovatively constructs a heterogeneous graph containing text entities, image entities, and modal hub nodes, and defines edges by calculating the intramodal and cross-modal semantic similarities between entities. On this basis, a graph attention network is used to adaptively aggregate node information. This mechanism can explicitly model and deeply explore the fine-grained semantic associations and potential inconsistencies between text and image content, improving the model's ability to understand complex cross-modal information and the accuracy of detection, and enhancing the model's interpretability, which is superior to the implicit or coarse-grained processing of inter-modal relationships in traditional fusion methods.

[0059] (3) Automated and efficient multi-task joint learning: This paper introduces an adaptive weighted loss function based on learnable parameters to automatically balance the contributions of the multimodal graph detection branch and the independent text and image detection branches. This mechanism can dynamically adjust the weight of each task in the overall loss based on its learning status, effectively avoiding the complexity and potential bias caused by manual adjustment of task weights, and enhancing the training efficiency and generalization ability of the model.

[0060] (4) Significantly Improved Detection Performance and Application Value: Thanks to the aforementioned systematic breakthroughs in information representation, cross-modal correlation analysis, and learning mechanisms, the MS-GAT framework of the present invention has achieved detection performance that surpasses existing representative technologies on multiple public benchmark datasets, with an average F1 score improvement of 1.9% to 15.5%. This not only demonstrates the advancement and effectiveness of its technical approach, but also demonstrates its strong practical capabilities and broad application prospects in addressing the complex and ever-changing problem of false information dissemination in the real world, providing strong technical support for purifying the network environment.

[0061] Second, the MS-GAT framework proposed in this invention has significant commercial potential and social benefits.

[0062] For social media platforms, news portals, and content aggregation platforms, this invention provides an automated, highly accurate, and efficient fake news detection tool. This significantly reduces manual review costs, allows for rapid response, and effectively curbs the spread of false information, thereby improving user experience, enhancing platform credibility, and helping platforms mitigate regulatory risks and brand reputation damage associated with the spread of false information. More accurate detection also means a safer advertising environment, boosting advertiser confidence and platform advertising value.

[0063] For government regulatory agencies and public opinion analysis departments: This technology can serve as a powerful cyberspace governance tool to monitor, identify, and trace the source and dissemination path of false information in real time, providing technical support for maintaining social stability, ensuring public safety, and guiding healthy public opinion.

[0064] For enterprise users: It can be used for brand reputation management and crisis warning, timely discovering malicious false information targeting enterprises or products on the Internet, and preventing the spread of negative impacts.

[0065] For ordinary users: it indirectly improves the overall quality of online information and protects the public from being misled by false information.

[0066] This invention fills the following key technical gaps in the field of multimodal fake news detection:

[0067] This method fills the gap in explicit cross-modal semantic association modeling: Existing technologies mostly use implicit fusion (such as simple feature concatenation or general attention mechanisms), failing to explicitly and fine-grainedly model the semantic associations and potential inconsistencies between text and images. This invention innovatively constructs a heterogeneous graph containing entities and hub nodes, defines edges based on semantic similarity, and then integrates a graph attention network for information aggregation. This is a new graph structure modeling method specifically designed for cross-modal consistency verification in fake news detection, which is lacking in existing technologies at home and abroad.

[0068] This method fills a gap in robust multi-channel feature fusion: Traditional methods often focus on a single feature dimension. This invention is the first to systematically integrate multi-channel features, such as the semantics and sentiment of text, and the spatial and frequency domains of images, and employs a gating mechanism for intra-modal fusion. This provides unprecedentedly rich and robust feature representation for detection, which is a significant addition and advancement in the field of fake news detection.

[0069] Filling the gap in adaptive balancing in multi-task learning: In multi-task fake news detection, how to automatically and efficiently balance the contributions of different detection branches (such as text, image, and fusion) has always been a challenge. This paper introduces an adaptive weighted loss function based on learnable parameters to automate this process, filling the gap in this field for efficient and adaptive multi-task optimization strategies.

[0070] This invention successfully addresses a core and thorny technical challenge in the field of multimodal fake news detection: how to deeply and explainably understand and utilize the complex relationship between text and images to identify deceptive content.

[0071] For a long time, researchers have been eager to solve the following problem: How to go beyond surface fusion and truly dig out subtle clues of "image-text inconsistency" or "image-text misleading"? Fake news makers often use seemingly related but actually deceptive combinations of images and texts. Traditional fusion methods have difficulty effectively identifying this "advanced" disguise. How to improve detection performance while enhancing the interpretability of the model and understanding its judgment basis? Many deep learning methods are like "black boxes" and it is difficult to explain why a piece of information is judged to be false. How to enable the model to automatically weigh the importance of different information sources (text, images, and inter-modal relationships) during the learning process? Manually adjusting weights is time-consuming and labor-intensive and has poor results. The present invention constructs a heterogeneous graph based on semantic similarity and applies a graph attention network (MS-GAT), so that the model can explicitly pay attention to and quantify the strength of the association between different content entities, thereby successfully capturing fine-grained cross-modal inconsistencies, solving the first problem, and to a certain extent improving interpretability through the attention weights of the graph structure, addressing the second problem. At the same time, the introduction of the adaptive weighted loss function effectively and successfully solves the problem of automatic balancing of multi-task weights that has long plagued researchers. Therefore, the present invention has made substantial breakthroughs in solving these technical challenges that have been long sought to be solved but not yet successfully solved.

[0072] Overcoming the "fusion is everything" bias: Traditional thinking might suggest that simply designing a sufficiently powerful fusion module (such as a Transformer or complex attention) will solve multimodal problems. This paper demonstrates that prior to fusion, high-quality, multi-angle unimodal feature extraction (multi-channel features) and the construction of an explicit, targeted cross-modal association structure (MS-GAT) are equally, if not more, crucial. It demonstrates that understanding "how to associate" is more effective than simply "forcing fusion."

[0073] Overcoming the preconception that multi-task learning requires manual parameter tuning: Balancing the weights of multi-task losses has long been considered an art form requiring extensive experimentation and experience. This paper, by introducing adaptive weighted losses, demonstrates that the model can be self-tuning, overcoming the reliance on manual experience and the inefficiency of manual parameter tuning. This advances the use of multi-task learning in fake news detection for a more automated and efficient application. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1This is a structural diagram of a fake news detection system based on a graph attention network with multimodal similarity perception provided by an embodiment of the present invention.

[0075] Figure 2 This is a diagram of the graph attention network structure for multimodal similarity perception provided by an embodiment of the present invention.

[0076] Figure 3 2 is a schematic diagram of a multimodal fake news case analysis provided by an embodiment of the present invention.

[0077] Figure 4 This is a flow chart of a fake news detection method based on a graph attention network with multimodal similarity perception provided by an embodiment of the present invention.

[0078] Figure 5 This is the T-SNE visualization result of the Logits output by the fake news detection method based on the multimodal similarity-aware graph attention network provided by an embodiment of the present invention in the test set samples of two public datasets.

[0079] Figure 6 This is a multimodal similarity-aware heterogeneous graph construction process of a fake news detection method based on a multimodal similarity-aware graph attention network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0081] like Figure 1 As shown in the figure, the MS-GAT framework proposed in this invention mainly consists of the following four core modules:

[0082] (1) Multimodal data preprocessing module: responsible for cleaning and formatting the original input text and image data, and identifying key entity information from them.

[0083] (2) Multi-channel feature extraction module: responsible for extracting multi-channel features such as semantics, emotions, spatial domain and frequency domain from the preprocessed text and images, and performing preliminary intra-modal feature fusion through the gating mechanism.

[0084] (3) Multimodal similarity-aware heterogeneous graph construction module: It is responsible for using the extracted entity information and the fused modal features to construct a heterogeneous graph that can explicitly express the fine-grained semantic associations between multimodal contents.

[0085] (4) False information detection module: It is responsible for using the graph attention network to perform information aggregation and feature learning on the constructed heterogeneous graph, combining the analysis results of the unimodal detection branch, and jointly optimizing through an adaptive weighted loss function to finally output the authenticity prediction of the news.

[0086] The following describes in detail the various modules of the MS-GAT framework and their working methods:

[0087] (1) Multimodal data preprocessing module

[0088] This module is responsible for cleaning, formatting and entity identification of raw multimodal data.

[0089] For the input text content T, first perform standard text preprocessing such as word segmentation and cleaning, and then apply the named entity recognition (NER) tool to identify text entities from the preprocessed text. Let the set of identified text entities be E T .

[0090] For the input image content I, standard image preprocessing operations are first performed, including size and pixel normalization. At the same time, optical character recognition (OCR) is performed to extract the embedded text in the image, and subject recognition is performed to determine the main visual focus and area of ​​the image. Subsequently, the NER tool is applied to the text results extracted by OCR to identify the text entities in the image. Finally, the subject recognition results are combined with the text entities in the image to form the final image entity E. I .

[0091] After this stage, we obtain standardized multimodal data and the text entity set E identified from it for building the graph structure. T and image entity set E I .

[0092] (2) Multi-channel feature extraction module

[0093] This module aims to extract multi-channel information from text and images that is useful for fake news detection, and perform preliminary fusion through a gate control mechanism to provide rich and integrated node features for the subsequent construction of the graph structure.

[0094] For text modalities, this paper extracts semantic and sentiment features. Semantic features are obtained by encoding the original text using a pre-trained Bert-based model, capturing the meaning and context of the text in a high-dimensional semantic space. Sentiment features are obtained using a Bert model fine-tuned on a sentiment annotation dataset, aiming to capture the emotional tendencies expressed in the text. These two features constitute a multi-channel representation of text.

[0095] For image modalities, this paper extracts spatial and frequency domain features. Spatial domain features are obtained by encoding images using a pre-trained ViT-base model, focusing on capturing the visual patterns and content of the image in pixel space. Frequency domain features are obtained by processing the image using a fast Fourier transform (FFT), analyzing the image's characteristics in frequency distribution, aiming to capture compression artifacts or potential editing operations. These features constitute a multi-channel representation of the image.

[0096] In order to effectively integrate the multi-channel features extracted from each modality, the present invention introduces a gate control mechanism for preliminary fusion. Specifically, the present invention designs a gate control unit based on MLP and Sigmoid activation, which receives semantic features and sentiment features as input, learns to generate dynamic weights to balance the contributions of the two, and outputs a fused text feature vector. Formally, let the semantic feature be T sem , the emotional feature is T senti , the output of the gate unit is g∈[0,1] D (D is the feature dimension), fusion feature T f It can be expressed as

[0097] T f =g⊙T sem +(1-g)⊙T senti

[0098] A similar gating mechanism is used to fuse the spatial domain features of the image. spa and frequency domain features I freq , get the final fused image feature I f Through this gated fusion, the model is able to adaptively adjust the importance of different feature channels based on the input content. These fused text and image feature vectors are not only used as the initial attributes of the modal hub nodes when constructing the graph structure, but are also directly input into subsequent unimodal detection for reasoning.

[0099] (3) Multimodal Similarity-Aware Heterogeneous Graph Construction Module

[0100] This module, a core component of the entire framework, transforms multimodal information into a graph structure that explicitly captures fine-grained cross-modal connections between text and image content. The constructed graph G = (V, E) serves as the input to the graph attention mechanism in subsequent detection models, guiding the propagation and aggregation of information across different nodes.

[0101] First, the present invention defines the node set V of the graph. Nodes are the basic units of the graph structure, and they represent key information points in multimodal information. Based on the entity recognition process, we divide the nodes into the following three categories: text entity nodes V T , image entity node V I and the modal hub node V M ={T f , I f The total number of nodes in the graph |V| = |V T |+|V I The initial feature representations of text entity nodes and image entity nodes are extracted by the pre-trained CLIP model, and the initial feature representation of modal hub nodes comes from the multi-channel feature vector fused in the previous section.

[0102] Next, the present invention defines the edge set E of the graph. The edge connects the nodes in the graph, reflecting the strength of the association between the nodes, and its weight w ij In GAT, it is used to guide information aggregation. In this study, the core of edge construction is to capture multimodal similarity associations. For any two entity nodes i, j∈V T ∪V I , their initial features will be calculated and The cosine similarity between them is used as the basis for measuring their semantic association:

[0103]

[0104] This similarity calculation is performed in a shared embedding space constructed by the CLIP cross-modal pre-trained model, which can effectively measure the multimodal semantic similarity between text entities, image entities, and cross-modal entities.

[0105] Based on the calculated entity similarity, the present invention constructs entity similarity edges. T , if Similarity(i,j)>θ t , then add an undirected edge between i and j with a weight of w ij =Similarity(i, j). The intramodal edges between image entities are constructed in a similar way, using a threshold θ iCross-modal entity similarity edges are the key to capturing multimodal associations. They connect entity nodes with high similarity between different modalities. For any text entity i∈V T and image entity j∈V I , if Similarity(i, j)>θ m , then add an undirected edge between i and j with a weight of w ij =Similarity(i, j).

[0106] In addition to the similarity edges between entities, the graph structure also contains entity-modality hub edges that connect entities to modality hubs. This type of edge is designed to aggregate entity information within the modality to the hub node. For each text entity i∈V T , add an undirected edge. For each image entity j∈V I , add an undirected edge. The weights of these edges are w ij Set to a fixed value w m .

[0107] After the construction is completed, the initial features of the node are obtained. Edge index and edge weight W = {w ij |(i, j)∈E}. This graph uses multimodal entities as basic units, builds connections by capturing fine-grained semantic similarity, and strengthens cross-modal information aggregation through modal hubs, providing rich structural and attribute information for subsequent graph neural network processing.

[0108] (4) False information detection module

[0109] This module performs information processing and multimodal fusion detection on the constructed multimodal similarity-aware heterogeneous graph G, and combines it with the unimodal detection branch to perform joint learning of the model.

[0110] On a heterogeneous graph G, this paper uses a multi-layer graph attention network (GAT) to learn node representations. The GAT layer uses the attributes of edges to perform similarity-aware message passing and aggregation, adaptively fusing neighbor information, and obtaining the final representation of all nodes in the graph. Subsequently, the global feature h of the graph is obtained by performing a global pooling operation on these final node embeddings G This feature is input into the multimodal graph detection branch, which outputs a graph-based

[0111] In addition to the detection branch based on graph information, the present invention also introduces a text detection branch and an image detection branch, which run in parallel to capture fake news clues from different perspectives. The text detection branch receives the extracted multi-channel features of the original text and outputs them after processing. The image detection branch receives the extracted multi-channel features of the original image and outputs them after processing

[0112] In order to effectively train the entire multi-task model and dynamically balance the contributions of the three detection branches: text, image, and graph, this paper adopts an adaptive weighted loss function based on learnable parameters. For each detection branch output, this paper calculates the cross entropy loss between it and the true label y: Total training loss is the weighted sum of these three losses, where the weights are automatically learned by the model. The loss function is defined as:

[0113]

[0114] where λ j is related to each loss term The associated learnable parameter represents the logarithmic variance of the output of the corresponding task. Therefore, exp(-λ j ) as The weight of λ is proportional to the estimated accuracy of the task: j The larger the exp(-λ j ) is smaller, the lower the weight of the loss term. j λ j The term plays a regularization role. By minimizing The model can automatically evaluate and adjust the importance of each branch in joint training, thereby optimizing the overall multimodal fake news detection performance and avoiding the tediousness and limitations of manual parameter adjustment.

[0115] To address the shortcomings of existing technologies in single-modal fake news detection, such as insufficient information utilization and the difficulty of mainstream multimodal fusion methods in effectively mining and integrating deep semantic connections between different modalities, the embodiments of the present invention propose a fake news detection framework, system and method based on the multimodal similarity-aware graph attention network MS-GAT.

[0116] like Figure 2 Figure 1 shows the overall architecture of MS-GAT, a multimodal fake news detection framework provided by an embodiment of the present invention. The framework includes a multimodal data preprocessing module, a multi-channel feature extraction module, a multimodal similarity-aware heterogeneous graph construction module, and a fake information detection module.

[0117] In a specific embodiment, the implementation and training of the MS-GAT model proposed in the present invention can be configured as follows:

[0118] 1. Implementation framework and hardware environment:

[0119] The MS-GAT model can be implemented using the PyTorch deep learning framework. All model training, validation, and testing experiments can be performed on hardware equipped with NVIDIA GPUs.

[0120] 2. Optimizer and training parameters:

[0121] The model training process uses the Adam optimizer to iteratively update the model parameters.

[0122] Initial Learning Rate: This can be set to 5 × 10-3. Depending on the dataset size and model convergence, a learning rate scheduler, such as exponential decay or cosine annealing, can be used.

[0123] Epochs: The total number of training iterations can be set to 3 epochs. This number of epochs should be adjusted based on the model's performance on the validation set to avoid underfitting or overfitting.

[0124] Batch size (BatchSize): can be set to 20. This value can be adjusted according to the GPU memory size.

[0125] 3. Model key hyperparameter settings:

[0126] In a specific implementation of the MS-GAT model, the key hyperparameters can be set as follows:

[0127] Number of Graph Attention Network (GAT) layers: set to 1.

[0128] GAT Attention Heads: Set to 4.

[0129] Node Embedding Dimension: Set to 768 dimensions.

[0130] Dropout ratio: Set to 0.3 in GAT layers or other fully connected layers to prevent overfitting.

[0131] Similarity threshold between entities:

[0132] Similarity threshold θ between text entities t : can be set to 0.6. Similarity threshold θ between image entities i: Can be set to 0.6. Similarity threshold θ between cross-modal entities ti : Can be set to 0.6. These thresholds are used to determine whether to add similarity edges between entities when building a heterogeneous graph.

[0133] Entity-modality hub connection weight w m : Set to 1.0. This is the fixed weight of the edge connecting the entity node to its corresponding modal hub node.

[0134] In order to further illustrate the working principle of the MS-GAT framework in detecting fake news in practical applications and its advantages in identifying inconsistencies between images and texts, the following will be combined with the attached Figure 3 A specific case is analyzed and explained in this paper.

[0135] like Figure 3 As shown, it shows a representative multimodal fake news sample selected from the public dataset CFND, and a schematic diagram of the heterogeneous graph structure constructed for the sample by the MS-GAT framework of an embodiment of the present invention.

[0136] like Figure 4 As shown in the figure, the MS-GAT framework proposed in this paper can correctly predict this case as fake news. The specific processing process and principle are as follows:

[0137] Step 1: Multimodal Data Preprocessing and Entity Recognition

[0138] The text content T is preprocessed and a NER tool is used to identify key text entities, such as "cable car," "fire," "tourists," and "life lost," to form a text entity set. The image content I is preprocessed and, using techniques such as OCR and subject recognition, image entities are identified, such as the main object in the image, "cable car," to form an image entity set.

[0139] Step 2: Multi-channel feature extraction and node initialization

[0140] The initial feature representations of text entities and image entities are obtained by encoding the pre-trained CLIP model. These encoded feature vectors will be used as the initial attributes of the corresponding nodes in the heterogeneous graph, such as the attached Figure 3 At the same time, the overall features of the text modality and the overall features of the image modality are also generated through the corresponding multi-channel feature extraction and fusion mechanism to initialize the text hub node and the image hub node.

[0141] Step 3: Multimodal Similarity-Aware Heterogeneous Graph Construction

[0142] Based on the entity features encoded by CLIP, the semantic similarity between cross-modal entities is calculated. In this embodiment, a similarity threshold can be set to 0.6. For this case, the cosine similarity calculated in the CLIP shared semantic space between the text entities describing catastrophic events (such as "fire" and "loss of life") and the apparently normal image entity "cable car" is significantly lower than the set threshold of 0.6. Therefore, when constructing a heterogeneous graph, no cross-modal entity similarity edges will be generated between these cross-modal entity pairs pointing to inconsistent content. The absence or weak connection of such key cross-modal association edges constitutes an important topological signal in the graph structure that characterizes the inconsistency between the text and image content.

[0143] Step 4: False Information Detection

[0144] The constructed heterogeneous graph is input into the graph attention network. GAT performs message transmission and node information aggregation through the edges existing in the graph structure. Due to the lack of strong semantic association between text entities describing negative events and image entities describing normal scenes, when the graph attention network aggregates information, the semantic gap or inconsistency signal reflected by this "disconnection" or "weak connection" is effectively captured and integrated into the final representation of the node. Ultimately, based on the learned graph representation containing this inconsistency signal, the classifier of the multimodal graph detection branch can accurately identify the news sample as "fake news."

[0145] This case study strongly demonstrates the advantages of our MS-GAT framework in capturing and exploiting fine-grained inconsistencies between text and image content. By constructing an explicit heterogeneous graph that reflects the strength of semantic connections between entities across modalities and applying graph neural networks to deeply analyze this structural information, our approach goes beyond superficial understanding of text and image content to delve deeper into the semantic consistency and potential contradictions between them, effectively identifying this typical type of misinformation.

[0146] An application embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of a fake news detection method based on a graph attention network with multimodal similarity perception.

[0147] An application embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor performs the steps of a fake news detection method based on a graph attention network with multimodal similarity perception.

[0148] An application embodiment of the present invention provides an information data processing terminal, which includes a fake news detection system based on a graph attention network with multimodal similarity perception.

[0149] 1. Specific application fields or related products of the present invention.

[0150] The fake news detection method based on multimodal similarity-aware graph attention network (MS-GAT) proposed in this paper has broad practical application value due to its innovative technical solution and excellent detection performance, and can effectively empower multiple key fields.

[0151] First, this technology can be directly applied to the intelligent content review systems of social media platforms, news portals, and various content distribution platforms. By integrating it into a core detection engine or API service, platforms can achieve real-time, automated detection of user-uploaded text and images. MS-GAT efficiently extracts multi-dimensional features from text and images and leverages its unique graph structure to deeply analyze cross-modal semantic consistency, thereby accurately identifying and addressing false information, effectively reducing manual review costs, purifying cyberspace, and enhancing the platform's credibility.

[0152] Secondly, this invention can be transformed into a professional media and fact-checking assistance tool for news organizations and fact-checking organizations. This tool not only provides risk assessments for false information but also, leveraging the interpretability potential of graph attention networks, reveals specific inconsistencies between graphic and text content, significantly improving the efficiency and accuracy of fact-checking efforts.

[0153] Finally, the present invention can also be used to develop Internet information literacy education tools or information query platforms for the public, helping ordinary netizens improve their ability to identify false information and promote the formation of a healthier and more rational Internet information environment.

[0154] In summary, the MS-GAT technical solution, by applying advanced multimodal analysis capabilities to actual scenarios, can create significant technical value, economic benefits and social benefits in multiple aspects such as content security, news authenticity assurance, enterprise risk management and public education.

[0155] 2. Relevant evidence of the technical effects obtained by the embodiments of the present invention.

[0156] In order to show the performance comparison experimental results of the proposed MS-GAT model and the existing baseline methods on the multimodal fake news detection task, the effectiveness and superiority of MS-GAT are fully verified.

[0157] This paper selects the following representative baseline methods for comparison:

[0158] (1) BERT: A unimodal method that uses a pre-trained BERT model to extract text features for classification.

[0159] (2) ViT: A unimodal method that uses a pre-trained ViT model to extract image features for classification.

[0160] (3) MCAN: A multimodal approach that fuses text and image features via a collaborative attention mechanism.

[0161] (4) UniSMMC: A multimodal approach for multimodal representation alignment using contrastive learning.

[0162] (5) NLIN: A method that unifies multimodal content into text and uses prompt-based large-scale language model reasoning for detection.

[0163] (6) GCN: A basic graph neural network method that propagates information by aggregating neighbor features.

[0164] (7)GAT: A graph neural network method based on attention mechanism that aggregates information by learning neighbor weights.

[0165] This paper compares the proposed MS-GAT model with baseline methods on the Weibo17 and CFND datasets. Table 1 shows the performance comparison results of each method on the test sets of the two datasets.

[0166] Table 1 Performance comparison of various methods on the dataset

[0167]

[0168] As can be seen from Table 1, the proposed MS-GAT model achieved the best detection performance on both datasets, fully verifying its effectiveness in the multimodal false information detection task.

[0169] Compared to baseline methods BERT and ViT that only use single-modal information for discrimination, MS-GAT fully integrates text and image information, and its performance has been significantly improved. Compared with traditional feature fusion methods MCAN and UniSMMC, MS-GAT constructs a multimodal similarity-aware heterogeneous graph, which elevates cross-modal information modeling from simple feature splicing or attention weighting to explicit graph structure association. It can more effectively capture the deep-level associations and potential conflicts between modalities, thereby achieving better fusion effects. At the same time, compared with existing graph baseline methods GCN and GAT, MS-GAT's unique multimodal heterogeneous graph construction method and similarity-aware graph attention mechanism using edge attributes enable it to demonstrate stronger capabilities in modeling multimodal content relationships.

[0170] Furthermore, MS-GAT demonstrates significant advantages over the recently proposed advanced method NLIN. NLIN attempts to unify multimodal content into a textual space for reasoning, but this can result in the loss of frequency-domain features of images and the sentiment intensity of text during the conversion process. MS-GAT, on the other hand, preserves multimodal features and structure and directly performs multimodal interaction and inconsistency analysis in graph space, making it more effective at capturing complex cues in multimodal disinformation. These comparative results fully demonstrate the superiority and advancement of the proposed MS-GAT framework in the task of multimodal disinformation detection.

[0171] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0172] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A fake news detection system based on multimodal similarity-aware graph attention network, characterized by: include: a. Multimodal data preprocessing module, used to clean and standardize input text and images and identify text entity sets and image entity sets; b. A multi-channel feature extraction module, which is used to obtain semantic feature vectors and sentiment feature vectors from text, and spatial domain feature vectors and frequency domain feature vectors from images. It then generates a fused overall text feature vector and overall image feature vector using a gating unit composed of a multilayer perceptron and a smooth monotonic curve activation function. c. A multimodal similarity-aware heterogeneous graph construction module, which uses entity feature vectors as node attributes and establishes edge weights by comparing the ratio of the dot product of any two node feature vectors to the product of the module length. It also adds fixed-weight edges between text entities and text hub nodes, and between image entities and image hub nodes, to obtain a heterogeneous graph with a complete definition of the node and edge sets. d. A false information detection module is used to input the heterogeneous graph into at least two layers of graph attention layers, complete node information aggregation, and obtain the overall graph vector through global pooling. This vector is input into the graph detection branch, text detection branch, and image detection branch respectively along with the fused overall text feature vector and the fused overall image feature vector. The module calculates three cross-entropy losses and automatically adjusts the weights of each item based on the learnable branch weight parameters to output the news authenticity prediction result.

2. The system according to claim 1, wherein The multimodal data preprocessing module uses named entity recognition tools to extract proper nouns, place names, and organization names from the text, and uses optical character recognition and subject recognition technology to extract image text information and main visual focus, and then merges the two types of results to generate an image entity set.

3. The system according to claim 1, wherein: During text processing, the multi-channel feature extraction module obtains semantic feature vectors based on the bidirectional encoder pre-trained language model, and obtains sentiment feature vectors based on a similar model fine-tuned on sentiment-annotated corpus; during image processing, the module obtains spatial domain feature vectors based on the visual converter pre-trained model, and obtains frequency domain feature vectors by analyzing the image frequency distribution through fast Fourier transform.

4. The system according to claim 1, wherein: The multimodal similarity-aware heterogeneous graph construction module sets text internal thresholds, image internal thresholds, and cross-modal thresholds. Only when the cosine similarity between two nodes is higher than the corresponding thresholds will an undirected edge be established between the nodes and the similarity value be used as the edge weight.

5. The system according to claim 1, wherein: The weight parameters of the three-branch loss in the false information detection module are back-propagated and updated as network parameters during the training process. Each weight parameter represents the logarithmic variance of the uncertainty of the output error of the corresponding branch. The system aims to minimize the sum of the weighted losses of the three branches and simultaneously penalize the sum of the weight parameters.

6. A fake news detection method based on multimodal similarity-aware graph attention network, characterized by: include: Step 1: Segment and denoise the input text and identify text entities. Normalize the size and pixels of the input image, perform text recognition and subject recognition, and generate an image entity set. Step 2: Use the bidirectional encoder pre-trained language model and the sentiment fine-tuned language model to obtain the text semantic feature vector and sentiment feature vector respectively, and fuse them into the overall text feature vector through the gating unit; use the visual converter pre-trained model and the fast Fourier transform to obtain the image spatial domain feature vector and frequency domain feature vector respectively, and fuse them into the overall image feature vector through the gating unit; Step 3: Use the cross-modal alignment pre-trained model to encode the entity feature vector, add the text entity node, image entity node, text hub node, and image hub node to the node set, generate the edge set with cosine similarity and fixed weight, and construct a heterogeneous graph; In step 4, the heterogeneous graph is input into the multi-layer graph attention network to obtain the overall graph vector. This vector, together with the overall text feature vector and the overall image feature vector, are input into the graph detection branch, text detection branch, and image detection branch respectively. The network parameters are optimized by automatically weighted fusion of the three cross-entropy losses, and the fake news detection results are output.

7. The method according to claim 6, wherein The gating unit in step 2 first uses a multilayer perceptron to generate a weight vector with the same dimension as the input feature vector, then multiplies the weight vector element-wise with the first input feature vector, subtracts one from the weight vector and multiplies it element-wise with the second input feature vector, and finally adds them together to obtain a fused feature vector.

8. The method according to claim 6, wherein The weight parameters of the three-branch cross entropy loss in step 4 are automatically adjusted by minimizing the overall objective function of "the sum of the weighted losses plus the sum of the weight parameters".

9. A computer device comprising a processor, a memory and a network interface, characterized in that: The processor is configured to perform a fake news detection process according to the method according to any one of claims 6 to 8, the memory is used to store programs and data required to execute the method, and the network interface is used to receive news data and output detection results.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by a computer, causes the computer to perform a fake news detection operation according to the method according to any one of claims 6 to 8.

Images