Data-enhanced fine-grained multi-mode false information detection method and system

Abstract

The invention discloses a data-enhanced fine-grained multi-mode false information detection method and system, and aims to improve the accuracy of image-text false news detection. The method comprises the following steps: acquiring a news text and associated pictures thereof, extracting a text core entity semantic sequence and a visual entity semantic sequence by respectively utilizing a pre-training language model and a visual entity recognition model, and performing knowledge enhancement on original word-level text representation and low-level visual features through an attention mechanism; calculating a correlation matrix between the enhanced text and the visual features, dividing consistent and inconsistent regions, and respectively extracting consistent and inconsistent features; and finally, fusing the two types of features and global text representation to generate classification features so as to judge the authenticity of the news. Through double-channel knowledge enhancement and fine-grained cross-modal consistency analysis, the detection performance is remarkably improved, and the method is suitable for scenes such as social media and news platforms.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of information processing, and particularly relates to a data-enhanced fine-grained multi-modal false information detection method and system. BACKGROUND

[0002] In recent years, with the rapid development of social media and online information platforms, the spread of fake news has become a serious social problem. Fake news not only misleads public perception and affects social stability, but also can have a profound negative impact on individual reputation, financial markets, and even national politics. Therefore, developing efficient and accurate fake news detection technology is of great significance for maintaining a clear and bright network information environment.

[0003] Early fake news detection methods mainly rely on text content analysis, which identifies language patterns, sentiment orientation, or factual errors in the text to make judgments. However, with the popularization of multimedia technology, more and more fake news uses a combination of text and images to spread. Compared with pure text news, fake news with pictures has more visual impact and persuasiveness, attracting more attention and accelerating the spread. Therefore, detection methods relying solely on text modality have been unable to meet the current challenges.

[0004] To address this challenge, multi-modal fake news detection technology has emerged. Existing technologies mainly fuse information from multiple modalities such as text, images, and social context to improve detection performance. For example, some technical solutions use recurrent neural networks (RNN) and attention mechanisms to fuse text, visual features, and user comments. However, such methods usually need to collect enough social context information at the later stage of news dissemination, limiting their detection ability in the early stages of news publication.

[0005] Another type of content-based detection method focuses on analyzing the consistency between text and image content. Intuitively, the text and image content in fake news often do not match or contradict each other. Existing technologies measure this consistency by calculating the similarity of global features between text and images. However, these methods have the following significant limitations:

[0006] First, to obtain global features, text word vectors or image region features usually need to be pooled, which suppresses important local information contained in specific text segments or image regions, causing key inconsistencies to be ignored.

[0007] Second, existing methods mostly fail to achieve fine-grained alignment between text and images. They cannot accurately associate a specific text word (such as "the president of a certain country") with a corresponding specific region in the image (such as the president's face), limiting the ability to precisely locate and mine contradictions between text and images.

[0008] In addition, the prior art is mostly based on low-level visual features such as color, texture and edge, and lacks deep understanding of high-level semantics in images such as character identity, landmark building and event scene, which makes it difficult for the model to understand the deep semantic relationship between the image and the text.

[0009] Therefore, how to design a fake news detection method that can deeply understand the semantic relationship between multi-modal content, realize fine-grained cross-modal alignment, and accurately identify inconsistencies between images and texts, so as to improve the accuracy, robustness and explainability of detection, is a key problem to be solved in the current technical field. SUMMARY

[0010] In view of the deficiencies of the current fake news detection method based on multi-modal information in cross-modal fine-grained alignment, semantic consistency modeling and content agnostic, the present application proposes a data-enhanced fine-grained multi-modal fake information detection method, which encodes the input news text and extracts and enhances the key words, extracts the visual features of the input image and combines visual entity recognition and enhancement, calculates the cross-modal correlation between the text and the image and divides the consistent and inconsistent regions, respectively models the consistency features and the inconsistency features, and inputs the classifier through the gating fusion mechanism for weighted fusion, and outputs the judgment result of the news authenticity.

[0011] The present application also proposes a data-enhanced fine-grained multi-modal fake information detection system for implementing the above-mentioned multi-modal fake information detection method, comprising:

[0012] The text encoding module is used for encoding the input news text, extracting word-level representation and global text representation;

[0013] The keyword enhancement module is used for extracting core keywords from the text, generating a text core entity semantic sequence, and injecting keyword semantic information into the word-level representation through an attention mechanism to generate knowledge-enhanced text word-level representation;

[0014] The image feature extraction module is used for extracting low-level visual features from the input news image to obtain a block visual feature sequence;

[0015] The visual entity enhancement module is used for identifying significant visual entities in the image, generating a visual entity semantic sequence, and injecting entity semantic information into the visual feature sequence through an attention mechanism to generate a semantic-enhanced visual feature sequence;

[0016] The cross-modal alignment module is configured to calculate word-region level correlations between the knowledge-enhanced text word-level representation and the semantic-enhanced visual feature sequence, construct a correlation matrix, and divide the correlation matrix into consistent parts and inconsistent candidate parts according to a preset threshold.

[0017] The consistency modeling module is configured to aggregate visual information to the text representation by attention based on the consistent parts, and generate a consistent feature vector.

[0018] The inconsistency modeling module is configured to perform feature splicing on the mismatched word-region pairs based on the inconsistent candidate parts, and generate an inconsistent feature vector.

[0019] The feature fusion module is configured to dynamically weight and fuse the consistent feature vector and the inconsistent feature vector through a gating network to obtain a fusion feature vector.

[0020] The classification module is configured to input the fusion feature vector into a classifier after being spliced with a global text representation, and output a classification result of the news being real or fake.

[0021] The present application also proposes an electronic device, which comprises:

[0022] a memory;

[0023] a processor;

[0024] The memory stores computer execution instructions.

[0025] The processor executes the computer execution instructions stored in the memory to implement the above-mentioned false news detection method based on multi-modal information.

[0026] The present application also proposes a computer storage medium, which stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the above-mentioned false news detection method based on multi-modal information.

[0027] The present application has the following advantages:

[0028] 1) Improve detection accuracy: through bidirectional knowledge enhancement of keywords and visual entities, the semantic representation ability of text and image modalities is significantly improved, and the sensitivity of the model to contradictions between text and images is enhanced;

[0029] 2) Realize fine-grained consistency analysis: through explicit partitioning of the cross-modal correlation matrix, the "consistent" and "inconsistent" clues are modeled respectively, and typical false news patterns such as text-image mismatch are effectively captured;

[0030] 3) Enhance interpretability: the model can output specific word-region pairs that are judged as "inconsistent", making the detection result more interpretable and helping to understand the basis for the model's decision.

[0031] 4) Improve model robustness: by introducing entity semantics, the model's ability to identify complex and hidden fake news is enhanced, and the model's generalization performance in different scenarios is improved;

[0032] 5) Strong compatibility: this method can be compatible with different types of text and image data, and is suitable for various application scenarios such as social media and news websites. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The specific implementation steps of the fake news detection method based on multi-modal information described in the present application;

[0034] Figure 2 The overall architecture diagram of the fake news detection system based on multi-modal information described in the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0036] The present application proposes a data-enhanced fine-grained multi-modal fake information detection method, which realizes high-precision detection of fake news by performing knowledge-enhanced semantic encoding on text and images and combining fine-grained cross-modal consistency analysis. In the present embodiment, as shown in Figure 1 , the specific process of the method is as follows:

[0037] For a multi-modal news data to be detected, first, the text content T and the associated picture I contained therein are obtained. Then, the pre-trained language model ERNIE is used to encode the text content T to generate a text word-level representation sequence T words =[t1,t2,…,t N ]∈R N×d and a global text representation x t for the classification task, where N is the number of text tokens, and d is the vector dimension. At the same time, a keyword extraction algorithm TextRank is used to extract k most representative keywords from the news text content T to form a keyword name set K = [k n1 ,k n2 ,…,k nk ]; each keyword is input into the ERNIE model as an independent text to obtain its semantic vector k iThis forms the semantic sequence K of the core entities in the text. emb =[k1,k2,…,K N ]∈R N×d Then, an attention mechanism is used to analyze the semantic sequence K of the core entities in the text. emb Semantic information injection into word-level representation T words Generate knowledge-enhanced text word-level representations T enh .

[0038] Simultaneously, the associated image I is processed. On one hand, a low-level visual feature sequence V = [v1, v2, ..., v] is extracted using a convolutional neural network VGG19. m [e1, e2, ..., e2], where m is the number of regions in the image; on the other hand, an external entity recognition model (such as the visual entity recognition service provided by Baidu AI platform) is used to identify the visual entities contained in image I, obtain a list of entity names, and input each entity name into the pre-trained language model ERNIE to obtain the corresponding semantic vector representation, thereby forming a visual entity semantic sequence E = [e1, e2, ..., e2]. n ], where n is the number of identified entities; based on the low-level visual feature sequence V and the visual entity semantic sequence E, a designed attention mechanism module injects high-level semantic information from E into the features of each region of V, generating a semantically enhanced visual feature sequence V. enh =[v′1,v′2,…,v′ m Then, the word-level representation T of the text is computed. enh With semantically enhanced visual feature sequence V enh The correlation matrix S between them, where elements S ij This represents the relevance score between the i-th text word and the j-th visual region. Based on a preset threshold λ, the elements in the relevance matrix S are divided into two sets: S... ij The part corresponding to the element >λ is defined as the consistent part S. m , will S ij The part corresponding to the elements ≤λ is defined as the inconsistency candidate part S. c ;

[0039] For the consistent part S respectively m and inconsistent candidate parts S c Feature exploration is performed. For the consistent part S m A cross-modal attention mechanism is employed to aggregate visual information onto text representations to explore consistency cues between text and image content; for inconsistency candidate parts S c Then, by using specific feature interaction operations, potential contradictions are uncovered to explore clues of inconsistency between the text and image content, and finally, consistent features z are obtained. m and inconsistency feature z cFinally, the consistency feature z is fused. m The inconsistency feature z c And the aforementioned global text representation x t The final classification feature vector is generated and input into the classifier, which outputs the judgment result of whether the news data is real or fake news.

[0040] The multimodal fake information detection system used in this embodiment includes three parts: multimodal deep semantic enhancement coding, cross-modal alignment and partitioning, and fine-grained consistency learning and classification. Specifically, it includes:

[0041] (I) Multimodal Deep Semantic Enhancement Coding

[0042] In this section, the input text and images are subjected to knowledge-rich deep semantic encoding, aiming to provide high-quality semantic representations for subsequent cross-modal analysis.

[0043] 1) Text-side encoding

[0044] Given input text T = [w1, w2, ..., w N ], where w i Let represent the i-th word, and N be the sequence length. The text TT is encoded using the pre-trained language model ERNIE to obtain the word-level semantic vector sequence T corresponding to each word. words =[t1,t2,…,t N ], and the global semantic representation x of the entire text. t To further enhance the semantic expression of the text, the TextRank keyword extraction algorithm is used to extract k of the most representative keywords from the news text content T, forming a keyword name set K = [k n1 ,k n2 ,…,k nk ]; Each keyword As an independent text input to the ERNIE model, its semantic vector k is obtained. i This forms the semantic sequence K of the core entities in the text. emb =[k1,k2,…,k N ]∈R N×d To inject the core semantic information carried by keywords into the original text representation, this embodiment employs an information fusion strategy based on an attention mechanism. Specifically, the original word-level representation T is calculated. words With keyword semantic sequence K emb The attention weights between words are used to aggregate keyword information and add it back to the original word vectors to generate a knowledge-enhanced text word-level representation T. enh =[t′1,t′2,…,t′ N Attention weights are calculated as follows:

[0045]

[0046] Aggregated keyword information:

[0047]

[0048] The final enhanced word-level representation is generated:

[0049] t′1=t i +z i

[0050] Therefore, the enhanced text word-level representation sequence is T. enh =[t′1,t′2,…,t′ N ].

[0051] 2) Image-side coding

[0052] During image-side processing, the system first extracts low-level visual features of the image, then obtains semantic entities in the image through visual entity recognition, and integrates the semantic information of these entities into the visual features to achieve semantically enhanced visual representation.

[0053] Given an input image I, a VGG19 convolutional neural network pre-trained on the ImageNet dataset is used as the visual feature extractor. Image I is input into the VGG19 model, which extracts the feature map from its penultimate layer and divides it into M spatial patches, each patch corresponding to a local region's visual feature vector. The final result is a low-level visual feature sequence V = [v1, v2, ..., v...]. m ].

[0054] To incorporate high-level semantic information, the system calls a visual entity recognition API (such as the visual entity recognition API on the Baidu AI platform). The input image I is used, and the API returns a list of recognized entity names. Each entity name is then input into the ERNIE model to obtain its corresponding semantic vector e. i This constitutes a visual entity semantic sequence E = [e1, e2, ..., e n ], where n is the number of entities identified.

[0055] To enhance the semantic expressive power of visual features, the system employs an attention mechanism to inject information from the semantic sequence E of visual entities into low-level visual features V. The relevance score between V and E is calculated, attention weights are generated, and E is weighted and aggregated. The aggregated semantic information is then added to the original visual features to obtain the semantically enhanced visual feature sequence V. enh =[v′1,v′2,…,v′ m The correlation scores are calculated as follows:

[0056] s ij =v i T e j

[0057] Attention weight normalization:

[0058]

[0059] Aggregate entity semantic information:

[0060]

[0061] Generate enhanced visual features:

[0062] v′ i =v i +y i

[0063] Therefore, the semantically enhanced visual feature sequence is V. enh =[v′1,v′2,…,v′ m ].

[0064] (II) Cross-modal alignment and partitioning

[0065] Fine-grained semantic interaction between text and image modalities is achieved through cross-modal alignment and partitioning modules, and cross-modal associations are explicitly partitioned based on the degree of semantic consistency, thus providing a structured analysis path for subsequent fine-grained consistency learning and classification.

[0066] To measure the semantic correspondence between text and images, the system calculates the knowledge-enhanced word-level representation sequence T of the text. enh =[t′1,t′2,…,t′ N [and semantically enhanced visual feature sequence V] enh =[v′1,v′2,…,v′ m Fine-grained similarity between words and regions. Specifically, for each word-region pair (i,j), its normalized cosine similarity is calculated as a cross-modal relevance score. ij The image features are first processed through a learnable linear projection matrix. Mapped to a semantic space consistent with text features. The relevance score is calculated as follows:

[0067]

[0068] Where, t′ i To enhance the semantic vector of the knowledge of the i-th word, v′ j For enhancing the visual features of the j-th image region, W cThis is a trainable parameter matrix used to achieve cross-modal alignment. All N×M correlation scores constitute the cross-modal correlation matrix S∈R. N×M , where each element s ij =score ij Reflects the text lexical unit w i With image region r j The degree of semantic matching between them.

[0069] To further distinguish between consistent and potentially conflicting information between modalities, the system performs a binary partitioning of the relevance matrix S based on a preset threshold λ, dividing the word-region association pairs into two mutually exclusive subsets: a high-relevance subset (i.e., the "consistent part") and a low-relevance subset (i.e., the "inconsistent candidate part").

[0070] The specific division method is as follows:

[0071] S m ={(i,j)∣s coreij >λ},S c ={(i,j)∣s coreij ≤λ}

[0072] Among them, S m The set of semantically consistent word-region associations reflects the collaborative support relationship between text and images in terms of content; S c This partitioning operation represents a candidate set that is semantically inconsistent or weakly correlated, potentially containing contradictory or misleading information found in fake news. This partitioning provides the model with a clear supervisory signal structure, enabling subsequent modules to focus on consistency modeling and anomaly pattern detection, respectively.

[0073] (III) Fine-grained Consistency Learning and Classification

[0074] Building upon cross-modal alignment and partitioning, this embodiment employs fine-grained consistency learning and classification modules to model consistency support cues and inconsistency contradiction cues between modalities, fusing their features to achieve final fake news identification. This module fully leverages the semantic alignment results generated by preceding modules, performing differentiated modeling for different semantic relationship paths, significantly improving the model's ability to capture potential contradictory information in multimodal content and its classification accuracy.

[0075] For the set S of highly relevant word-region pairs identified in cross-modal partitioning m The system believes that this reflects the semantic synergy between text and images. To enhance the expression of such consistency information, this embodiment employs an attention mechanism to enhance the semantically rich visual feature sequence V. enh =[v′1,v′2,…,v′ m Aggregation to knowledge-enhanced text word-level representation T enh= [t'1, t'2,..., t'N] N , thereby generating text representations with more cross-modal semantic support.

[0076] Specifically, for the i-th text token, its attention weight is computed based on its relevance to all image regions satisfying (i,j) ∈ S m , and the corresponding visual features are weighted-summed to obtain a context-aware enhanced representation Subsequently, the original representation t' i is concatenated with the aggregated information and input into a small multi-layer perceptron (MLP) to output a consistency score for the token measuring its strength of consistency across modalities.

[0077] Finally, by weighted-aggregating the enhanced representations of all tokens, the overall consistency feature vector z is generated.

[0078]

[0079] where is generated by the following MLP network:

[0080]

[0081] This consistency feature z m effectively aggregates the evidence of cross-modal semantic synergy for supporting the identification of real news.

[0082] For the set of low-relevance or weakly-matching word-region pairs S c identified in the cross-modal partition, the system regards them as potential semantic contradiction regions, which might correspond to the typical features of fake news such as the inconsistency between text and images, misleading image captions, etc. To explicitly model such inconsistencies, the embodiment performs an element-wise addition operation for each pair (i,j) belonging to S c , fusing its text and visual semantic information to generate a contradiction representation vector c ij :

[0083]

[0084] Subsequently, each c ij is input into another independent small MLP network to output its corresponding inconsistency score measuring the degree of contradiction for the word-region pair. Finally, by weighted-aggregating the contradiction representations in all S c , the inconsistency feature vector z c is generated.

[0085]

[0086] where S c(i) denotes all low-relevance image regions related to the ith word. The inconsistency feature z c Explicitly capturing the conflict information between text and image modalities helps to improve the sensitivity of the model to fake content.

[0087] To integrate the consistency and inconsistency clues and achieve a robust final judgment, the system introduces a gating fusion mechanism to dynamically weight and fuse z m and z c . Specifically, the two are concatenated and input into a learnable gating network (implemented by a single-layer neural network), which outputs a scalar weight g∈[0,1]:

[0088]

[0089] where σ(·) is the Sigmoid function, W g and b g are trainable parameters. The fused feature is represented as:

[0090] z fused =g·z m +(1-g)·z c

[0091] Finally, the fused feature z fused is concatenated with the global representation of the text x t , input into a two-layer fully connected neural network, and output the binary classification probability through the Softmax function to determine whether the input news is real or fake:

[0092]

[0093] where W f and b f are classification layer parameters. The classification result is the multi-modal fake news detection judgment output by the embodiment.

[0094] The embodiment also gives the specific training process of the multi-modal fake information detection model, which specifically includes the following steps:

[0095] 1) Dataset and preprocessing

[0096] The dataset used to train, validate and test the present application contains two labels: true news (0) and fake news (1). The training dataset adopts a public multi-modal fake news dataset, such as the Weibo-21 dataset. This dataset contains news samples collected from social media platforms, and each sample is composed of news text, associated pictures and true or false labels. To ensure the generalization ability of the model, the dataset is divided by event, that is, all news samples of the same event are divided into the same set in the training set, the validation set or the test set, avoiding the leakage of samples of the same event in the training and testing stages. For text data, word segmentation processing is performed; for picture data, it is normalized to 224x224 pixels to meet the input requirements of the VGG19 model.

[0097] 2) Model initialization

[0098] Each module in the model is initialized. The ERNIE model used in the text encoding module and the visual entity encoding module is initialized using the pre-trained weights on a large-scale Chinese corpus. The VGG19 model used in the image processing module is initialized using the pre-trained weights on the ImageNet dataset. Other parameters in the model, such as the W e , W c matrices in the attention mechanism, the parameters of the gating network and the MLP, are randomly initialized using the Xavier method.

[0099] 3) Loss function

[0100] The present application uses a composite loss function to guide the training of the model. The total loss function LtotalLtotal is composed of the main classification loss L cls and the partition loss L partition .

[0101] ① Main classification loss

[0102] A binary cross-entropy loss function is used to optimize the final true or false classification task. Its expression is:

[0103]

[0104] where N is the total number of samples, y i is the true label (0 or 1) of the i-th sample, and p i is the probability that the model predicts the sample to be fake news.

[0105] ② Partition loss

[0106] The mean square error (MSE) loss is used to guide the model to learn the correct decision logic. For the real news sample, the partition label is [1, 0], which encourages the model to rely more on the clues of the consistent part when making decisions; for the fake news sample, the partition label is [0, 1], which encourages the model to rely more on the clues of the inconsistent candidate part when making decisions. The loss function expression is:

[0107]

[0108] where g i is the dynamic weight of the gating network output of the i-th sample, is the corresponding partition label.

[0109] ③Total loss function

[0110] The main classification loss and the partition loss are weighted and summed to obtain the total loss function:

[0111] L total =L cls +αL partition

[0112] Wherein, alpha is a balance hyperparameter, and in the embodiment, the value is 0.5.

[0113] In summary, the data enhancement fine-grained multi-modal fake information detection method and system proposed by the application effectively improves the deep semantic understanding ability of the model to the text and image content by introducing knowledge-enhanced deep semantic encoding. Through the design of semantic-enhanced visual representation and cross-modal alignment mechanism, fine-grained association between text words and image regions is realized. Through fine-grained consistency learning and classification module, consistent clues and inconsistent clues of the text and image content can be explored respectively, and dynamic fusion is realized through the gating network, so that the model pays more attention to consistent information when judging real news, and pays more attention to contradictory information when judging fake news. The method of the application can be widely applied to social media platforms, news aggregation websites and other scenes, and can effectively improve the detection accuracy and explainability of fake news, which has important significance for maintaining a clear and bright network information environment.

[0114] Although the embodiments of the application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the application, and the scope of the application is defined by the appended claims and their equivalents.

Claims

1. A data-enhanced, fine-grained, multimodal fake information detection method, characterized in that, The method includes: semantically encoding the input news text, extracting word-level representations and global text representations, and obtaining a semantic sequence of core text entities through keyword extraction and semantic encoding; injecting the information of the semantic sequence of the core text entities into the word-level representation using an attention mechanism to generate a knowledge-enhanced text word-level representation; extracting low-level visual features from the input news image to obtain a block-based visual feature sequence, and obtaining a visual entity semantic sequence through visual entity recognition and semantic encoding; injecting the information of the visual entity semantic sequence into the block-based visual feature sequence using an attention mechanism to generate a semantically enhanced visual feature sequence; and calculating the knowledge-enhanced text word-level representation and the semantically enhanced visual... Cross-modal correlations between feature sequences are analyzed to construct a word-region-level correlation matrix. The correlation matrix is ​​then divided into a consistent component and an inconsistency candidate component based on a preset threshold. Based on the consistent component, visual features are aggregated into text representations through attention, and the consistency strength is calculated to generate a consistent feature vector. Based on the inconsistency candidate component, mismatched word-region pairs are concatenated, and the inconsistency strength is calculated to generate an inconsistency feature vector. A gated fusion mechanism is used to weightedly fuse the consistent and inconsistent feature vectors to obtain a fused feature vector. This fused feature vector is then concatenated with the global text representation and input into a classifier to output the classification result for the authenticity of the news.

2. The method according to claim 1, characterized in that, The keyword extraction and semantic encoding include: using the TextRank algorithm or calling the keyword extraction API to extract k core keywords from the news text, inputting each keyword as an independent text into the ERNIE model to obtain its semantic vector, and forming a semantic sequence of the core entities of the text.

3. The method according to claim 1, characterized in that, The visual entity recognition and semantic encoding includes: calling the visual entity recognition API to identify salient objects or scenes in the image, obtaining a list of entity names, inputting each entity name into the ERNIE model, obtaining its semantic vector, and forming a visual entity semantic sequence.

4. The method according to claim 1, characterized in that, The cross-modal correlation calculation includes: calculating the correlation score between the i-th word and the j-th image region using the following formula: in, To enhance the knowledge of the i-th word, v j ′ is the semantic enhancement visual vector of the j-th image region, w c It is a learnable cross-modal projection matrix.

5. The method according to claim 1, characterized in that, The method for dividing the consistent portion and the candidate inconsistent portion is as follows: S m ={(i,j)∣S ij >λ},S c ={(i,j)∣S ij ≤λ} Wherein, λ is a preset threshold, and its value range is (0,1).

6. The method according to claim 1, characterized in that, The generation of the consistency feature vector includes: aggregating visual features to text representations for consistent word-region pairs using an attention mechanism to obtain enhanced text vectors; calculating the consistency strength score of each enhanced text vector using a first small MLP; and weighting and aggregating the enhanced text vectors corresponding to all consistency strength scores to obtain the consistency feature vector.

7. The method according to claim 1, characterized in that, The gated fusion mechanism includes: concatenating consistent and inconsistent feature vectors and inputting them into a gated network, outputting dynamic fusion weights; and calculating the fusion feature vector using the following formula: z fused =g·z m +(1-g)·z c Among them, z m For a consistent feature vector, z c denoted as the inconsistency feature vector, and g represents the weights output by the gating network.

8. A data-enhanced fine-grained multimodal fake information detection system, characterized in that, The method for detecting fake information based on multimodality as described in any one of claims 1 to 7 comprises: a text encoding module for semantically encoding input news text and extracting word-level representations and global text representations; a keyword enhancement module for extracting text keywords and generating semantic sequences of core text entities, injecting them into the word-level representation through an attention mechanism to generate knowledge-enhanced text word-level representations; an image feature extraction module for extracting low-level visual features from input news images to obtain block-level visual feature sequences; a visual entity enhancement module for identifying visual entities in the image and generating semantic sequences of visual entities, injecting them into the block-level visual feature sequences through an attention mechanism to generate semantically enhanced visual feature sequences; a cross-modal alignment module for calculating word-region-level correlation between text and image, constructing a correlation matrix, and dividing it into consistent parts and inconsistent candidate parts according to a threshold; a consistency modeling module for generating consistent feature vectors based on consistent parts; an inconsistency modeling module for generating inconsistent feature vectors based on inconsistent candidate parts; a feature fusion module for fusing consistent and inconsistent feature vectors through a gating mechanism; and a classification module for concatenating the fused features with the global text representation and classifying them to output the news authenticity judgment result.

9. A general-purpose artificial intelligence-generated image detection device, characterized in that, The device includes: Memory; processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement a data-enhanced fine-grained multimodal fake information detection method as described in any one of claims 1 to 10.

10. A computer storage medium, characterized in that, The computer storage medium stores computer execution instructions, which, when executed by a processor, are used to implement a data-enhanced fine-grained multimodal fake information detection method as described in any one of claims 1 to 7.

Images