Multi-modal false information recognition method and system based on multi-granularity consistency

Abstract

The present application relates to the technical field of information security, and particularly relates to a multi-modal false information recognition method and system based on multi-granularity consistency, multi-modal information data is collected to form an information data set, samples include false data and non-false data, and corresponding labels 0 and 1 are set; a BERT model is used as a text encoder to generate text feature representation, and a ViT model is used as an image encoder to generate visual feature representation of the image; three levels of consistency metrics are fused, token-level fine-grained consistency based on a multi-head cross attention mechanism, phase-level medium-grained consistency based on a graph attention network and global-level coarse-grained consistency based on a CLIP model are introduced. The present application considers semantic matching between different levels of text and images, comprehensively reveals the internal relationship of multi-modal information, and improves the recognition accuracy and efficiency of false information.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a multimodal spoofing identification method and system based on multi-granularity consistency. Background Technology

[0002] Over the past few decades, social media services have expanded rapidly. Their convenience and immediacy have enabled information to spread quickly worldwide, and more and more people are using the internet to publish and browse information. The rapid development of computer vision technology has led to a surge in multimodal content on social media platforms, resulting in the widespread application of multimodal content analysis in the field of online misinformation detection. Therefore, developing an automated, assisted multimodal misinformation detection method is crucial for protecting people's lives and property and maintaining social stability.

[0003] The main challenges of multimodal fake information detection lie in exploring intermodal relationships across multimodal multimedia data, addressing the heterogeneity between modalities, and capturing consistency information between textual and visual modalities. Imbalanced and complementary relationships exist between different modalities. Because each modality has unique features and cannot be perfectly mapped to other modalities, the amount of information they provide when expressing the same semantics may differ. Most existing techniques simply connect different modal features together or fuse them only at a global level, inevitably missing some important information and ignoring the more complex components of the two modalities, resulting in low utilization of information at different granularities. A few studies have focused on the consistency of multimodal features, considering only single-granularity consistency at the token level, without measuring multi-granularity alignment consistency on image-text pairs. Summary of the Invention

[0004] The purpose of this invention is to address the problems existing in the prior art and propose a multimodal misinformation identification method and system based on multi-granularity consistency. Through a three-level granularity multimodal consistency measurement mechanism, it can more comprehensively reveal the intrinsic relationship of multimodal information and effectively improve the detection accuracy of misinformation.

[0005] To achieve the above objectives, the technical solution adopted is:

[0006] A multimodal spoofing identification method based on multi-granularity consistency, comprising:

[0007] Multimodal information data is collected to form an information dataset. The samples include fake data and non-fake data, and corresponding labels 0 and 1 are set.

[0008] The BERT model is used as the text encoder to generate text feature representations, while the ViT model is used as the image encoder to generate visual feature representations of the images.

[0009] A token-level fine-grained consistency mechanism based on multi-head cross-attention is introduced to generate text and visual feature vectors at this level, and token-level consistency between text and images is measured.

[0010] We introduce phase-level granularity consistency based on graph attention networks to generate text and visual feature vectors at this level, and measure the phase-level consistency between text and images.

[0011] We introduce a global-level coarse-grained consistency model based on CLIP to generate text and visual feature vectors at this level, and perform a global-level consistency measurement between text and images.

[0012] A granular adaptive attention mechanism is adopted to assign weights to the consistency features at each level to obtain an aggregated feature vector; the model is optimized by minimizing the cross-entropy loss function; and the optimized model is used for multimodal false information detection.

[0013] According to the multimodal spoofing identification method based on multi-granularity consistency of the present invention, the BERT model is further used as a text encoder to generate text feature representation O. T ={t1,…t i …,t n}, where n represents the length of the text tag, t i Representing embedded features; using the ViT model as an image encoder to generate visual feature representations of the image O. v ={v1,…v j …,v m}, where v j This represents the feature of the j-th region of the image.

[0014] According to the multimodal fake information identification method based on multi-granularity consistency of the present invention, the token-level fine-grained consistency based on the multi-head cross-attention mechanism further includes:

[0015] A multi-head cross-attention mechanism is used to map data from two modalities to a common semantic space; the query vector Q is derived from text features O. T The key vector K and value vector V are generated from visual features O. v The calculation formula for the multi-head cross-attention mechanism is as follows:

[0016] MultiHeadCrossAttention(Q,K,V)=Concat(head1,…,head h W O

[0017]

[0018] in, W is the linear transformation matrix of the i-th attention head. O The output transformation matrix is ​​given by h, where h is the number of attention heads, and d is the number of attention heads. k This refers to the dimension of the query key-value vector; each attention head generates an updated text representation, which is combined and transformed through linear layers to produce the final text feature representation.

[0019] The inner product is used to measure atomic-level consistency, and the token-level consistency score C is finally obtained through a fully connected layer and a softmax function. Token :

[0020]

[0021] in, v represents the i-th word element in the text. m This represents the j-th image block.

[0022] According to the multimodal spoofing identification method based on multi-granularity consistency of the present invention, the granularity consistency in the phase-level of the graph attention network further includes:

[0023] Graph attention networks are used to model text and images, with words in the text and image patches as nodes in the graph, and edges established through dependencies and geometric adjacency. The text embedding representation and the visual embedding representation are as follows:

[0024]

[0025] in, The attention score between node i in the l-th layer of GAT and its neighboring node j; Let be the attention score between node i and itself in the l-th layer of GAT; W is the learnable parameter matrix of the l-th layer of GAT. Let i be the feature vector of node i in the (l+1)th layer of GAT. This represents the summation of all neighboring nodes of node i; the final text embedding is represented as... The final visual embedding representation is The final phase-level consensus score C is obtained. Phase :

[0026]

[0027] According to the multimodal spoofing identification method based on multi-granularity consistency of the present invention, the global-level coarse-grained consistency based on the CLIP model further includes:

[0028] The CLIP model is used to encode coarse-grained features to obtain text feature representations. and visual feature representation The CLIP-encoded features are fused, and a global-level consistency score C is calculated using cosine similarity. Global :

[0029]

[0030] According to the multi-modal spoofing identification method based on multi-granularity consistency of the present invention, the process of solving the aggregated feature vector is further as follows: The normalized weights w... Token w Phase and w Global The consistent feature vectors F of the three levels respectively Token F Phase and F Global Multiply each eigenvector separately, then sum them to obtain the aggregated eigenvector F. Agg :

[0031]

[0032] F Agg =w Token ·F Token +w Phase ·F Phase +w Global ·F Global .

[0033] According to the multi-modal spoofing identification method based on multi-granularity consistency of the present invention, the aggregated feature vector F is further... Agg The input is passed to a two-layer fully connected layer to obtain the predicted label. Finally, the model is optimized by minimizing the cross-entropy loss function:

[0034]

[0035] Where y is the real label. It is a predicted label.

[0036] Furthermore, this invention also proposes a multimodal fake information identification system based on multi-granularity consistency, used to implement the multimodal fake information identification system based on multi-granularity consistency as described above, comprising:

[0037] The dataset construction module is used to collect multimodal information data to form an information dataset. The samples include fake data and non-fake data, and are labeled with corresponding labels 0 and 1.

[0038] The text image initial feature representation module is used to generate text feature representations by using the BERT model as a text encoder, and at the same time, it uses the ViT model as an image encoder to generate visual feature representations of the image.

[0039] The token-level consistency measurement module is used to introduce token-level fine-grained consistency based on a multi-head cross-attention mechanism, generate text and image feature vectors at this level, and perform token-level consistency measurement between text and images.

[0040] The phase-level consistency measurement module is used to introduce granular consistency in the phase-level of graph attention network, generate text and image feature vectors at this level, and perform phase-level consistency measurement between text and image.

[0041] The global-level consistency measurement module is used to introduce global-level coarse-grained consistency based on the CLIP model, generate text and image feature vectors at this level, and perform global-level consistency measurement between text and images.

[0042] The model training module employs a granular adaptive attention mechanism to assign weights to consistent features at each level, resulting in an aggregated feature vector. It optimizes the model by minimizing the cross-entropy loss function and uses the optimized model to detect multimodal false information.

[0043] The beneficial effects achieved by adopting the above technical solution are:

[0044] 1. The purpose of this invention is to detect multimodal misinformation, i.e., to determine the type of information. This invention constructs a three-level granularity multimodal consistency measurement mechanism: fine-grained, medium-grained, and coarse-grained. Considering that each modality has an inherent hierarchical structure, it comprehensively understands the complex semantic relationships between images and text. Specifically, this paper introduces token-level fine-grained consistency based on a multi-head cross-attention mechanism, phase-level medium-grained consistency based on a graph attention network, and global-level coarse-grained consistency based on the CLIP model. This multi-level multimodal feature fusion framework can comprehensively reveal the intrinsic connections between multimodal information, fully consider the semantic matching between text and images at different levels, and effectively improve the detection capability of complex multimodal misinformation.

[0045] 2. This invention utilizes medium-granularity semantic alignment technology in graph attention networks to capture deep semantic dependencies between text and images, measuring phase-level consistency. This method improves the accuracy of multimodal information alignment, further enhancing the precision and accuracy of fake information identification, and providing more reliable technical support for fake information detection.

[0046] 3. This invention employs the CLIP visual language model and a multi-head cross-attention mechanism, which aligns text and image features in the same semantic space, thereby more accurately reflecting the correlation between the two and further improving classification accuracy and detection accuracy.

[0047] 4. To address the challenge of effectively fusing multimodal features across different granularities, this invention introduces a granularity-based adaptive attention mechanism to adaptively allocate weights to consistent features at each level, reducing redundant information and helping to generate more representative feature representations, enabling the classifier to accurately identify false information. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. The drawings are merely illustrative of some embodiments of the present invention and are not intended to limit the scope of the present invention to all embodiments.

[0049] Figure 1 This is a flowchart illustrating the multimodal misinformation identification method based on multi-granularity consistency according to an embodiment of the present invention.

[0050] Figure 2 This is the model framework of the multimodal fake information identification method based on multi-granularity consistency in the embodiments of the present invention;

[0051] Figure 3 This is an example diagram of the three-level structure of images and text in an embodiment of the present invention. Detailed Implementation

[0052] The exemplary solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art.

[0053] like Figures 1 to 3 As shown, the multimodal spoofing identification method based on multi-granularity consistency in this embodiment constructs a more comprehensive multi-level multimodal feature fusion framework by combining a three-level consistency metric, thereby better revealing the complexity of multimodal information. The method includes the following steps:

[0054] Step S101: Collect multimodal information data to form an information dataset. The samples include fake data and non-fake data, and set corresponding labels 0 and 1. Preprocessing of the data includes cleaning and denoising special symbols.

[0055] Step S102: Use the BERT model as the text encoder to generate text feature representation O. T ={t1,…t i …,t n}, where n represents the length of the text tag, t i Representing embedded features; using the ViT model as an image encoder to generate visual feature representations of the image O. V ={v1,…v j …,v m}, where v j This represents the feature of the j-th region of the image.

[0056] Step S103: Introduce token-level fine-grained consistency based on multi-head cross-attention mechanism, generate text and visual feature vectors at this level, and measure the token-level consistency between text and image.

[0057] To accurately assess token-level consistency between text and images, a multi-head cross-attention mechanism is used to map data from both modalities into a common semantic space, enhancing the utilization of multimodal relationships between image regions and sentence lexical units. In this embodiment, the query vector Q is derived from text features O. T The key vector K and value vector V are generated from visual features O. V The calculation formula for the multi-head cross-attention mechanism is as follows:

[0058] MultiHeadCrossAttention(Q,K,V)=Concat(head1,…,head h W O

[0059]

[0060] in, W is the linear transformation matrix of the i-th attention head. O The output transformation matrix is ​​given by h, where h is the number of attention heads, and d is the number of attention heads. k This is the dimension of the query key-value vector; each attention head generates an updated text representation, which is combined and transformed through linear layers to produce the final text feature representation. It combines the original text information with cross-attention information from the image. Based on this, it uses the inner product to measure atomic-level consistency and, through a fully connected layer and a softmax function, finally obtains the token-level consistency score C. Token :

[0061]

[0062] in, v represents the i-th word element in the text. m This represents the j-th image block.

[0063] Step S103: Introduce phase-level granularity consistency based on graph attention network, generate text and visual feature vectors at this level, and measure the phase-level consistency between text and image.

[0064] Graph attention networks are used to model text and images, constructing text and visual graphs to capture the complex structural relationships between the input text and images. Lexicons in the text and image patches are treated as nodes in the graphs, and edges are established through dependencies and geometric adjacency. Graph attention networks enable semantic information to propagate through graph edges, thereby learning a combined representation. The text embedding representation and visual embedding representation are as follows:

[0065]

[0066] in, The attention score between node i in the l-th layer of GAT and its neighboring node j; Let be the attention score between node i and itself in the l-th layer of GAT; W is the learnable parameter matrix of the l-th layer of GAT. Let i be the feature vector of node i in the (l+1)th layer of GAT. This represents the summation of all neighboring nodes of node i; the final phrase-level embedding of the text is represented as... The final phrase-level embedding representation of the image is as follows The final phase-level consensus score C is obtained. Phase :

[0067]

[0068] Step S105: Introduce global-level coarse-grained consistency based on the CLIP model, generate text and visual feature vectors at this level, and perform global-level consistency measurement between text and image.

[0069] The CLIP model is used to encode coarse-grained features to obtain text feature representations. and visual feature representation CLIP-encoded features are fused into robust coarse-grained features that reflect global semantic relevance, and a global-level consistency score C is calculated using cosine similarity. Global The CLIP model's output can effectively eliminate the gap between modes through contrastive learning.

[0070]

[0071] Step S106: Employ a granular adaptive attention mechanism to assign weights to the consistency features at each level, thereby obtaining the aggregated feature vector.

[0072] To aggregate features and consistency scores at different levels, an attention mechanism is employed to adaptively assign weights to consistency features at each level, allowing for flexible adjustment of their relative importance. Specifically, consistency features at three granular levels are treated as single features with three independent channels. Global average pooling is applied to compress the features along the spatial dimension into vectors of length C (C=3), where C represents the number of channels. Each channel feature is compressed into a real number with a certain degree of global receptive field, enabling the generated channel-level statistics to contain contextual information and thus mitigating the channel dependency problem. Subsequently, a multilayer perceptron (MLP) architecture is implemented to learn the weights for each channel, used for reweighting feature mappings to strengthen important information in consistency features while suppressing unimportant information.

[0073] Next, normalize the weights w Token w Phase and w Global The consistent feature vectors F of the three levels respectively Token F Phase and F Global Multiply each eigenvector separately, then sum them to obtain the aggregated eigenvector F. Agg :

[0074]

[0075] F Agg =w Token ·F Token +w Phase ·F Phase +w Global ·F Global .

[0076] Step S07: Optimize the model by minimizing the cross-entropy loss function; use the optimized model to detect multimodal false information.

[0077] Aggregate feature vector F AggThe input is fed into a two-layer fully connected layer, and the second fully connected layer acts as a classifier to obtain the predicted label. Finally, the model is optimized by minimizing the cross-entropy loss function:

[0078]

[0079] Where y is the real label. It is a predicted label.

[0080] Corresponding to the above method, this embodiment also proposes a multimodal fake information identification system based on multi-granularity consistency, comprising:

[0081] The dataset building module is used to collect multimodal information data to form an information dataset. The samples include fake data and non-fake data, and are labeled with corresponding labels 0 and 1.

[0082] The text image initial feature representation module is used to generate text feature representations by using the BERT model as a text encoder, and at the same time, it uses the ViT model as an image encoder to generate visual feature representations of the image.

[0083] The token-level consistency measurement module is used to introduce token-level fine-grained consistency based on a multi-head cross-attention mechanism, generate text and visual feature vectors at this level, and perform token-level consistency measurement between text and images.

[0084] The phase-level consistency measurement module is used to introduce granular consistency in the phase-level of graph attention network, generate text and visual feature vectors at this level, and perform phase-level consistency measurement between text and image.

[0085] The global-level consistency measurement module is used to introduce global-level coarse-grained consistency based on the CLIP model, generate text and visual feature vectors at this level, and perform global-level consistency measurement between text and images.

[0086] The model training module employs a granular adaptive attention mechanism to assign weights to consistent features at each level, resulting in an aggregated feature vector. It optimizes the model by minimizing the cross-entropy loss function and uses the optimized model to detect multimodal false information.

[0087] To verify the effectiveness of the proposed model, experiments were conducted on three datasets to evaluate the performance of the method of this invention.

[0088] Compared to models that simply perform multimodal interaction connections (such as SpotFake+, MVAE, and EANN), models based on collaborative attention mechanisms (HMCAN) and multimodal semantic consistency (MCNN and CAFE) perform better, indicating that collaborative attention mechanisms can better integrate multimodal information, which provides theoretical support for this invention. HMCAN achieves better results than the other methods mentioned above, demonstrating that capturing inter-modal and intra-modal relationships in multimodal data can better integrate multimodal features. This invention further outperforms methods that only consider collaborative attention (HMCAN) and models that only consider multimodal single-granularity semantic consistency (MCNN and CAFE). Compared to recently proposed state-of-the-art methods, the accuracy of this invention reaches 91.9% and 89.6% on the Weibo dataset (as shown in Table 1) and the Phome dataset (as shown in Table 2), respectively, representing improvements of approximately 1.2% and 1.8% in accuracy compared to the current state-of-the-art methods. This result demonstrates the effectiveness and superiority of this invention.

[0089] Table 1

[0090]

[0091] Table 2

[0092]

[0093] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0095] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0096] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0097] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal spoofing identification method based on multi-granularity consistency, characterized in that, Include: Multimodal information data is collected to form an information dataset. The samples include fake data and non-fake data, and corresponding labels 0 and 1 are set. The BERT model is used as the text encoder to generate text feature representations, while the ViT model is used as the image encoder to generate visual feature representations of the images. Introducing a token-level fine-grained consistency mechanism based on multi-head cross-attention to generate text feature vectors at this level. and visual feature vectors The token-level consistency between text and images is measured using inner product, fully connected layers, and the softmax function, resulting in a token-level consistency score. ; Introducing phase-level granularity consistency in graph attention networks to generate text feature vectors for that level. and visual feature vectors Furthermore, the phase-level consistency between text and image is measured using inner product, fully connected layers, and the softmax function, resulting in a phase-level consistency score. ; Introducing global-level coarse-grained consistency based on the CLIP model to generate text feature vectors at this level. and visual feature vectors Furthermore, cosine similarity is used to measure the global-level consistency between text and images, resulting in a global-level consistency score. ; A granular adaptive attention mechanism is adopted to assign weights to consistent features at each level, resulting in an aggregated feature vector; the model is optimized by minimizing the cross-entropy loss function; and the optimized model is used for multimodal false information detection. The process of solving the aggregated feature vector is as follows: normalize the weights... , and Consistency feature vectors at three different levels , and Multiply each component separately, then sum them to obtain the aggregated feature vector. : ; ; ; .

2. The multimodal spoofing identification method based on multi-granularity consistency according to claim 1, characterized in that, The BERT model is used as the text encoder to generate text feature representations. Where n represents the length of the text tag, Represents embedded features; Using the ViT model as an image encoder, visual feature representations of images are generated. ,in This represents the feature of the j-th region of the image.

3. The multimodal spoofing identification method based on multi-granularity consistency according to claim 2, characterized in that, Token-level fine-grained consistency based on multi-head cross-attention mechanism includes: A multi-head cross-attention mechanism is used to map data from two modalities to a common semantic space; the query vector Q is composed of text features. The key vector K and value vector V are generated from visual features. The calculation formula for the multi-head cross-attention mechanism is as follows: ; ; ;in, It is the linear transformation matrix of the i-th attention head. The output transformation matrix is ​​given by h, where h is the number of attention heads. This refers to the dimension of the query key-value vector; each attention head generates an updated text representation, which is combined and transformed through linear layers to produce the final text feature representation. ; The inner product is used to measure atomic-level consistency, and the token-level consistency score is finally obtained through a fully connected layer and a softmax function. : ; ;in, This represents the i-th word element in the text. This represents the j-th image block.

4. The multimodal spoofing identification method based on multi-granularity consistency according to claim 3, characterized in that, Phase-level granularity consistency in graph attention networks includes: Graph attention networks are used to model text and images, with words in the text and image patches as nodes in the graph, and edges established through dependencies and geometric adjacency. The text embedding representation and the visual embedding representation are as follows: ; ;in, The attention score between node i in the l-th layer of GAT and its neighboring node j; Let be the attention score between node i and itself in the l-th layer of GAT; W is the learnable parameter matrix of the l-th layer of GAT. Let i be the feature vector of node i in the (l+1)th layer of GAT. This represents the summation of all neighboring nodes of node i; the final text embedding is represented as... The final visual embedding representation is The final phase-level consensus score is obtained. : .

5. The multimodal spoofing identification method based on multi-granularity consistency according to claim 4, characterized in that, Global-level coarse-grained consistency based on the CLIP model includes: The CLIP model is used to encode coarse-grained features to obtain text feature representations. and visual feature representation The CLIP-encoded features are fused, and a global-level consistency score is calculated using cosine similarity. : .

6. The multimodal spoofing identification method based on multi-granularity consistency according to claim 5, characterized in that, Aggregate feature vectors The input is passed to a two-layer fully connected layer to obtain the predicted label. Finally, the model is optimized by minimizing the cross-entropy loss function. Where y is the real label, It is a predicted label.

7. A multimodal fake information identification system based on multi-granularity consistency, characterized in that, A method for implementing the multimodal spoofing identification method based on multi-granularity consistency as described in any one of claims 1-6, comprising: The dataset construction module is used to collect multimodal information data to form an information dataset. The samples include fake data and non-fake data, and are labeled with corresponding labels 0 and 1. The text image initial feature representation module is used to generate text feature representations by using the BERT model as a text encoder, and at the same time, it uses the ViT model as an image encoder to generate visual feature representations of the image. The token-level consistency measurement module is used to introduce fine-grained token-level consistency based on a multi-head cross-attention mechanism, generating text feature vectors for this level. and visual feature vectors The token-level consistency between text and images is measured using inner product, fully connected layers, and the softmax function, resulting in a token-level consistency score. ; The phase-level consistency measurement module is used to introduce granular consistency into the phase-level of the graph attention network and generate the text feature vector for that level. and visual feature vectors Furthermore, the phase-level consistency between text and image is measured using inner product, fully connected layers, and the softmax function, resulting in a phase-level consistency score. ; The global-level consistency measurement module is used to introduce global-level coarse-grained consistency based on the CLIP model and generate text feature vectors at this level. and visual feature vectors Furthermore, cosine similarity is used to measure the global-level consistency between text and images, resulting in a global-level consistency score. ; The model training module employs a granular adaptive attention mechanism to assign weights to consistent features at each level, resulting in an aggregated feature vector. It optimizes the model by minimizing the cross-entropy loss function and uses the optimized model to detect multimodal false information.

Images