Multi-modal fake news detection method based on multi-level fusion and attention mechanism

Abstract

The application discloses a kind of based on multi-level fusion and new type attention mechanism Multimodal fake news detection method, mainly includes the following steps: S1, input news;S2, construct multimodal fake news detection model;S3, original text and image features are respectively input into the two-way branch of multimodal fake news detection model, respectively obtain modal in local-global interaction feature, modal in local-local self-attention feature, and cross-modal collaborative attention feature;S4, the authenticity classification result R of current news is calculated using the multiple features obtained in S3, R and the authenticity label Y constitute classification loss for constraining the training of entire multimodal fake news detection model.The application adopts novel multi-level cross-modal fusion infrastructure, and proposes a new type of attention mechanism, which makes progress in real scene fine-grained fake news classification task and improves the sensitivity to fake news under the condition of real training sample imbalance.

Description

Technical Field

[0001] This invention belongs to the fields of information security and media regulation, specifically relating to a multimodal fake news detection method based on multi-layered cross-modal fusion and a novel attention mechanism. More specifically, this invention achieves fine-grained fake news classification in real-world scenarios by employing a novel multi-layered cross-modal fusion architecture, and improves the accuracy of fake news detection under imbalanced real-world training sample conditions by using a novel attention mechanism. Background Technology

[0002] With the diversification of social media and digital content, news information is increasingly being disseminated by combining visual content such as images and videos with traditional text. This combination has led to the spread of more deceptive and inflammatory fake news, causing increasingly serious harm to society in recent years. Achieving more granular detection of multimodal fake news, targeting potential fraudulent elements within multimodal information, has become a challenging and urgent issue in the fields of information security and media regulation.

[0003] In recent years, deep learning technology has spurred automated detection methods in fake news detection research, solving the problems of high labor costs, high domain knowledge dependence, and poor model generalization ability caused by early reliance on manually designed linguistic representations to capture text features. Furthermore, as Convolutional Neural Networks (CNNs) have become a powerful tool for processing visual information, they have also achieved significant results in the field of fake news detection. By using CNNs to perform more refined modeling of news data of different modalities, the accuracy and robustness of fake news detection can be improved. This has led to two branches in existing fake news detection methods: unimodal fake news detection and multimodal fake news detection. Unimodal fake news detection methods are entirely based on Natural Language Processing (NLP) methods, focusing only on feature extraction and classification of textual information. However, in the current context where fake media information increasingly relies on multimodal content dissemination, unimodal text detection methods are fundamentally unable to fully characterize the original features and presentation scenarios of fake news. In contrast, multimodal fake news detection methods comprehensively utilize information from multiple modalities such as text and images, and have significant advantages in terms of the comprehensiveness of capturing original news information, multimodal consistency detection, and a richer feature representation space. They have now become a major focus in the field of fake news detection.

[0004] Existing multimodal fake news detection methods only address the interaction between local information within each modality of news information, neglecting the interaction between local and global information. While these methods are simple and effective in detecting numerous cases of multimodal fake news, their lack of consideration for the interaction between local and global information prevents them from effectively integrating features at different granularities. A major method of multimodal news fraud in reality involves making subtle alterations to originally authentic text and images. This leads to semantic discrepancies between the fine-grained local information and the coarse-grained global information in the altered information. Therefore, existing methods struggle to comprehensively cover the data characteristics corresponding to various common types of fraud in real-world multimodal news, and consequently, cannot specifically detect each type of real-world news fraud.

[0005] Furthermore, recent multimodal fake news detection methods often incorporate attention mechanisms to improve performance. Attention mechanisms are widely used in deep learning models to simulate the selective concentration and weighted allocation of attention in human information processing. They analyze the importance of different parts of the input data and assign corresponding attention weights to each part. Most existing multimodal fake news detection methods employ the more basic self-attention mechanism to calculate the correlation between features from different modalities. While this allows the model to more accurately capture highly correlated information in fake news and increase the proportion of consistent representations in the data features, focusing solely on consistent representations can lead to insensitivity to inconsistencies. Since the fundamental characteristic of real-world multimodal news fraud is information mismatch within or between modalities, most existing multimodal fake news detection models that only use self-attention mechanisms struggle to achieve better overall performance across all categories of news fraud in real-world fake news detection tasks. Summary of the Invention

[0006] To address the shortcomings and current state of existing technologies, this invention proposes a multimodal fake news detection method based on a novel feature fusion approach. Specifically, this method incorporates a novel multi-level feature fusion method and a novel attention mechanism. Compared to existing methods that only employ a single-level feature fusion approach using intramodal local information-local information, the multi-level fusion method simultaneously considers both intramodal local information-local information feature fusion and intramodal local information-global information feature fusion. This allows the model to fully utilize the correlation between global and local information within a modality, fundamentally covering the data characteristics corresponding to more common types of fake news in real-world multimodal news. It specifically improves the overall detection performance for various types of fake news in real-world multimodal news, especially image-text mismatch and text / image content tampering. Meanwhile, this invention proposes a novel attention mechanism, namely the inverse attention mechanism. Compared with the self-attention mechanism used in existing methods, which focuses more on the consistency representation between information, the inverse attention mechanism can explicitly learn and intuitively reflect the information mismatch caused by real-world multimodal news fabrication. Compared with existing methods, it can significantly improve the model's overall sensitivity to fake news and enhance interpretability.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A multimodal fake news detection method based on multi-level fusion and attention mechanisms is characterized by: employing an inverse attention mechanism to achieve the fusion of local features within a modality, the fusion of local features with global features, and the fusion of local features between different modalities. The method includes the following steps:

[0009] S1. Obtain multimodal news input, i.e., obtain a set of corresponding news text-image pairs, and obtain the original text features F through a pre-trained feature extractor. t and original image features F i ;

[0010] S2. Construct a multimodal fake news detection model. This model first processes the original text features and original image features through text feature extraction branches and image feature extraction branches respectively, to obtain multiple text fusion features and image fusion features. Then, these fusion features are concatenated and spliced ​​to obtain the final result. The text feature extraction branch and the image feature extraction branch have the same structure. They are both composed of a local-to-local feature fusion module within the modality and a cross-modality feature fusion module connected in series, and then connected in parallel with the local-to-global feature fusion module within the modality. Each module adopts a special attention mechanism, that is, it calculates the self-attention mechanism matrix and the inverse attention mechanism matrix respectively, and then concatenates the two. The inverse attention mechanism refers to multiplying the self-attention weight matrix obtained by multiplying the query vector and the key vector obtained by the self-attention operation by a negative weight and adding it to the positive boundary value. The resulting new weight matrix is ​​then multiplied again by the value vector obtained by the self-attention operation, normalized, and then input into the Softmax function.

[0011] S3, extract the original text features F t and original image features F i Input the intra-modal local-global feature fusion module and the intra-modal local-local feature fusion module of the corresponding modality respectively, and obtain the intra-modal local-global fused features of the text modality. Intramodal local-global fusion features of image modalities and intra-modal local-local fusion features of text modalities Intramodal local-local fusion features of image modalities After that, and Inputting each feature into the corresponding modality's cross-modal feature fusion module yields the cross-modal text feature F, which incorporates image features. t-i and cross-modal image features F that incorporate text features i-t The above six features will be used to calculate the loss and constrain the training process of the network;

[0012] S4. Utilize the intra-modal local-global fusion features of the two modalities. Intramodal local-local fusion features of the two modalities and two cross-modal fusion features F t-i F i-t The current news is classified as true or false, R. R and the authenticity label Y constitute the classification loss, which is used to constrain the training of the entire multimodal fake news detection model.

[0013] Furthermore, step S1 specifically includes:

[0014] Obtain the text data T and image data I from the original news data, and input them into the text feature extractor and image feature extractor respectively to extract the original text features and original image features:

[0015] F t =BERT(T)

[0016] F i =ViT(I)

[0017] Where BERT indicates that the text feature extractor uses a pre-trained BERT model, ViT indicates that the image feature extractor uses a pre-trained Vision Transformer model, and F... t F represents the original text features. i These represent the original image features, and the two are collectively referred to as single-modal original features.

[0018] Furthermore, step S2 specifically includes:

[0019] Each of the text feature extraction branch and the image feature extraction branch has an intra-modal local-global feature fusion module. This module processes the original single-modal features of the input modality into intra-modal local-global features of the corresponding modality. For the text modality, the intra-modal local-global feature fusion module first processes the original text features of the input modality into intra-modal global features and intra-modal local features, then performs element-wise multiplication on the intra-modal global features and intra-modal local features, and then uses the self-attention mechanism and the aforementioned inverse attention mechanism to obtain the self-attention matrix and inverse attention matrix, respectively. Finally, the self-attention matrix and inverse attention matrix are multiplied element-wise with the corresponding original single-modal features and then concatenated to obtain the intra-modal local-global fused features of the text modality.

[0020]

[0021]

[0022]

[0023]

[0024] Where softmax represents the softmax function, and ⊙ represents element-wise multiplication. and Let represent two learnable weight matrices. d represents the average vector obtained by performing average pooling on vector X. k The vector dimension representing the initial text features. The constant hyperparameter is represented by |·|, which represents taking the absolute value of the matrix elements, and Cat represents the concatenation operation; the intramodal local-global feature fusion module of the image modality obtains the intramodal local-global fused features of the image modality. The acquisition method is similar to the intra-modal local-global fusion features of the aforementioned text modality. The acquisition methods are completely identical;

[0025] Each of the text feature extraction branch and the image feature extraction branch has an intra-modal local-local feature fusion module. This module processes the original single-modal features of the corresponding modality into intra-modal local-local feature fusion modules for the corresponding modality. For the text modality, the intra-modal local-local feature fusion module first adds positional encoding to the original text features, and then inputs them into a 6-layer collaborative attention Transformer encoder that simultaneously performs self-attention and inverse attention mechanisms to obtain the intra-modal local-local fusion features of the text modality.

[0026] F t′ =F t +Pos(F t )

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] Where Pos(·) represents the positional encoding of the tensor computation. This indicates a transpose operation on the tensor. Each represents a learnable weight matrix. LN(·) represents a constant hyperparameter, and LN(·) represents layer normalization of the feature tensor. The intramodal local-local feature fusion module of the image modality obtains the intramodal local-local fused features of the image modality. The acquisition method is similar to the intra-modal local-local fusion features of the aforementioned text modality. The acquisition methods are completely identical;

[0033] Each of the text feature extraction branch and the image feature extraction branch has a cross-modal feature fusion module. This module processes the input intra-modal local-local feature fusion module of the corresponding modality into cross-modal fused features of the corresponding modality. For the text modality, the cross-modal feature fusion module of the text modality inputs the input intra-modal local-local fused features of the text modality into a 6-layer collaborative attention Transformer encoder that simultaneously performs self-attention mechanism operation and the inverse attention mechanism operation. The Transformer encoder obtains the cross-modal text feature F that fuses the image features. t-i :

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] in, Both represent learnable weight matrices, A t-i Represents constant hyperparameters; cross-modal image features that incorporate text features. The acquisition method is similar to the cross-modal text feature F that integrates image features mentioned above. t-i The acquisition methods are exactly the same.

[0040] Furthermore, step S3 specifically includes:

[0041] The original text features F t and original image features F i Input the intra-modal local-global feature fusion modules of the corresponding modal feature extraction branches respectively to obtain the intra-modal local-global fused features of the two modalities. and

[0042] The original text features F t and original image features F i The inputs are fed into the intra-modal local-local feature fusion modules corresponding to the respective modal branches to obtain the intra-modal local-local fused features for both modalities. and Then, both are input into the cross-modal feature fusion module of the corresponding modal branch to obtain the cross-modal text feature F that fuses the image features. t-i and cross-modal image features F that incorporate text features i-t ;

[0043] Through training, the model can better learn the information characteristics in multimodal news.

[0044] Furthermore, step S4 specifically includes:

[0045] Intramodal local-global interaction features of the two modalities Intramodal local-local self-attention features of the two modalities after average pooling. and two cross-modal collaborative attention features F t-i F i-t Perform the concatenation and calculate the classification result R:

[0046]

[0047] R = MLP(F)

[0048] Wherein, MLP(·) represents a multilayer perceptron, which is composed of multiple mapping units connected in series. Each mapping unit consists of a linear mapping layer, a ReLU activation function layer, and a Dropout layer connected in series.

[0049] The classification result R and the authenticity label Y constitute the classification loss L, which is used to constrain the training of the entire multimodal fake news detection model. The loss used is the cross-entropy loss, and the specific formula is as follows:

[0050] R′=softmax(R)

[0051] L=-∑Y·logR′

[0052] Here, ∑(·) represents the summation operation over all categories, and log(·) represents the calculation of the natural logarithm; this loss is used to constrain the training process of the entire network model.

[0053] The advantages of this invention are as follows: Compared with the prior art, this invention has the following beneficial effects:

[0054] 1. Compared with existing technologies, this invention proposes a novel multi-level feature fusion method. By explicitly calculating the local-global fusion features within a modality, it specifically improves the model's sensitivity to detecting local-global inconsistencies within a modality caused by single-modal forgery. This can improve the model's detection performance for common forgery types such as content tampering and image-text mismatch.

[0055] 2. Compared with existing technologies, this invention proposes a novel attention mechanism, namely the inverse attention algorithm mechanism, which makes up for the design shortcoming of existing technologies that rely entirely on traditional attention mechanisms to calculate the consistency of multimodal information, making it difficult to specifically detect the inconsistency of multimodal information. This enables the model to deeply learn the information inconsistency in fake news and improve the overall performance of the model in predicting multiple categories of fake news.

[0056] 3. The present invention integrates multi-level fusion within a modality and cross-modal fusion to achieve interpretability of the model corresponding to real-world multimodal fake news classification. It can fully leverage the advantages of the model's explicit structure and combine with a novel attention mechanism to achieve fine-grained fake news classification capabilities that are largely unavailable in existing technologies. Attached Figure Description

[0057] Figure 1 This is a flowchart of the multimodal fake news detection method of the present invention;

[0058] Figure 2 This is a diagram showing the overall network structure of the multimodal fake news detection model of this invention.

[0059] Figure 3 This is a structural diagram of the intramodal local-global feature fusion module of the multimodal fake news detection model of the present invention;

[0060] Figure 4 This is a structural diagram of the intramodal local-local feature fusion module of the multimodal fake news detection model of the present invention;

[0061] Figure 5 This is a structural diagram of the cross-modal feature fusion module of the multimodal fake news detection model of the present invention;

[0062] Figure 6 The introduction of the intramodal local-global feature fusion module and the introduction of inverse attention in each module in this invention affect the performance indicators of multimodal fake news detection.

[0063] Figure 7 The introduction of the intramodal local-global feature fusion module and the introduction of inverse attention in each module in this invention affect the detection accuracy of each fine-grained fake news category;

[0064] Figure 8 This paper compares the feature distribution of the present invention with that of existing methods on authoritative Chinese and English datasets. Detailed Implementation

[0065] To better understand the design scheme proposed in this invention, this section will provide specific implementation steps and illustrate them with experimental results and accompanying drawings. The specific embodiments described herein are only for more clearly illustrating the invention and should not be construed as limiting the scope of protection of the invention.

[0066] See Figure 1 , Figure 2 , Figure 3 and Figure 4 This invention discloses a multimodal fake news detection method based on multi-level fusion and a novel attention mechanism, comprising the following steps:

[0067] S1. Obtain multimodal news input, i.e., obtain a set of corresponding news text-image pairs, and obtain the original text features F through a pre-trained feature extractor. t and original image features F i ;

[0068] S2. Construct a multimodal fake news detection model (e.g.) Figure 2 As shown), the model consists of parallel text feature extraction and image feature extraction branches, each of which comprises three modules, including: an intra-modal local-global feature fusion module (such as...). Figure 3 As shown), intramodal local-local feature fusion module (such as...) Figure 4 As shown), cross-modal feature fusion module (such as...) Figure 5 (as shown);

[0069] S3, extract the original text features F t and original image features F i Input the intra-modal local-global feature fusion module and the intra-modal local-local feature fusion module of the corresponding modality respectively, and obtain the intra-modal local-global fused features of the text modality. Intramodal local-global fusion features of image modalities and intra-modal local-local fusion features of text modalities Intramodal local-local fusion features of image modalities After that, and Inputting each feature into the corresponding modality's cross-modal feature fusion module yields the cross-modal text feature F, which incorporates image features. t-i and cross-modal image features F that incorporate text features i-t The above six features will be used to calculate the loss and constrain the training process of the network;

[0070] S4. Utilize the intra-modal local-global fusion features of the two modalities. Intramodal local-local fusion features of the two modalities and two cross-modal fusion features F t-i F i-t The current news is classified as true or false, R. R and the authenticity label Y constitute the classification loss, which is used to constrain the training of the entire multimodal fake news detection model.

[0071] Step S1 specifically includes:

[0072] Obtain the text data *f* and image data *I* from the original news data, and perform feature extraction using text feature extractors and image feature extractors respectively. The text feature extractor uses a pre-trained BERT model, truncating or padding each piece of text to a uniform length of 196 characters. The image feature extractor uses a pre-trained VisionTransformer model, preprocessing each input image to a 224×224 pixel size and dividing it into 16×16 patches, resulting in 196 patches of the same length as the text characters. Divide the dataset into training and test sets in a 9:1 ratio.

[0073] Step S2 specifically includes:

[0074] like Figure 3 As shown, the intra-modal local-global feature fusion module in the feature extraction branch of each modality is initially composed of two parallel branches: a local feature extraction branch and a global feature extraction branch. The local feature extraction branch contains a feedforward network consisting of a linear layer, a ReLU activation layer, and a Dropout layer connected in series, with a Dropout ratio of 0.5. The global feature extraction branch consists of an average pooling layer and a feedforward network connected in series, with the feedforward network structure being the same as that in the local feature extraction branch. The results from the two branches are then element-wise multiplied and passed through a softmax layer to obtain the attention matrix and inverse attention matrix, respectively. Finally, the attention matrix and inverse attention matrix are element-wise multiplied with the initial input single-modal features and concatenated to obtain the intra-modal local-global fusion feature of the corresponding modality. and

[0075] like Figure 4 As shown, the intramodal local-local feature fusion module for each modality first concatenates the positional encoding layer and the summation layer, then passes through a 6-layer Transformer encoder. Each Transformer encoder first obtains attention triples (query vector, key vector, and value vector) through three parallel linear mapping layers. Then, the attention matrix and inverse attention matrix are obtained sequentially through inner multiplication of the query vector and key vector, normalization, inner multiplication of the value vector, and softmax layers. The attention matrix and inverse attention matrix are then input into parallel feedforward networks. Each network consists of a linear mapping layer, a ReLU activation layer, and a Dropout layer concatenated and repeated twice, with a Dropout ratio of 0.5. Finally, the final attention coefficients and inverse attention coefficients are tensor-concatenated, layer-normalized, and Dropout is applied before being fed back into the feedforward network to obtain intramodal local-local self-attention features for both modalities. This feedforward network has the same structure as the feedforward network into which the attention matrix was previously entered in this module;

[0076] like Figure 5 As shown, the cross-modal feature fusion module for each modality consists of a cascaded 6-layer Transformer encoder employing a multi-head collaborative attention mechanism. In each Transformer encoder, the query vector is first calculated using the intra-modal local-local fusion features of the corresponding modality. Then, the key vector and value vector are calculated using the local-local fusion features of another modality. Subsequently, the query vector and key vector are multiplied together, normalized, multiplied together, and then processed by a softmax layer to obtain the attention matrix and inverse attention matrix, respectively. These matrices are then input into parallel feedforward networks. Each feedforward network consists of a linear mapping layer, a ReLU activation layer, and a Dropout layer, which are concatenated and repeated twice, with a Dropout ratio of 0.5. Finally, the final attention coefficients and inverse attention coefficients are tensor-concatenated, layer-normalized, and Dropout-reduced before being fed back into the feedforward network to obtain the two cross-modal collaborative attention features F. t-i F i-t This feedforward network is the same as the feedforward network into which the attention matrix was previously entered in this module;

[0077] Step S3 specifically includes:

[0078] Based on step S2, a multimodal fake news detection model is constructed, which incorporates the original news text features F t and the original image features of the news F i The inputs are fed into the intra-modal local-global feature fusion module and the intra-modal local-local feature fusion module of the corresponding modality to obtain the intra-modal local-global fused features of the text modality. Intramodal local-global fusion features of image modalities and intramodal local-local fusion features of text modality Intramodal local-local fusion features of image modalities

[0079] Intramodal local-local fusion features of two modalities and The inputs are respectively fed into the cross-modal feature fusion module of the corresponding modality, and the output is the cross-modal text feature F that fuses the image features. t-i and cross-modal image features F that incorporate text features i-t .

[0080] Through training, the model can better learn the information characteristics in multimodal news.

[0081] Step S4 specifically includes:

[0082] The intra-modal local-global fusion features of the two modalities obtained in step S3 are used to fuse the features of the two modalities. and And intra-modal local-local fusion features of the two modes. and The feature vectors obtained by average pooling respectively and And the cross-modal fusion feature F of the two modes t-i and F i-t The data is concatenated and passed through a set of linear mapping layers to obtain the final classification result of the current news. The classification result R and the authenticity label Y are used to calculate the loss, which is the cross-entropy loss. The specific formula is as follows:

[0083] R′=softmax(R)

[0084] L=-∑Y·log R′

[0085] Here, ∑(·) represents the summation operation over all categories, and log(·) represents the calculation of the natural logarithm; this loss is used to constrain the training process of the entire network model.

[0086] Figure 6 This paper demonstrates the impact of introducing the intra-modal local-global feature fusion module and the inverse attention mechanism proposed in this invention on various performance indicators of multimodal fake news detection. It is evident that removing the intra-modal local-global feature fusion module significantly reduces the model's detection accuracy for both real and fake news. Similarly, removing the inverse attention mechanism from a certain module also results in a noticeable decrease in the model's detection accuracy for both real and fake news. This verifies the analysis of this invention, namely that both the intra-modal local-global feature fusion module and the inverse attention mechanism proposed in this invention can improve the overall accuracy of the model in multimodal fake news detection tasks, specifically in detecting both real and fake news.

[0087] Figure 7This paper demonstrates the impact of introducing the intra-modal local-global feature fusion module and the inverse attention mechanism in each module on the detection accuracy of each fine-grained fake news category. It is evident that removing the intra-modal local-global feature fusion module or the inverse attention mechanism in any module does not significantly change the model's detection accuracy for real news, but it significantly reduces the overall detection accuracy for fake news. This verifies that the intra-modal local-global feature fusion module and the inverse attention mechanism proposed in this invention can improve the overall detection accuracy of the model for fake news. In particular, the introduction of the intra-modal local-global feature fusion module and the inverse attention mechanism enables the model to achieve a particularly significant improvement in detection accuracy for image-text mismatch and original content tampering types. This verifies that the intra-modal local-global feature fusion module and the inverse attention mechanism proposed in this invention can, in principle, address both inter-modal and intra-modal inconsistencies in news content.

[0088] Figure 8 This paper presents a comparison of the feature distribution of the proposed method with existing methods on authoritative Chinese and English datasets. It is evident that the proposed method achieves higher similarity compactness and heterogeneity dispersion compared to state-of-the-art methods on both Chinese and English datasets, demonstrating excellent data clustering results and significant class separation. Furthermore, it offers better multimodal fake news detection performance and greater interpretability.

Claims

1. A multimodal fake news detection method based on multi-level fusion and attention mechanisms, characterized in that: This method employs a reverse attention mechanism to achieve the fusion of local features within a modality, the fusion of local features with global features, and the fusion of local features between different modalities. The method includes the following steps: S1. Obtain multimodal news input, i.e., obtain a set of corresponding news text-image pairs, and obtain the original text features F through a pre-trained feature extractor. t and original image features F i ; S2. Construct a multimodal fake news detection model. This model first processes the original text features and original image features through text feature extraction branches and image feature extraction branches respectively, to obtain multiple text fusion features and image fusion features. Then, these fusion features are concatenated and spliced ​​to obtain the final result. The text feature extraction branch and the image feature extraction branch have the same structure. They are both composed of a local-to-local feature fusion module within the modality and a cross-modality feature fusion module connected in series, and then connected in parallel with the local-to-global feature fusion module within the modality. Each module adopts a special attention mechanism, that is, it calculates the self-attention mechanism matrix and the inverse attention mechanism matrix respectively, and then concatenates the two. The inverse attention mechanism refers to multiplying the self-attention weight matrix obtained by multiplying the query vector and the key vector obtained by the self-attention operation by a negative weight and adding it to the positive boundary value. The resulting new weight matrix is ​​then multiplied again by the value vector obtained by the self-attention operation, normalized, and then input into the Softmax function. S3, extract the original text features F t and original image features F i Input the intra-modal local-global feature fusion module and the intra-modal local-local feature fusion module of the corresponding modality respectively, and obtain the intra-modal local-global fused features of the text modality. Intramodal local-global fusion features of image modalities and intra-modal local-local fusion features of text modalities Intramodal local-local fusion features of image modalities After that and Inputting each feature into the corresponding modality's cross-modal feature fusion module yields the cross-modal text feature F, which incorporates image features. t-i and cross-modal image features F that incorporate text features i-t The above six features will be used to calculate the loss and constrain the training process of the network; S4. Utilize the intra-modal local-global fusion features of the two modalities. Intramodal local-local fusion features of the two modalities and two cross-modal fusion features F t-i F i-t The current news is classified as true or false, R. R and the authenticity label Y constitute the classification loss, which is used to constrain the training of the entire multimodal fake news detection model.

2. The multimodal fake news detection method based on multi-level fusion and attention mechanism according to claim 1, characterized in that: Step S1 specifically includes: Obtain the text data T and image data I from the original news data, and input them into the text feature extractor and image feature extractor respectively to extract the original text features and original image features: F t =BERT(T) F i =ViT(I) Where BERT indicates that the text feature extractor uses a pre-trained BERT model, ViT indicates that the image feature extractor uses a pre-trained Vision Transformer model, and F... t F represents the original text features. i These represent the original image features, and the two are collectively referred to as single-modal original features.

3. The multimodal fake news detection method based on multi-level fusion and attention mechanism according to claim 1, characterized in that: Step S2 specifically includes: Each of the text feature extraction branch and the image feature extraction branch has an intra-modal local-global feature fusion module. This module processes the original single-modal features of the input modality into intra-modal local-global features of the corresponding modality. For the text modality, the intra-modal local-global feature fusion module first processes the original text features of the input modality into intra-modal global features and intra-modal local features, then performs element-wise multiplication on the intra-modal global features and intra-modal local features, and then uses the self-attention mechanism and the aforementioned inverse attention mechanism to obtain the self-attention matrix and inverse attention matrix, respectively. Finally, the self-attention matrix and inverse attention matrix are multiplied element-wise with the corresponding original single-modal features and then concatenated to obtain the intra-modal local-global fused features of the text modality. Where softmax represents the softmax function, and ⊙ represents element-wise multiplication. and This represents two learnable weight matrices. d represents the average vector obtained by performing average pooling on vector X. k The vector dimension representing the initial text features. The constant hyperparameter is represented by |·|, which represents taking the absolute value of the matrix elements, and Cat represents the concatenation operation; the intramodal local-global feature fusion module of the image modality obtains the intramodal local-global fused features of the image modality. The acquisition method is similar to the intra-modal local-global fusion features of the aforementioned text modality. The acquisition methods are completely identical; Each of the text feature extraction branch and the image feature extraction branch has an intra-modal local-local feature fusion module. This module processes the original single-modal features of the corresponding modality into intra-modal local-local feature fusion modules for the corresponding modality. For the text modality, the intra-modal local-local feature fusion module first adds positional encoding to the original text features, and then inputs them into a 6-layer collaborative attention Transformer encoder that simultaneously performs self-attention and inverse attention mechanisms to obtain the intra-modal local-local fusion features of the text modality. F t′ =F t +Pos(F t ) Where Pos(·) represents the positional encoding of the tensor computation, (·) T This indicates a transpose operation on the tensor. Both represent learnable weight matrices. LN(·) represents a constant hyperparameter, and LN(·) represents layer normalization of the feature tensor. The intramodal local-local feature fusion module of the image modality obtains the intramodal local-local fused features of the image modality. The acquisition method is similar to the intra-modal local-local fusion features of the aforementioned text modality. The acquisition methods are completely identical; Each of the text feature extraction branch and the image feature extraction branch has a cross-modal feature fusion module. This module processes the input intra-modal local-local feature fusion module of the corresponding modality into cross-modal fused features of the corresponding modality. For the text modality, the cross-modal feature fusion module of the text modality inputs the input intra-modal local-local fused features of the text modality into a 6-layer collaborative attention Transformer encoder that simultaneously performs self-attention mechanism operation and the inverse attention mechanism operation. The Transformer encoder obtains the cross-modal text feature F that fuses the image features. t-i : in, Both represent learnable weight matrices, A t-i Represents constant hyperparameters; cross-modal image features that incorporate text features. The acquisition method is similar to the cross-modal text feature F that integrates image features mentioned above. t-i The acquisition methods are exactly the same.

4. The multimodal fake news detection method based on multi-level fusion and attention mechanism according to claim 1, characterized in that: Step S3 specifically includes: The original text features F t and original image features F i Input the intra-modal local-global feature fusion modules of the corresponding modal feature extraction branches respectively to obtain the intra-modal local-global fused features of the two modalities. and The original text features F t and original image features F i The inputs are fed into the intra-modal local-local feature fusion modules corresponding to the respective modal branches to obtain the intra-modal local-local fused features for both modalities. and Then, both are input into the cross-modal feature fusion module of the corresponding modal branch to obtain the cross-modal text feature F that fuses the image features. t-i and cross-modal image features F that incorporate text features i-t ; Through training, the model can better learn the information characteristics in multimodal news.

5. The multimodal fake news detection method based on multi-level fusion and attention mechanism according to claim 1, characterized in that: Step S4 specifically includes: Intramodal local-global interaction features of the two modalities Intramodal local-local self-attention features of the two modalities after average pooling. and two cross-modal collaborative attention features F t-i F i-t Perform the concatenation and calculate the classification result R: R = MLP(F) Wherein, MLP(·) represents a multilayer perceptron, which is composed of multiple mapping units connected in series. Each mapping unit consists of a linear mapping layer, a ReLU activation function layer, and a Dropout layer connected in series. The classification result R and the authenticity label Y constitute the classification loss L, which is used to constrain the training of the entire multimodal fake news detection model. The loss used is the cross-entropy loss, and the specific formula is as follows: R′=softmax(R) L=-∑Y·log R′ Here, ∑(·) represents the summation operation over all categories, and log(·) represents the calculation of the natural logarithm; this loss is used to constrain the training process of the entire network model.

Images