Multi-modal false news detection method combined with multi-channel causal attention mechanism

Abstract

The invention particularly relates to a multi-mode false news detection method combined with a multi-channel causal attention mechanism, which comprises the following steps of: extracting noun information of a text by adopting a word segmentation tool, detecting a salient object in an image by utilizing a target detection model, and extracting key elements in two modes of the text and the image; and generating an intervention text and an intervention image by masking the text nouns and the image objects one by one. Text features of the text and the intervention text are extracted through a pre-training language model, and image features of the image and the intervention image are extracted through a pre-training image model. A cross-modal cross attention structure is introduced into the shared semantic space, and cross-modal fusion features are generated. And calculating a causal score of each key element, and feeding back the causal score to an attention mechanism to further guide attention distribution. A multi-channel attention network is constructed, and various attention information is dynamically fused through a gating mechanism. Finally, causal-guided fusion features are used as input, and a cross entropy loss function is adopted to carry out classification training and model optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer science and artificial intelligence, in particular to a multi-modal fake news detection method combining multi-channel causal attention mechanism. BACKGROUND

[0002] Multi-modal fake news detection is a research hotspot in the field of artificial intelligence in recent years, with wide development potential and application prospect, and is widely used in scenes such as false content screening of online news platforms, real-time monitoring of network public opinion, and information verification of news media. The existing multi-modal fake news detection methods mostly rely on the modeling of the surface semantic correlation between text and image modalities, often lacking the mining of deep causal relationships between information, making it difficult to identify false content with complex logical misleading and careful disguise, and thus having high misjudgment rate and insufficient explainability in actual application.

[0003] Peng Guangchuan, Wu Fei (Fake news detection method based on cross-modal interaction and feature fusion network. Computer Science, 2024, 51(11): 23-29.) In order to solve the problem of difficult generation of discriminative shared representation between modalities in multi-modal feature fusion, a fake news detection method based on cross-modal interaction and feature fusion network is proposed. The method designs a feature generation module based on supervised contrast learning and a multi-stage cross-modal interaction structure, performs supervised contrast learning within and between modalities at the same time to ensure higher similarity of similar features and lower similarity of different features, and uses consistency attention mechanism to efficiently fuse modality-specific features and interaction features. The model achieves good recognition accuracy on public datasets. However, this method still focuses on similarity measurement between modalities and does not introduce causal mechanism to evaluate the substantial impact of missing key modality features on model judgment results, making it difficult to accurately measure the importance of information and provide causal explanation, thereby limiting the stability and credibility of the model in complex multi-modal scenarios.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The existing multi-modal fake news detection methods often focus on surface semantic similarity or consistency modeling between modalities when selecting text and image features closely related to model decision-making, ignoring the real causal impact of key information in different modalities on the final prediction results, which does not meet the problem requirements and actual application situations of explainability and decision reliability. The present application proposes a multi-modal fake news detection method combining multi-channel causal attention mechanism to improve the recognition efficiency and detection accuracy of key evidence information and enhance the explainability of the system.

[0006] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0007] According to a first aspect of the present invention, a multimodal fake news detection method combining a multichannel causal attention mechanism is provided, the method comprising:

[0008] Obtain text data and image data separately;

[0009] Text data is input into a word embedding layer to extract text nouns and mask them to generate intervention text. The intervention text and the original text are input into a pre-trained language model to extract original text features and intervention text features. The original text features and intervention text features are mapped to a unified dimension through a fully connected layer to output aligned original text features and intervention text features respectively.

[0010] Image data is input into a convolutional neural network layer to extract salient objects in the image and mask them to generate intervention images. The intervention images and original images are input into a pre-trained image model to extract features of the original images and intervention images. The features of the original images and intervention images are mapped to a unified dimension through a fully connected layer and then the aligned features of the original images and intervention images are output respectively.

[0011] The first feature pair, composed of aligned original text features and intervention text features, the second feature pair, composed of aligned intervention text features and original image features, and the third feature pair, composed of aligned intervention image features and original text features, are input into the attention layer. Cross-modal enhancement is performed on the three feature pairs using cross-attention, generating three pairs of enhanced features.

[0012] Three pairs of enhanced features are input into a fully connected layer and fused through a multilayer perceptron (MLP) to obtain the first fused feature generated by the aligned original text features and the original image features, the second fused feature generated by the aligned intervention text features and the original image features, and the third fused feature generated by the aligned intervention image features and the original text features.

[0013] The first fusion feature, the second fusion feature, and the third fusion feature are respectively processed through a multilayer perceptron and a softmax activation function to obtain the prediction probability of the original model, the prediction probability after masking nouns, and the prediction probability after masking salient objects;

[0014] The text noun causal score is calculated based on the original model and the predicted probability after masking the noun; the image salient object causal score is calculated based on the original model and the predicted probability after masking the salient object.

[0015] The causal scores of text nouns and image salient objects are integrated into a causal score matrix, which is then input into a multi-channel attention system for fusion to obtain causal attention, original attention, similarity attention, and inconsistency attention.

[0016] The final features are obtained by fusing causal attention, original attention, similarity attention, and inconsistency attention through a gating mechanism.

[0017] The final features are input into a classifier to obtain the predicted probability of whether the news is true or false.

[0018] In some exemplary embodiments, the step of extracting text nouns from the text data input word embedding layer and masking them to generate intervention text specifically involves:

[0019] The set of nouns N in the text T It is a subset of text T, represented as:

[0020]

[0021] Where, n p Let k represent the p-th noun in the text, k be the number of nouns, and all nouns come from the text T;

[0022] Take the noun n in text T p The corresponding position is replaced with a mask marker, denoted as [MASK], that is:

[0023]

[0024] Among them, T masked,p For the noun n p Intervention text generated by masking.

[0025] In some exemplary embodiments, the step of inputting image data into a convolutional neural network layer to extract salient objects from the image and generating an intervention image by masking them specifically involves:

[0026] The set of salient objects O in the image I It is a subset of image I, represented as:

[0027]

[0028] Among them, o q Let l represent the q-th salient object in the image, l be the number of objects, and all objects come from image I;

[0029] The salient object o in image I q The corresponding position is replaced with a mask marker, denoted as [MASK], that is:

[0030]

[0031] Among them, I masked,q For salient object o q Intervention images generated by masking.

[0032] In some exemplary embodiments, the step of mapping the original text features and the intervention text features to a unified dimension through a fully connected layer to output aligned original text features and intervention text features respectively is as follows:

[0033] h t =W T F T +b T

[0034]

[0035] Among them, h t For the features of the original aligned text, W T , The trainable parameter matrices F for the original text and the intervention text, respectively. T , These are the original text features and the intervened text features, respectively. T , It is a bias term.

[0036] In some exemplary embodiments, the step of mapping the original image features and the intervention image features to a unified dimension through a fully connected layer to output aligned original image features and intervention image features respectively is specifically as follows:

[0037] h i =W I F I +b I

[0038]

[0039] Among them, h i For the aligned features of the original image, W I , The trainable parameter matrices F for the original image and the intervention image, respectively. I , These are the original image features and the intervened image features, respectively. I , It is a bias term.

[0040] In some exemplary embodiments, the step of obtaining the prediction probabilities of the original model, the prediction probabilities after masking nouns, and the prediction probabilities after masking salient objects by passing the first fusion feature, the second fusion feature, and the third fusion feature through a multilayer perceptron and a Softmax activation function, respectively, specifically involves:

[0041] P(y) = Softmax(z1W2 + b2)

[0042] P(y|T masked,p =Softmax(z2W3+b3)

[0043] P(y|I masked,q =Softmax(z3W4+b4)

[0044] Where P(y) represents the predicted probability of the original model, P(y|T) masked,p ) represents the mask noun n p The predicted probability after, P(y|I masked,q ) represents the masked significant object o q The predicted probabilities are W2, W3, W4 and b2, b3, b4, which are the learnable weights and biases of the linear classifier.

[0045] In some exemplary implementations, the calculation of text noun causal scores based on the original model and the predicted probabilities after masking the nouns specifically involves:

[0046] score(n p )=P(y)-P(y|T masked,p )

[0047] Wherein, score(n p () represents the causal score of the nouns in the text.

[0048] In some exemplary embodiments, the calculation of the causal score of salient objects in the image based on the predicted probability after masking the original model and the salient objects specifically involves:

[0049] score(o q )=P(y)-P(y|I masked,q )

[0050] Among them, score(o q () represents the causality score of salient objects in the image.

[0051] In some exemplary embodiments, the integration of text noun causal scores and image salient object causal scores into a causal score matrix, which is then input into a multi-channel attention system for fusion, yields causal attention, original attention, similarity attention, and inconsistency attention. Specifically:

[0052] adjustment=1+αs j

[0053]

[0054] A sim=cosine_sim(Q,K)

[0055] A diff =softmax(|QK|)

[0056] Where adjustment is a causal adjustment matrix constructed based on causal scores, α represents the scaling factor, and A causal A att A sim A diff These are causal attention, original attention, similarity attention, and inconsistency attention, respectively. Q and K represent the query matrix and key matrix of the original features, respectively, and d k It is the feature dimension, cosine_sim represents the cosine similarity calculation function, s j This is a causal score matrix.

[0057] In some exemplary embodiments, the step of fusing causal attention, original attention, similarity attention, and inconsistency attention through a gating mechanism to obtain the final feature is specifically as follows:

[0058] A fused =g1·A causal +g2·A att +g3·A sim +g4·A diff

[0059] g i =Sigmoid(Conv3D([A causal A att A sim A diff ]))

[0060] Among them, A fused This represents the fused attention weight matrix, where g1, g2, g3, and g4 correspond to the gating coefficients of the four attention channels, sigmoid is the activation function, and Conv3D is the 3D convolution operation; the final feature representation z is obtained.

[0061] z = A fused ·V

[0062] Where V represents the value matrix of the original features.

[0063] According to a second aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the multimodal fake news detection method combining a multichannel causal attention mechanism as described in the first aspect.

[0064] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the multimodal fake news detection method combining a multichannel causal attention mechanism as described in the first aspect above.

[0065] According to a fourth aspect of the present invention, an electronic device is provided, comprising:

[0066] Processor; and

[0067] Memory for storing the executable instructions of the processor;

[0068] The processor is configured to implement the multimodal fake news detection method combining a multichannel causal attention mechanism as described in the first aspect above by executing the executable instructions.

[0069] To achieve accurate identification and interpretable modeling of key evidence information in multimodal fake news, this invention proposes a multimodal fusion model that combines a multi-channel causal attention mechanism. It employs text noun extraction and image salient object detection techniques, extracts image and text features using pre-trained language models and pre-trained image models, and completes classification training and optimization by using causal intervention and multi-channel causal attention re-gating weighting methods.

[0070] The method of this invention can identify key evidence information in text and images. Compared with traditional multimodal fake news detection methods, it can more accurately mine the key elements that play a decisive role in the judgment results in different modalities. Through multi-channel causal attention and gating weighting mechanisms, it can effectively improve the model's ability to focus on key information in multiple modalities, thereby significantly improving the accuracy of fake news detection.

[0071] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0073] Figure 1 This is a structural diagram of a multimodal fake news detection method combining a multi-channel causal attention mechanism according to an embodiment of the present invention.

[0074] Figure 2 This is a flowchart of a multimodal fake news detection method combining a multi-channel causal attention mechanism according to an embodiment of the present invention.

[0075] Figure 3(a) shows the test plot of the features before inputting into the model;

[0076] Figure 3(b) shows the test plot of the features after inputting them into the model. Detailed Implementation

[0077] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0078] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0079] This invention proposes a multimodal fake news detection method combining a multi-channel causal attention mechanism, primarily involving causal inference technology and a multimodal deep feature fusion strategy. Causal inference technology is used to identify and quantify the true impact of input elements such as text keywords and salient regions of images on the model's final judgment, helping to discover causal features that play a key role in identifying fake information. The multimodal feature fusion method aims to align and integrate semantic information between heterogeneous modalities, improving the model's ability to consistently model cross-modal content in complex situations. By combining the advantages of causal inference and multimodal feature fusion, this invention not only improves the accuracy of the detection model but also significantly enhances the system's interpretability, possessing broad application prospects and promotional value, and is suitable for multiple AI security application scenarios such as public opinion monitoring, content review, and news credibility assessment.

[0080] To achieve more reasonable and effective multimodal fake news detection, this invention proposes a multimodal fusion model combining a multi-channel causal attention mechanism. For text, word segmentation tools are used to extract noun information, and object detection models are used to detect salient objects in images, extracting key elements in both text and image modalities. Then, intervention text and intervention images are generated by masking text nouns and image objects one by one. Text and intervention texts are processed using a pre-trained language model to extract text features, and images and intervention images are processed using a pre-trained image model to extract image features, with a linear mapping unifying the modal dimension. A cross-modal cross-attention structure is introduced in a shared semantic space to achieve deep coupling and alignment between modalities, generating cross-modal fusion features. Then, the causal score of each key element is calculated and fed back into the attention mechanism to further guide the attention distribution. Based on this, a multi-channel causal attention mechanism is designed, constructing a multi-channel attention network including original attention, causal attention, modal similarity attention, and modal inconsistency attention. A gating mechanism is used to dynamically fuse various attention information, guiding the model to focus more on key text and image elements. Finally, using causal-guided fusion features as input, the cross-entropy loss function is employed for classification training and model optimization.

[0081] This invention introduces a multi-channel causal attention mechanism, which not only improves the model's recognition ability in complex scenarios such as modal conflicts and the presence of interfering information, but also significantly enhances the interpretability and stability of the model's prediction process. The significant advantages of this method in terms of accuracy, F1 score, and robustness have been verified on multiple real-world fake news datasets.

[0082] refer to Figure 2 As shown, the specific steps may include:

[0083] Step 1: Build a multimodal fake news detection algorithm model that combines a multi-channel causal attention mechanism. The model structure is as follows: Figure 1The system comprises 10 layers: one input layer (text and image), one output layer, and eight hidden layers, including one word embedding layer and convolutional neural network layer, one pre-trained model (text and image), two attention layers, and four fully connected layers. The neural network input consists of text and image. First, the word embedding layer and convolutional neural network layer extract nouns from the text and salient objects from the image, respectively. These are then masked to generate intervention text and images, which are input into the pre-trained model (text and image) along with the text and image, with output dimensions of 768 and 1024, respectively. These are then mapped to a unified dimension (512 dimensions) through fully connected layers. The two sets of information pass through one attention layer (without changing dimensions) and then through one fully connected layer, resulting in an output dimension of 128. A preliminary prediction is performed through the fully connected layer to obtain the causal score for each intervention information, which is then fed back to the attention layer. Finally, a fully connected layer performs gated fusion to obtain the final fused feature representation. The output results are classified using a fully connected classifier, with the real labels of the data as the supervision signal. The cross-entropy loss function is used for training and optimization, and the final output of the model is the classification and recognition result of fake news detection.

[0084] The model was trained using gradient descent and backpropagation algorithms, as follows:

[0085] according to Figure 1 A multimodal fake news detection algorithm model combining a multi-channel causal attention mechanism is constructed, and all parameters and weights are initialized with random numbers.

[0086] Text T is a sequence of words, represented as:

[0087] T = {t1, t2, ..., t} n}

[0088] Among them, t i This represents the i-th word in the text, where n is the length of the text.

[0089] Image I is a collection of visual units (such as pixels and region features), represented as:

[0090] I = {i1, i2, ..., i} m}

[0091] Among them, i j Let represent the j-th visual unit in the image, and m be the number of image units.

[0092] For the text, nouns are extracted using a word segmentation tool (word embedding layer). The set of nouns N in the text is... T It is a subset of text T, represented as:

[0093]

[0094] Where, n p Let k represent the p-th noun in the text, k be the number of nouns, and all nouns come from the text T.

[0095] Take the noun n in text T p The corresponding position is replaced with a mask marker, denoted as [MASK], that is:

[0096]

[0097] Among them, T masked,p For the noun n p Intervention text generated by masking.

[0098] For an image, a target detection model (convolutional neural network layer) is used to detect salient objects in the image. The set of salient objects in the image is O. I It is a subset of image I, represented as:

[0099]

[0100] Among them, o q Let l represent the q-th salient object in the image, l be the number of objects, and all objects come from image I.

[0101] The salient object o in image I q The corresponding position is replaced with a mask marker, denoted as [MASK], that is:

[0102]

[0103] Among them, I masked,q For salient object o q Intervention images generated by masking.

[0104] The model is trained on a multimodal news database: First, the original text T and the intervention text T are compared. masked,p The original text features and intervention text features are extracted using a pre-trained language model. Then, the original image I and the intervention image I are combined. masked,q The original image features and the intervention image features are extracted using a pre-trained image model, as shown in the formula:

[0105] F T =BERT(T)

[0106] F Tmasked =BERT(T) masked,p )

[0107] F I =Swin(I)

[0108]

[0109] Among them, F T and F I These are the original text and original image features of the news report, respectively. and These represent the intervention text and intervention image features of the news report, respectively. BERT is a pre-trained language model, and Swin is a pre-trained image model.

[0110] Then, through a linear projection module (fully connected layer), the four high-dimensional features mentioned above are mapped to a unified shared representation space for subsequent cross-modal fusion processing, as follows:

[0111] h t =W T F T +b T

[0112]

[0113] h i =W I F I +b I

[0114]

[0115] Among them, h t and h i These are the original text features and original image features after news alignment, respectively. and These are the intervention text features and intervention image features after news alignment, respectively, W T and W I These are the trainable parameter matrices of the original text and the original image, respectively. and These are the trainable parameter matrices for the intervention text and the intervention image, respectively, b T , b I and It is a bias term.

[0116] (1) After obtaining the aligned features, cross attention is used to apply attention to the three feature pairs: the aligned original text features and the original image features (h). t ,h i Aligned intervention text features and original image features Aligned original text features and intervention image features Cross-modal enhancement is performed to obtain discriminative cross-modal features.

[0117] Using aligned original text features and original image features (h t ,hi For example, for text features in image enhancement, the text feature h t As a query, image feature h i As keys and values. For image features in text enhancement, image feature h i As a query, text feature h t As keys and values. Taking image-enhanced text features as an example, the process is as follows:

[0118]

[0119] in, It is a learnable parameter matrix used for linear transformations, Q i ,K i V i This is the new matrix obtained after the corresponding linear transformation. Then, the attention score and enhanced features are calculated, represented as:

[0120]

[0121] Among them, A att This represents the attention score. It is K i The transpose of d h Q is after linear transformation i ,K i V i The feature dimension is , and Softmax(·) is the activation function. The text features represent image enhancement. Similarly, the image features for text enhancement can be obtained.

[0122] Text and image features are fused using a cross-modal fusion module (fully connected layer). First, they are concatenated, and then passed through a multilayer perceptron (MLP) to obtain the fused features, as shown below: The formula is:

[0123]

[0124] z1 = ReLU(BN(z0W1+b1))

[0125] Where concat represents concatenation, ReLU is a non-linear activation function, BN represents batch normalization, W1 is the weight matrix of the multilayer perceptron, b1 is the bias of the MLP, and z1 represents the aligned original text features and original image features (h t ,h i The final fused features are obtained. Similarly, aligned intervention text features and original image features can be obtained. The final generated fused feature z2, the aligned original text features, and the intervention image features The final generated fusion feature is z3.

[0126] The predicted probabilities are obtained by applying the three fused features to a multilayer perceptron and a softmax activation function, respectively.

[0127] P(y) = Softmax(z1W2 + b2)

[0128] P(y|T masked,p =Softmax(z2W3+b3)

[0129] P(y|I masked,q =Softmax(z3W4+b4)

[0130] Where P(y) represents the predicted probability of the original model, P(y|T) masked,p ) represents the mask noun n p The predicted probability after, P(y|I masked,q ) represents the masked significant object o q The predicted probabilities are W2, W3, W4 and b2, b3, b4, which are the learnable weights and biases of the linear classifier.

[0131] Then, the predicted probability is input to obtain the causal score for each piece of information, the text noun causal score score(n). p The calculation formula is:

[0132] score(n p )=P(y)-P(y|T masked,p )

[0133] Image salient object causality score(o) q The calculation formula is:

[0134] score(o q )=P(y)-P(y|I masked,q )

[0135] After obtaining the causal scores, the causal scores of all text nouns and the causal scores of salient objects in the image are concatenated into a column vector matrix, denoted as the causal score matrix s, in the order of "text part → image part". j The input is then fused using a multi-channel attention mechanism, as shown in the following formula:

[0136] adjustment=1+αs j

[0137]

[0138] A sim =cosine_sim(Q,K)

[0139] A diff =softmax(|QK|)

[0140] Here, adjustment is a causal adjustment matrix constructed based on causal scores, and α represents the scaling factor. causal A att A sim A diff These are causal attention, original attention, similarity attention, and inconsistency attention, respectively. Q and K represent the query matrix and key matrix of the original features, respectively, and d k This refers to the feature dimension, where `cosine_sim` represents the cosine similarity calculation function. Multi-channel attention is fused using a gating mechanism, and is represented as:

[0141] A fused =g1·A causal +g2·A att +g3·A sim +g4·A diff

[0142] g i =Sigmoid(Conv3D([A causal A att A sim A diff ]))

[0143] Among them, A fused This represents the fused attention weight matrix, where g1, g2, g3, and g4 correspond to the gating coefficients of the four attention channels, sigmoid is the activation function, and Conv3D is the 3D convolution operation. Then, the final feature representation z is obtained:

[0144] z = A fused ·V

[0145] Where V represents the value matrix of the original features.

[0146] Finally, a classifier is used to obtain the predicted probability of the news being true or false, as shown in the following formula:

[0147]

[0148] in, y0 indicates that the prediction is fake news, and y1 indicates that the prediction is real news.

[0149] Next, the loss function is calculated, which includes the cross-entropy loss, as shown below:

[0150]

[0151] Where y is the real label, y0 represents fake news, and y1 represents real news.

[0152] The model is trained by backpropagating the loss function.

[0153] Step 2: Using untrained data from the dataset as test instances, a multimodal fake news detection algorithm model incorporating a multi-channel causal attention mechanism is used for identification to obtain the final classification result. The specific process is as follows:

[0154] (1) Text features and image features are extracted by pre-trained language model and pre-trained image model respectively. After being mapped to the shared feature space by the feature projection module, they are input into the cross attention module to generate preliminary fusion features. Based on the preliminary fusion features, feature interaction is performed on the text and image features, and the initial prediction results are output through the fully connected classifier.

[0155] (2) Extract text keywords and image salient regions, construct text mask intervention samples and image occlusion intervention samples, input them into the model for re-prediction, and calculate the difference between the original prediction probability and the intervention prediction probability as the causal score; the causal score is fed back into the attention mechanism to further guide the attention distribution, and various attention information is dynamically fused through a multi-channel attention network and gating mechanism including causal attention, original attention, modal similarity attention and modal inconsistency attention to obtain enhanced features; finally, the enhanced features are fused again and input into the classifier to output the final recognition result.

[0156] Step 1 is the construction of the main model, and Step 1 is the inventive point of this invention.

[0157] The effectiveness of this invention was verified on a publicly available Weibo dataset. The Weibo dataset contains 6858 samples in the training set and 2285 samples in the test set, meeting a training-to-test ratio of 3:1. Each sample includes a Chinese Weibo text and a matching image, labeled as "fake news" or "real news." The evaluation metrics are classification accuracy and F1 score, and a t-test was used for significance testing at a significance level of 0.05. The pre-trained language model and pre-trained image model have 768 and 1024 hidden nodes, respectively. The attention layer has 512 hidden nodes in each model, and the four fully connected layers have 128, 2, 128, and 2 hidden nodes, respectively. The learning rate of the pre-trained model is 1e-5, and the other learning rates are 1e-3. The test results are shown in Figure 3(a) before input to the model. The classification results of the multimodal fake news detection model combined with a multi-channel causal attention mechanism are shown in Figure 3(b). The F1 score is 91.13%, and the classification accuracy is 91.20%, demonstrating the effectiveness of the method described in this invention.

[0158] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0159] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0160] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is defined only by the appended claims.

Claims

1. A multimodal fake news detection method combining a multi-channel causal attention mechanism, characterized in that, include: Obtain text data and image data separately; Text data is input into a word embedding layer to extract text nouns and mask them to generate intervention text. The intervention text and the original text are input into a pre-trained language model to extract original text features and intervention text features. The original text features and intervention text features are mapped to a unified dimension through a fully connected layer to output aligned original text features and intervention text features respectively. Image data is input into a convolutional neural network layer to extract salient objects in the image and mask them to generate intervention images. The intervention images and original images are input into a pre-trained image model to extract features of the original images and intervention images. The features of the original images and intervention images are mapped to a unified dimension through a fully connected layer and then the aligned features of the original images and intervention images are output respectively. The first feature pair, composed of aligned original text features and intervention text features, the second feature pair, composed of aligned intervention text features and original image features, and the third feature pair, composed of aligned intervention image features and original text features, are input into the attention layer. Cross-modal enhancement is performed on the three feature pairs using cross-attention, generating three pairs of enhanced features. Three pairs of enhanced features are input into a fully connected layer and fused through a multilayer perceptron (MLP) to obtain the first fused feature generated by the aligned original text features and the original image features, the second fused feature generated by the aligned intervention text features and the original image features, and the third fused feature generated by the aligned intervention image features and the original text features. The first fusion feature, the second fusion feature, and the third fusion feature are respectively processed through a multilayer perceptron and a softmax activation function to obtain the prediction probability of the original model, the prediction probability after masking nouns, and the prediction probability after masking salient objects; The text noun causal score is calculated based on the original model and the predicted probability after masking the noun; the image salient object causal score is calculated based on the original model and the predicted probability after masking the salient object. The causal scores of text nouns and image salient objects are integrated into a causal score matrix, which is then input into a multi-channel attention system for fusion to obtain causal attention, original attention, similarity attention, and inconsistency attention. The final features are obtained by fusing causal attention, original attention, similarity attention, and inconsistency attention through a gating mechanism. The final features are input into a classifier to obtain the predicted probability of whether the news is true or false.

2. The method according to claim 1, characterized in that, The step of extracting text nouns from the text data input word embedding layer and generating intervention text by masking them is as follows: The set of nouns N in the text T It is a subset of text T, represented as: Where, n p Let k represent the p-th noun in the text, k be the number of nouns, and all nouns come from the text T; Noun n in text T p The corresponding position is replaced with a mask marker, denoted as [MASK], that is: Among them, T masked,p For the noun n p Intervention text generated by masking.

3. The method according to claim 1, characterized in that, The process of inputting image data into a convolutional neural network layer to extract salient objects and then masking them to generate an intervention image specifically involves: The set of salient objects O in the image I It is a subset of image I, represented as: Among them, o q Let l represent the q-th salient object in the image, l be the number of objects, and all objects come from image I; The salient object o in image I q The corresponding position is replaced with a mask marker, denoted as [MASK], that is: Among them, I masked,q For salient object o q Intervention images generated by masking.

4. The method according to claim 1, characterized in that, The process of mapping the original text features and the intervened text features to a unified dimension through a fully connected layer and outputting the aligned original text features and the intervened text features respectively is as follows: h t =W T F T +b T Among them, h t For the features of the original aligned text, W T , The trainable parameter matrices F for the original text and the intervention text, respectively. T , These are the original text features and the intervened text features, respectively. T , It is a bias term.

5. The method according to claim 1, characterized in that, The process of mapping the original image features and the intervention image features to a unified dimension through a fully connected layer and outputting aligned original image features and intervention image features respectively is as follows: h i =W I F I +b I Among them, h i For the aligned features of the original image, W I , The trainable parameter matrices F for the original image and the intervention image, respectively. I , These are the original image features and the intervened image features, respectively. I , It is a bias term.

6. The method according to claim 1, characterized in that, The process of obtaining the prediction probabilities of the original model, the prediction probabilities after masking nouns, and the prediction probabilities after masking salient objects by passing the first fusion feature, the second fusion feature, and the third fusion feature through a multilayer perceptron and a Softmax activation function, respectively, is as follows: P(y) = Softmax(z1W2 + b2) P(y|T masked,p )=Softmax(z2W3+b3) P(y|I masked,q )=Softmax(z3W4+b4) Where P(y) represents the predicted probability of the original model, P(y|T) masked,p ) represents the mask noun n p The predicted probability after, P(y|I masked,q ) represents the masked significant object o q The predicted probabilities are W2, W3, W4 and b2, b3, b4, which are the learnable weights and biases of the linear classifier.

7. The method according to claim 6, characterized in that, The calculation of text noun causal scores based on the predicted probabilities after masking the nouns, specifically, is as follows: score(n p )=P(y)-P(y|T masked,p ) Wherein, score(n p () represents the causal score of the nouns in the text.

8. The method according to claim 6, characterized in that, The calculation of the causal score of salient objects in the image based on the predicted probability after masking the original model and the salient objects is specifically as follows: score(o q )=P(y)-P(y|I masked,q ) Among them, socre(o q () represents the causality score of salient objects in the image.

9. The method according to claim 1, characterized in that, The process involves integrating text noun causal scores and image salient object causal scores into a causal score matrix, which is then input into a multi-channel attention system for fusion to obtain causal attention, original attention, similarity attention, and inconsistency attention. Specifically: adjustment=1+αs j A sim =cosine_sim(Q,K) A diff =softmax(|Q-K|) Where adjustment is a causal adjustment matrix constructed based on causal scores, α represents the scaling factor, and A causal A att A sim A diff These are causal attention, original attention, similarity attention, and inconsistency attention, respectively. Q and K represent the query matrix and key matrix of the original features, respectively, and d k It is the feature dimension, cosine_sim represents the cosine similarity calculation function, s j This is a causal score matrix.

10. The method according to claim 1, characterized in that, The final feature is obtained by fusing causal attention, original attention, similarity attention, and inconsistency attention through a gating mechanism, specifically as follows: A fused =g1·A causal +g2·A att +g3·A sim +g4·A diff g i =Sigmoid(Conv3D([A causal ,A att ,A sim ,A diff ])) Among them, A fused This represents the fused attention weight matrix, where g1, g2, g3, and g4 correspond to the gating coefficients of the four attention channels, sigmoid is the activation function, and Conv3D is the 3D convolution operation; the final feature representation z is obtained. z=A fused ·In Where V represents the value matrix of the original features.

Images