A multi-modal fake news detection method based on a dual-branch cross-modal enhancement of a Transformer and Mamba

Abstract

The application discloses a kind of multi-modal false news detection methods based on Transformer and Mamba double-branch cross-modal enhancement, belong to data processing technical field.Method includes: extracting text feature sequence and image feature sequence of picture-text news;Realize picture-text two-way fine-grained alignment by the cross attention branch of Transformer, and capture intra-modal long-range dependence by Mamba sequence modeling branch;Adaptive integration double-branch output using gate fusion unit, classify in combination with global feature, cross-modal correlation feature and sequence fusion feature, output true or false news detection result.The application improves the accuracy and robustness of false news detection under complex picture-text scene.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a multimodal fake news detection method based on dual-branch cross-modal enhancement, and in particular a dual-branch network structure that integrates the Transformer cross-attention mechanism and the state space model Mamba, for automatic identification of fake news in social multimedia environments. Background Technology

[0002] With the widespread adoption of social media platforms, multimodal fake news presented in a combination of text and images is spreading rapidly globally, seriously impacting social order and public perception. Traditional fake news detection methods are primarily based on a single modality, such as analyzing only text or only images, making it difficult to effectively utilize the complementary information in multimodal data. In recent years, deep learning-based multimodal fake news detection methods have made significant progress, the core of which lies in capturing the semantic connections and inconsistencies between text and images through cross-modal interaction mechanisms.

[0003] Existing multimodal fake news detection methods mostly employ the Transformer architecture to achieve cross-modal interaction. For example, C3N (Cross-modal Content Correlation Network) utilizes multi-layered co-attention modules to bidirectionally enhance text sequences and image region sequences to extract shared information between modalities. However, the computational complexity of the Transformer's attention mechanism increases quadratically with sequence length. Although the sequence is relatively short in this task, stacking multiple layers of attention still introduces a large number of parameters, leading to the model being prone to overfitting on small datasets and limiting its generalization performance. Furthermore, while the attention mechanism excels at capturing global dependencies, its ability to model local fine-grained features is limited.

[0004] In recent years, the Mamba architecture, based on State Space Models (SSMs), has attracted widespread attention as an emerging sequence modeling method due to its linear time complexity and powerful long sequence modeling capabilities. By introducing a selective scanning mechanism based on input dependencies, Mamba can dynamically focus on key information in a sequence, demonstrating performance superior to Transformers in multiple domains. However, research on applying Mamba to multimodal fusion tasks is still in its early stages, especially regarding how to efficiently model intra-sequence contextual dependencies using Mamba while retaining the powerful cross-modal alignment capabilities of Transformers; a mature solution is yet to be found.

[0005] Therefore, there is an urgent need for a multimodal fake news detection method that can combine the advantages of Transformer's cross-modal interaction with Mamba's efficient sequence modeling capabilities to improve detection accuracy and model generalization ability. Summary of the Invention

[0006] To address the problems of existing technologies, this invention proposes a multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement. The method employs CLIP to extract initial features from the text and images, constructs a multi-layered enhancement network consisting of a Transformer cross-attention branch and a Mamba sequence modeling branch, and obtains a comprehensive feature representation through gating fusion, multi-scale correlation modeling, and global feature fusion. After training, it outputs the true / false category detection result for the text and images to be detected, thereby improving the accuracy and robustness of multimodal fake news detection.

[0007] This invention is achieved through the following technical solution: a multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement, comprising the following steps: Step 1: Multimodal Feature Extraction: Using pre-trained CLIP text encoder and CLIP image encoder as feature extractors, the input text and image news is converted into initial text feature sequences and image feature sequences; Part-of-speech tagging is performed on the text to extract nouns. The complete text and the extracted nouns are concatenated to form a text sequence of length N. This sequence is then input into the CLIP text encoder, and the output is the initial feature sequence of the text. ,in It is the length of the text sequence. It refers to the feature dimension, and the formula is as follows:

[0008] The image is processed using Faster R-CNN to extract salient regions and cropped. The complete image and the cropped image are then concatenated to form an image sequence of length M. This sequence is then input into the CLIP image encoder, and the output is the initial feature sequence of the image. ,in It is the length of the image sequence, and the formula is as follows:

[0009] Where CLIP-Text represents the CLIP text encoding function, which is used to map a text sequence to a text feature sequence; CLIP-Image represents the CLIP image encoding function, which is used to map an image sequence to an image feature sequence; textsequence represents the input text sequence, and imagesequence represents the input image sequence.

[0010] The initial input features are:

[0011] Step 2: Multi-layer bi-branch cross-modal enhancement: Construct an L-layer bi-branch cross-modal enhancement module, each layer containing a parallel Transformer cross-attention branch and a Mamba sequence modeling branch, and adaptively fuse the two outputs through a gated fusion unit; perform iterative enhancement layer by layer based on the initial text feature sequence and the initial image feature sequence, and simultaneously achieve bidirectional fine-grained alignment between modalities and long-range context dependency modeling within modalities; Let the input text features of the l-th layer be... Image features are .

[0012] First, layer normalization is applied to both text and image features. Then, a multi-head cross-attention mechanism is used for bidirectional enhancement. The text is used as the query, and the image as the key and value. The text-to-image cross-attention is calculated as follows:

[0013] in , , , For learnable projection matrices, Given the dimensions of each head, the output dimension after concatenating multiple heads is... .

[0014] Similarly, using images as queries and text as keys and values, calculate the cross-attention from image to text:

[0015] in , , .

[0016] in, , These represent the text query matrix and the image query matrix, respectively. , These represent the text key matrix and the image key matrix, respectively. , These represent the text value matrix and the image value matrix, respectively. , These represent the text query projection matrix and the image query projection matrix, respectively. , These represent the text key projection matrix and the image key projection matrix, respectively. , These represent the text value projection matrix and the image value projection matrix, respectively. All of these projection matrices are learnable parameters with a common dimension of [missing information]. MultiHead represents a multi-head attention function, which is used to compute multiple attention heads in parallel and concatenate the results of each head for output.

[0017] The features obtained after bidirectional enhancement are:

[0018]

[0019] in, This represents the intermediate text representation obtained after text features are enhanced across modalities using image information. The image representation is obtained by enhancing image features across modalities with text information; LayerNorm represents the layer normalization function. and These represent the cross-attention functions for text-to-image and image-to-text, respectively.

[0020] Subsequently, each goes through a feedforward network (FFN) and a residual connection:

[0021]

[0022] in, and These represent the text output features and image output features of the Transformer branch after passing through the feedforward network and the residual connection, respectively. and These represent the feedforward network functions for the text branch and the image branch, respectively.

[0023] The FFN consists of two linear layers and one Composition of nonlinear activation functions:

[0024] Wherein, FFN represents the feedforward network function; Linear1 and Linear2 represent the first and second linear mapping layers in FFN, respectively; This represents the modified linear unit activation function, used to introduce nonlinear feature transformations.

[0025] Output of the Transformer branch and Two separate Mamba modules are used to capture long-running dependencies within the modality. For example... Figure 4 As shown, the Mamba module is based on a selective state-space model. Given an input sequence ,in The dimension is , The sequence length of the current modality; when the input is text features. When the input is image features Sequence modeling is performed using the following discretized state-space equations:

[0026]

[0027] in It is in a hidden state, and its dimension is ; For the state dimension; This indicates that the Mamba module is in the... The output at each time step is used to avoid misinterpreting the true label in the loss function. Confusion; and These represent the discretized state transition matrix and input matrix, respectively; The state transition matrix has the following dimensions: ; and For the projection matrix, The dimension is , The dimension is ; The time scale parameter has the following dimensions: All of the above parameters are input. Dynamically generated via linear projection:

[0028] Here, Projection is a linear projection layer used to project the input... Mapped to , and ;enter The dimension is , The dimension is , The dimension is , The dimension is This is used to dynamically generate input parameters for the Mamba module.

[0029] For text features After passing through the Mamba module, the following is obtained:

[0030] For image features ,get:

[0031] in, and These represent the Mamba sequence modeling functions for text branches and image branches, respectively. and These represent the output sequences of text features and image features after being modeled by the Mamba module, respectively.

[0032] like Figure 5 As shown, to dynamically balance the contributions of the two branches, a learnable gating mechanism is introduced. First, the output of the Transformer branch is passed through a gating linear layer to calculate the gating signal:

[0033]

[0034] in For linear layers, This is the Sigmoid function, with an output range of (0,1).

[0035] in, and These represent the gating weight vectors for the text branch and the image branch, respectively; and Let represent the learnable gated linear mapping functions for the text branch and the image branch, respectively, and σ represent the Sigmoid activation function.

[0036] Then, the output of the Mamba branch is scaled using a gating signal and residually fused with the output of the Transformer branch.

[0037]

[0038] in This represents element-wise multiplication. After L layers of processing, the final enhanced text features are obtained. and image features .

[0039] Step 3: Calculate cross-modal correlation features: Based on the final enhanced text features and image features, construct a cosine similarity matrix, and extract cross-modal correlation features through multi-scale one-dimensional convolution and average pooling; Construct a cosine similarity matrix based on the enhanced feature sequences. The (i,j)th element is the cosine similarity between the i-th token feature in the text and the j-th region feature in the image.

[0040] in for The i-th vector, for The j-th vector.

[0041] in, Represents the dot product of vectors; The L2 norm of a vector; Indicates the text number The token and image Cosine similarity between regions.

[0042] Then, multi-scale one-dimensional convolution and average pooling are applied to the similarity matrix to extract multi-scale correlation features:

[0043] in, The output feature of the k-th convolution is... Let k be the k-th one-dimensional convolution function, with a kernel size of . , This represents the size of the k-th convolutional kernel in the sequence dimension, with its width fixed at the length of the image sequence. The output channel count is 64; the outputs of all convolutional kernels are concatenated to obtain the relevance feature vector, where This indicates the number of convolution kernels.

[0044] in, This represents the output feature of the k-th convolution; This represents the size of the k-th convolutional kernel in the sequence dimension; Represents a nonlinear activation function; This represents the average pooling function, used to aggregate the convolution output into relevance features of a fixed dimension.

[0045] Step 4 Global Feature Extraction and Multi-Feature Fusion: Based on the final enhanced text features and final enhanced image features obtained in Step 2, global text features and global image features are extracted respectively. After MLP dimensionality reduction, they are concatenated with the cross-modal correlation features and Mamba sequence fusion features obtained in Step 3 to obtain the final comprehensive feature representation. The features of the first token in the enhanced text feature sequence are taken as the global text features. The features of the first token in the image feature sequence are taken as the global features of the image. The dimensions were reduced to 64 dimensions using fully connected layers:

[0046] in, and These represent the dimensionality-reduced global features of the text and the global features of the image, respectively. and These represent the global text features and global image features extracted from the final enhanced text features and the final enhanced image features, respectively; and These represent the multilayer perceptron dimensionality reduction functions for global text features and global image features, respectively.

[0047] The correlation features are also reduced to 64 dimensions using a fully connected layer:

[0048] in, This represents the original cross-modal correlation feature vector obtained in step 3; This represents the cross-modal correlation features output by the correlation feature dimensionality reduction network; A multilayer perceptron dimensionality reduction function representing correlation features.

[0049] At the same time, the enhanced text feature sequence and image feature sequences By concatenating along the length dimension, a cross-modal sequence is obtained. The sequence is then fed into a lightweight Mamba module (Mamba-Fusion) for sequence-level fusion, and the output of the first token is taken as the sequence fusion feature. :

[0050] in, This represents the sequence fusion features after cross-modal sequences have undergone lightweight Mamba fusion; This represents a lightweight Mamba sequence fusion function; This represents a cross-modal sequence obtained by concatenating text feature sequences and image feature sequences along the length dimension.

[0051] The four feature vectors are concatenated to obtain the final fused feature representation used for classification. Final comprehensive features The dimension is In this embodiment ,therefore :

[0052] in, This represents the final comprehensive feature representation used for classification; , , and These represent the text global features, image global features, correlation features, and sequence fusion features that are concatenated into the comprehensive features, respectively; the semicolon ";" indicates a vector concatenation operation.

[0053] Step 5: Model Training: Input the comprehensive features into the classifier and perform a linear mapping. Use the Softmax function to calculate the probability distribution of news in the true and false categories, and use the cross-entropy loss function to train the model. Combine regularization strategies to improve generalization ability.

[0054] Fusion features Inputting the fully connected layer yields the probability distribution for binary classification prediction of news authenticity. :

[0055] in, This represents the normalization exponential function, used to convert the classifier output into a true / false class probability distribution; The function representing the classifier is the multilayer perceptron. This represents the comprehensive feature representation input to the classifier; This represents the probability distribution of the predicted true and false categories. and These represent the probability of real news and the probability of fake news, respectively.

[0056] in, For classifiers, a multilayer perceptron, This represents the probability distribution of whether a news story is true or false.

[0057] The cross-entropy loss function is used for training to correctly predict the truth or falsehood polarity of news, as shown in the following formula:

[0058] Where B is the total number of training samples, y is the true label, and p is the predicted probability. Dropout regularization, weight decay, and learning rate decay strategies are introduced during training to prevent overfitting.

[0059] Dropout represents a random deactivation regularization method used to randomly disable some neurons during training to reduce the risk of overfitting; weight decay is used to limit the size of model parameters; and learning rate decay is used to gradually adjust the training step size.

[0060] in, This represents the value of the cross-entropy loss function; Indicates the total number of training samples; Indicates the training sample index; This represents a category index, used in binary classification tasks. ; Indicates the first The sample at the th Real labels in each category; Indicates the first The sample at the th Predicted probabilities for each category; This represents the predicted probability distribution output by the classifier.

[0061] Step 6: Fake News Detection: Input the image and text news to be detected into the trained model, and sequentially perform multimodal feature extraction, bi-branch cross-modal enhancement, cross-modal correlation calculation, and multi-feature fusion to obtain the probability distribution of true and false categories. The category with the highest probability is output as the detection result.

[0062] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention introduces a dual-branch parallel structure into multimodal fake news detection. By leveraging the complementary advantages of Transformer and Mamba, it solves the problems of redundant parameters and overfitting inherent in traditional Transformer, and the insufficient cross-modal fusion applications of Mamba. Simultaneously, the introduction of a gated fusion mechanism enables dynamic weighting of the outputs from the two branches, avoiding noise that may be introduced by simple addition. The addition of multi-scale correlation features and sequence fusion features comprehensively characterizes the correlation and differences between images and text from multiple dimensions. Furthermore, Mamba's linear time complexity maintains high performance while controlling the model parameter size, significantly enhancing the model's generalization ability on small datasets and improving the accuracy and robustness of multimodal fake news detection. Attached Figure Description

[0063] Figure 1 This is a system architecture diagram of a multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement. Figure 2 This is a model structure diagram of a multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement. Figure 3 This is a schematic diagram of the dual-branch cross-modal enhancement module structure; Figure 4 This is a schematic diagram of the internal structure of a Mamba module; Figure 5 This is a schematic diagram of a gating fusion unit; Figure 6 This is a schematic diagram of the multi-feature fusion and classification module. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0065] A multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement, the process is as follows: Figure 1 As shown, the overall model structure is as follows Figure 2 As shown; among them, the dual-branch cross-modal enhancement module is as follows: Figure 3 As shown, the internal structure of the Mamba module is as follows: Figure 4 As shown, the gating fusion unit is as follows Figure 5 As shown, the multi-feature fusion and classification module is as follows: Figure 6 As shown.

[0066] 1. Multimodal feature extraction.

[0067] The task of this invention is defined as: for a given multimodal news containing text content and image content, predict the probability that the news is real news or fake news.

[0068] First, a pre-trained CLIP text encoder and CLIP image encoder are used as feature extractors to convert the input text and image news into initial text feature sequences and image feature sequences. Part-of-speech tagging is performed on the text to extract nouns. The complete text and the extracted nouns are concatenated to form a text sequence of length N=31. This sequence is then input into the CLIP text encoder, and the output is the initial feature sequence of the text. Where 512 is the feature dimension of CLIP, and the formula is as follows:

[0069] Five salient regions were extracted from the image using Faster R-CNN and cropped. The complete image was then concatenated with the five cropped images to form an image sequence of length M=6. This sequence was then input into the CLIP image encoder, and the output was the initial feature sequence of the image. The formula is as follows:

[0070] Where CLIP-Text represents the CLIP text encoding function, which is used to map a text sequence to a text feature sequence; CLIP-Image represents the CLIP image encoding function, which is used to map an image sequence to an image feature sequence; textsequence represents the input text sequence, and imagesequence represents the input image sequence.

[0071] The initial input features are:

[0072] 2. Multi-layer bi-branch cross-modal enhancement.

[0073] Construct a two-branch cross-modal augmentation network with L=8 layers. For example... Figure 2 and Figure 3 As shown, the input to each layer is the text features output from the previous layer. and image features .

[0074] (1) Transformer cross attention branch

[0075] First, text and image features are normalized using a layer, then bidirectional enhancement is performed using an 8-head cross-attention mechanism. The text is used as the query, and the image as the key and value; the text-to-image cross-attention is calculated as follows:

[0076] in , , , The learning projection matrix has a dimension of 64 for each head, and the output dimension is 512 after concatenating 8 heads.

[0077] Similarly, using images as queries and text as keys and values, calculate the cross-attention from image to text:

[0078] in , , .

[0079] in, , These represent the text query matrix and the image query matrix, respectively. , These represent the text key matrix and the image key matrix, respectively. , These represent the text value matrix and the image value matrix, respectively. , These represent the text query projection matrix and the image query projection matrix, respectively. , These represent the text key projection matrix and the image key projection matrix, respectively. , These represent the text value projection matrix and the image value projection matrix, respectively. All of these projection matrices are learnable parameters with a common dimension of [missing information]. MultiHead represents a multi-head attention function, which is used to compute multiple attention heads in parallel and concatenate the results of each head for output.

[0080] The features obtained after bidirectional enhancement are:

[0081]

[0082] in, This represents the intermediate text representation obtained after text features are enhanced across modalities using image information. This represents the intermediate image representation obtained after image features are enhanced cross-modal with textual information. Indicates the layer normalization function; and These represent the cross-attention functions for text-to-image and image-to-text, respectively.

[0083] Subsequently, each goes through a feedforward network (FFN) and a residual connection:

[0084]

[0085] in, and These represent the text output features and image output features of the Transformer branch after passing through the feedforward network and the residual connection, respectively. and These represent the feedforward network functions for the text branch and the image branch, respectively.

[0086] The FFN consists of two linear layers and a ReLU nonlinear activation function, with the intermediate layer having a dimension of . .

[0087] Wherein, FFN represents the feedforward network function; Linear1 and Linear2 represent the first and second linear mapping layers in FFN, respectively; ReLU represents the modified linear unit activation function, used to introduce nonlinear feature transformation.

[0088] (2) Mamba sequence modeling branch

[0089] Output of the Transformer branch and Two separate Mamba modules are used to capture long-running dependencies within the modality. For example... Figure 4 As shown, the state dimensions of the Mamba module The expansion factor is 2. Given the input sequence... ,in The dimension is , The sequence length of the current modality; when the input is text features. When the input is image features Sequence modeling is performed using the following discretized state-space equations:

[0090]

[0091] in It is in a hidden state, and its dimension is ; This indicates that the Mamba module is in the... The output at each time step is used to avoid misinterpreting the true label in the loss function. Confusion; and These represent the discretized state transition matrix and input matrix, respectively; The state transition matrix has the following dimensions: ; and For the projection matrix, The dimension is , The dimension is ; The time scale parameter has the following dimensions: All of the above parameters are input. Dynamically generated via linear projection:

[0092] Here, Projection is a linear projection layer used to project the input... Mapped to , and ;enter The dimension is , The dimension is , The dimension is , The dimension is This is used to dynamically generate input parameters for the Mamba module.

[0093] For text features After passing through the Mamba module, the following is obtained:

[0094] For image features ,get:

[0095] in, and These represent the Mamba sequence modeling functions for text branches and image branches, respectively. and These represent the output sequences of text features and image features after being modeled by the Mamba module, respectively.

[0096] (3) Gated fusion unit

[0097] like Figure 5 As shown, to dynamically balance the contributions of the two branches, a learnable gating mechanism is introduced. First, the output of the Transformer branch is passed through a gating linear layer to calculate the gating signal:

[0098]

[0099] in For linear layers, This is the Sigmoid function, with an output range of (0,1).

[0100] in, and These represent the gating weight vectors for the text branch and the image branch, respectively; and Let represent the learnable gated linear mapping functions for the text branch and the image branch, respectively, and σ represent the Sigmoid activation function.

[0101] Then, the output of the Mamba branch is scaled using a gating signal and residually fused with the output of the Transformer branch.

[0102]

[0103] in This represents element-wise multiplication. After 8 layers of processing, the final enhanced text features are obtained. and image features .

[0104] 3. Calculation of cross-modal correlation features.

[0105] Construct a cosine similarity matrix based on the enhanced feature sequences. The (i,j)th element is the cosine similarity between the i-th token feature in the text and the j-th region feature in the image.

[0106] in for The i-th vector, for The j-th vector.

[0107] in, Represents the dot product of vectors; The L2 norm of a vector; Indicates the text number The token and image Cosine similarity between regions.

[0108] Then, three one-dimensional convolutional kernels of different sizes (1×6, 2×6, 3×6) and average pooling are applied to the similarity matrix to extract multi-scale correlation features:

[0109] Each convolutional kernel has 64 output channels. The outputs of the three convolutional kernels are concatenated to obtain the correlation feature vector. Here, 1×6, 2×6, and 3×6 represent the kernel dimensions of 1, 2, and 3 in the sequence dimension, respectively, with the width fixed to the image sequence length. .

[0110] in, This represents the k-th one-dimensional convolution function, used to extract local cross-modal correlation patterns in the sequence dimension of the similarity matrix; Represents a nonlinear activation function; This represents the average pooling function, used to aggregate the convolution output into relevance features of a fixed dimension.

[0111] 4. Global feature extraction and multi-feature fusion.

[0112] like Figure 6 As shown, the features of the first token in the enhanced text feature sequence are taken as the global text features, and the features of the first token in the image feature sequence are taken as the global image features. Dimensionality is reduced to 64 dimensions using a two-layer MLP for each feature.

[0113] in, and These represent the dimensionality-reduced global features of the text and the global features of the image, respectively. and These represent the global text features and global image features extracted from the final enhanced text features and the final enhanced image features, respectively; and These represent the multilayer perceptron dimensionality reduction functions for global text features and global image features, respectively.

[0114] The relevance features are also reduced to 64 dimensions using a two-layer MLP:

[0115] in, This represents the original cross-modal correlation feature vector obtained in step 3; This represents the cross-modal correlation features output by the correlation feature dimensionality reduction network; A multilayer perceptron dimensionality reduction function representing correlation features.

[0116] Simultaneously, the enhanced text feature sequence and image feature sequence are concatenated along the length dimension to obtain a cross-modal sequence. This sequence is then input into a lightweight Mamba module (Mamba-Fusion, expand=1) for sequence-level fusion, and the output of its first token is taken as the sequence fusion feature.

[0117] in, This represents the sequence fusion feature after cross-modal sequences are fused using lightweight Mamba; MambaFusion represents the lightweight Mamba sequence fusion function. This represents a cross-modal sequence obtained by concatenating text feature sequences and image feature sequences along the length dimension.

[0118] The four feature vectors are concatenated to obtain the final fused feature representation used for classification. Final comprehensive features The dimension is In this embodiment ,therefore :

[0119] in, This represents the final comprehensive feature representation used for classification; , , and These represent the text global features, image global features, correlation features, and sequence fusion features that are concatenated into the comprehensive features, respectively; the semicolon ";" indicates a vector concatenation operation.

[0120] 5. Model training.

[0121] Fusion features Input a three-layer MLP classifier to obtain the probability distribution of binary classification prediction for news authenticity. :

[0122] in, This represents the normalization exponential function, used to convert the classifier output into a true / false class probability distribution; The function representing the classifier is the multilayer perceptron. This represents the comprehensive feature representation input to the classifier; This represents the probability distribution of the predicted true and false categories. and These represent the probability of real news and the probability of fake news, respectively.

[0123] in, This represents the probability distribution of whether a news story is true or false.

[0124] The cross-entropy loss function is used for training to correctly predict the truth or falsehood polarity of news, as shown in the following formula:

[0125] Where B is the total number of training samples, y is the true label, and p is the predicted true / false probability. During training, Dropout (p=0.3), weight decay (1e-5), and cosine learning rate decay strategies are introduced to prevent overfitting. The Adam optimizer is used for parameter updates, and the initial learning rate is 2e-5.

[0126] Dropout represents the random deactivation regularization method. This indicates a random inactivation probability of 0.3; Adam represents the adaptive moment estimation optimizer; weight decay is used to limit the size of model parameters, and cosine learning rate decay is used to gradually adjust the learning rate during training.

[0127] in, This represents the value of the cross-entropy loss function; Indicates the total number of training samples; Indicates the training sample index; This represents a category index, used in binary classification tasks. ; Indicates the first The sample at the th Real labels in each category; Indicates the first The sample at the th Predicted probabilities for each category; This represents the predicted probability distribution output by the classifier.

[0128] During the training phase, a labeled public Weibo fake news dataset was used. The difference between the predicted and the true labels was calculated using the cross-entropy loss function, and all learnable parameters were updated using the backpropagation algorithm. An early stopping strategy was adopted during training: training was stopped when the accuracy on the validation set no longer improved for 5 consecutive epochs.

[0129] 6. Fake news detection.

[0130] During the inference phase, the news article to be detected is input into the trained model. After feature extraction, bi-branch cross-modal enhancement, cross-modal correlation calculation, and multi-feature fusion, the classifier outputs the probability distribution of real and fake news, and the category with the highest probability is taken as the final detection result. For example... Figure 6 As shown, the result is output by the multi-feature fusion and classification module.

Claims

1. A multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement, characterized in that, Includes the following steps: Step 1: Multimodal Feature Extraction: Using pre-trained CLIP text encoder and CLIP image encoder as feature extractors, the input text and image news is converted into initial text features and image features; Step 2: Multi-layer bi-branch cross-modal enhancement: Construct an L-layer bi-branch cross-modal enhancement module, each layer containing a parallel Transformer cross-attention branch and a Mamba sequence modeling branch, and adaptively fuse the two outputs through a gated fusion unit; perform iterative enhancement layer by layer based on the initial text feature sequence and the initial image feature sequence, achieve bidirectional fine-grained alignment between modalities, and capture long-range contextual dependencies within modalities; Step 3: Calculate cross-modal correlation features: Based on enhanced text and image features, construct a cosine similarity matrix, and extract cross-modal correlation features through multi-scale one-dimensional convolution and average pooling. Step 4 Global Feature Extraction and Multi-Feature Fusion: Based on the final enhanced text features and final enhanced image features obtained in Step 2, global text features and global image features are extracted respectively. After MLP dimensionality reduction, they are concatenated with the cross-modal correlation features and Mamba sequence fusion features obtained in Step 3 to obtain the final comprehensive feature representation. Step 5: Model Training: Input the comprehensive feature representation into the classifier, pass it through a fully connected layer and the Softmax function to obtain the probability distribution of true and false classes, train the model using the cross-entropy loss function, and optimize the model parameters by combining Dropout regularization, weight decay and learning rate decay strategies. Dropout represents a random deactivation regularization method, which is used to randomly disable some neurons during training to reduce the risk of overfitting; weight decay is used to limit the size of model parameters; and learning rate decay is used to gradually adjust the training step size. Step 6: Fake News Detection: Input the news article to be detected into the trained model, obtain the probability distribution of true and false categories through forward propagation, and output the category with the highest probability as the detection result.

2. The multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement according to claim 1, characterized in that, In step 1, the specific method is as follows: Part-of-speech tagging is performed on the text to extract nouns. The complete text and the extracted nouns are concatenated to form a text sequence of length N. This sequence is then input into the CLIP text encoder, and the output is the initial feature sequence of the text. Where N is the length of the text sequence and d is the feature dimension, the formula is as follows: The image is processed using Faster R-CNN to extract salient regions and cropped. The complete image and the cropped image are then concatenated to form an image sequence of length M. This sequence is then input into the CLIP image encoder, and the output is the initial feature sequence of the image. Where M is the length of the image sequence, the formula is as follows: Where CLIP-Text represents the CLIP text encoding function, which maps a text sequence to a text feature sequence; CLIP-Image represents the CLIP image encoding function, which maps an image sequence to an image feature sequence; textsequence represents the input text sequence, and imagesequence represents the input image sequence; The initial input features are: in, This represents the initial text feature sequence input to the first-layer bi-branch cross-modal enhancement module, and ; This represents the initial image feature sequence input to the first-layer bi-branch cross-modal enhancement module, and N represents the length of the text sequence, M represents the length of the image sequence, and d represents the feature dimension.

3. The multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement according to claim 1, characterized in that, In step 2, the specific method is as follows: Let the input text features of the l-th layer be... Image features are ; Step 2.1 Transformer Cross Attention Branch First, layer normalization is applied to the text and image features. Then, a multi-head cross-attention mechanism is used for bidirectional enhancement, with text as the query and images as the key and value. The text-to-image cross-attention is calculated as follows: in , , , For learnable projection matrices, The dimension for each head is d; after concatenating multiple heads, the output dimension remains d. Using images as queries and text as keys and values, calculate the cross-attention from image to text: in , , ; in, , These represent the text query matrix and the image query matrix, respectively. , These represent the text key matrix and the image key matrix, respectively. , These represent the text value matrix and the image value matrix, respectively. , These represent the text query projection matrix and the image query projection matrix, respectively. , These represent the text key projection matrix and the image key projection matrix, respectively. , These represent the text value projection matrix and the image value projection matrix, respectively. All of these projection matrices are learnable parameters with a common dimension of [missing information]. MultiHead represents a multi-head attention function, used to compute multiple attention heads in parallel and concatenate the results of each head for output. The features obtained after bidirectional enhancement are: in, This represents the intermediate text representation obtained after text features are enhanced across modalities using image information. The image representation is obtained by enhancing image features across modalities with textual information; LayerNorm represents the layer normalization function. and These represent the text-to-image and image-to-text cross-attention functions, respectively. Subsequently, each goes through a feedforward network (FFN) and a residual connection: in, and These represent the text output features and image output features of the Transformer branch after passing through the feedforward network and the residual connection, respectively. and These represent the feedforward network functions for the text branch and the image branch, respectively; The FFN consists of two linear layers and one Composition of nonlinear activation functions: Wherein, FFN represents the feedforward network function; Linear1 and Linear2 represent the first and second linear mapping layers in FFN, respectively; This represents the modified linear unit activation function, used to introduce nonlinear feature transformations; Step 2.2 Mamba Sequence Modeling Branch Output of the Transformer branch and The input sequence is fed into two separate Mamba modules to capture long-program column dependencies within the modality. The Mamba modules are based on a selective state-space model. ,in The dimension is , The sequence length of the current modality; when the input is text features. When the input is image features Sequence modeling is performed using the following discretized state-space equations: in It is in a hidden state, and its dimension is ; For the state dimension; This indicates that the Mamba module is in the... The output at each time step is used to avoid misinterpreting the true label in the loss function. Confusion; and These represent the discretized state transition matrix and input matrix, respectively; The state transition matrix has the following dimensions: ; and Let be the projection matrix. The dimension is , The dimension is ; The time scale parameter has the following dimensions: All of the above parameters are input. Dynamically generated via linear projection: Here, Projection is a linear projection layer used to project the input... Mapped to , and ;enter The dimension is , The dimension is , The dimension is , The dimension is This is to dynamically generate the input parameters for the Mamba module; For text features After passing through the Mamba module, the following is obtained: For image features ,get: in, and These represent the Mamba sequence modeling functions for text branches and image branches, respectively. and These represent the output sequences of text features and image features after being modeled by the Mamba module, respectively. Step 2.3 Gating Fusion Unit A learnable gating mechanism is introduced to dynamically balance the contributions of the two branches. First, the output of the Transformer branch is passed through a gating linear layer to calculate the gating signal: in For linear layers, This is the Sigmoid function, with an output range of (0,1). in, and These represent the gating weight vectors for the text branch and the image branch, respectively; and Let represent the learnable gated linear mapping functions for the text branch and the image branch, respectively, and σ represent the Sigmoid activation function; Then, the output of the Mamba branch is scaled using a gating signal and residually fused with the output of the Transformer branch. in This represents element-wise multiplication, which, after being processed through L layers, yields the final enhanced text features. and image features .

4. The multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement according to claim 1, characterized in that, In step 3, the specific method is as follows: Construct a cosine similarity matrix based on the enhanced feature sequences. The (i,j)th element is the cosine similarity between the i-th token feature in the text and the j-th region feature in the image. in for The i-th vector, for The j-th vector; in, Represents the dot product of vectors; The L2 norm of a vector; Indicates the text number The token and image Cosine similarity between regions; Then, multi-scale one-dimensional convolution and average pooling are applied to the similarity matrix to extract multi-scale correlation features: in, The output feature of the k-th convolution is... Let k be the k-th one-dimensional convolution function, with a kernel size of . , This represents the size of the k-th convolutional kernel in the sequence dimension, with its width fixed at the length of the image sequence. The output channel count is 64; the outputs of all convolutional kernels are concatenated to obtain the relevance feature vector, where Indicates the number of convolution kernels; in, This represents the output feature of the k-th convolution; This represents the size of the k-th convolutional kernel in the sequence dimension; Represents a nonlinear activation function; This represents the average pooling function, used to aggregate the convolution output into relevance features of a fixed dimension.

5. The multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement according to claim 1, characterized in that, In step 4, the specific method is as follows: The features of the first token in the enhanced text feature sequence are taken as the global text features. The features of the first token in the image feature sequence are taken as the global features of the image. The dimensions were reduced to 64 dimensions through fully connected layers: in, and These represent the dimensionality-reduced global features of the text and the global features of the image, respectively. and These represent the global text features and global image features extracted from the final enhanced text features and the final enhanced image features, respectively; and These represent the multilayer perceptron dimensionality reduction functions for global text features and global image features, respectively. The correlation features are also reduced to 64 dimensions using a fully connected layer: in, This represents the original cross-modal correlation feature vector obtained in step 3; This represents the cross-modal correlation features output by the correlation feature dimensionality reduction network; A dimensionality reduction function for a multilayer perceptron representing correlation features; At the same time, the enhanced text feature sequence and image feature sequences By concatenating along the length dimension, a cross-modal sequence is obtained. The sequence is input into a lightweight Mamba module for sequence-level fusion, and the output of the first token is taken as the sequence fusion feature. : in, This represents the sequence fusion features after cross-modal sequences have undergone lightweight Mamba fusion; This represents a lightweight Mamba sequence fusion function; This represents a cross-modal sequence obtained by concatenating text feature sequences and image feature sequences along the length dimension. The four feature vectors are concatenated to obtain the final fused feature representation used for classification. Final comprehensive features The dimension is : in, This represents the final comprehensive feature representation used for classification; , , and These represent the text global features, image global features, correlation features, and sequence fusion features that are concatenated into the comprehensive features, respectively; the semicolon ";" indicates a vector concatenation operation.

6. The multimodal fake news detection method based on Transformer and Mamba dual-branch cross-modal enhancement according to claim 1, characterized in that, The specific methods in steps 5 and 6 are as follows: During the training phase, the fused feature F is input into the classifier to obtain the probability distribution for binary classification prediction of news authenticity: in, This represents the normalization exponential function, used to convert the classifier output into a true / false class probability distribution; The function representing the classifier is the multilayer perceptron. This represents the comprehensive feature representation input to the classifier; This represents the probability distribution of the predicted true and false categories. and These represent the probability of real news and the probability of fake news, respectively. in, For classifiers, a multilayer perceptron, This represents the probability distribution for predicting whether a news story is true or false. The model is trained using the cross-entropy loss function, enabling it to correctly learn the boundary between real and fake news. The formula is as follows: Where B is the total number of training samples. Let i be the true label of the i-th sample. Let be the predicted probability of the true category corresponding to the i-th sample; during training, Dropout regularization, weight decay, and learning rate decay strategies are introduced to prevent overfitting; in the detection stage, the image and text news to be detected is input into the trained model, the true and false category probability distribution is output, and the category with the highest probability value is determined as the fake news detection result; Dropout represents a random deactivation regularization method, which is used to randomly disable some neurons during training to reduce the risk of overfitting; weight decay is used to limit the size of model parameters; and learning rate decay is used to gradually adjust the training step size. in, This represents the value of the cross-entropy loss function; Indicates the total number of training samples; Indicates the training sample index; This represents a category index, used in binary classification tasks. ; Indicates the first The sample at the th Real labels in each category; Indicates the first The sample at the th Predicted probabilities for each category; This represents the predicted probability distribution output by the classifier.

Images