False news detection method combining attention and image-text consistency matching

Abstract

The invention provides false news detection combining attention and image-text consistency matching, and the method comprises the following steps: extracting text features and image features by using BERT and Swindow-Transform pre-training models respectively, and obtaining frequency domain features through discrete Fourier transform; the extracted text features and image features are sent to an image-text consistency matching module for semantic alignment; the frequency domain features and the image features are sent to a frequency domain fusion module to obtain fused feature representation; and performing a cross-modal attention mechanism on the text features and the fusion features to obtain final feature representation, and sending the final feature representation into a classifier to realize news authenticity identification. According to the method and the device, multi-mode false news with image tampering and image text description ambiguity can be identified, and high-reliability and high-precision false news detection is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal fake news detection, in particular to a fake news detection method combining attention and text-image consistency matching. BACKGROUND

[0002] Existing fake news detection technology has gone through three stages: the first stage is the early manual verification and manual feature extraction method. Researchers extract shallow features such as text sentiment polarity and user forwarding frequency, and combine traditional machine learning classification such as Naive Bayes and Support Vector Machine (SVM). This method relies on manual feature design, is low in efficiency and difficult to adapt to large-scale scenarios, and does not fully utilize the essential differences between fake and real news in terms of transmission motivation, so the detection capability is limited.

[0003] The second stage is the single-modal deep learning method. After the breakthrough of deep learning technology, researchers use neural networks to automatically extract high-order features: such as capturing text temporal dependence through recurrent neural networks (RNN), or introducing graph convolutional networks (GCN) to model document-word graph structure association. The single-modal detection effect is significantly improved. However, this method is designed for a single modality and is difficult to deal with complex deception patterns such as tampered images and text-image semantic mismatch, which has obvious limitations.

[0004] The third stage is the multi-modal feature fusion method, which has promoted new progress in detection technology in recent years: some researchers have verified the effectiveness of multi-modal methods by fusing text, image and background features through attention mechanisms; some researchers have proposed an enhanced adversarial neural network (EANN) to enhance the generalization ability, or a variational autoencoder (MVAE) to jointly optimize feature reconstruction and classification; after the rise of the Transformer architecture, multi-modal pre-training models such as CLIP and ViT improve text-image representation ability through cross-modal alignment, and some researchers combine BERT and ELECTRA to extract text features and introduce lightweight NasNet Mobile to extract image features to reduce speed.

[0005] However, existing multi-modal methods have two major limitations: first, the deep semantic mining of text features and image features is insufficient. Traditional text models (LSTM, GRU) are difficult to capture implicit positions and emotions, and BERT-like pre-training models lack explicit modeling of sentiment polarity and intensity. Visual models (ResNet, VGG) are easily affected by local noise, and ViT lacks frequency domain information utilization, making it difficult to detect frequency domain artifacts of tampered images, forming a blind area. Second, the multi-modal fusion mechanism is not perfect, and simple feature concatenation or shallow attention interaction is often used, lacking fine-grained alignment and semantic consistency modeling, and being easily disturbed by noise; for text-image mismatch news, explicit similarity modeling is lacking, and subtle semantic differences cannot be distinguished, so the detection performance needs to be improved.

[0006] In view of the above problems, the application provides a fake news detection model combining attention and text-image consistency matching, and provides a high-precision and high-reliability fake news detection model. By extracting features of three different modalities, cross-modal attention mechanism fusion is realized, and a text-image matching consistency module and a frequency domain fusion module are combined to better learn and fuse semantic information of different modalities. SUMMARY

[0007] Therefore, the embodiments of the present application aim to provide a fake news detection method combining attention and text-image consistency matching, to solve the problems of insufficient deep semantic mining of text features and image features and imperfect multi-modal fusion mechanism in the background art.

[0008] To achieve the above technical purposes, the technical solutions adopted by the present application are as follows:

[0009] The present application provides a fake news detection method combining attention and text-image consistency matching,

[0010] The method comprises:

[0011] Step (1): using Swin-Transformer pre-training model and BERT pre-training model and CNN model to extract image features, text features and frequency domain features, using Swin-Transformer pre-training model to extract image features, using BERT pre-training model to extract text features, and using CNN-based network model to extract frequency domain features;

[0012] Step (2): realizing semantic alignment of image features and text features through a text-image consistency matching module, and realizing fusion of image features and frequency domain features through a frequency domain fusion module;

[0013] Step (3): passing the text features and the fused image features and frequency domain features through two consecutive cross-modal attention mechanisms to realize twice feature fusion, and obtaining final feature representation;

[0014] Step (4): inputting the final feature output by step 3 into a classifier to predict the authenticity of the news.

[0015] Further, the step (1) comprises:

[0016] (1) Text feature extraction: a pre-training model BERT is used to obtain an aggregated sequence representation as a text feature, represented as:

[0017]

[0018] wherein, is the token sequence input to the BERT model, is the word embedding matrix; a one-dimensional convolution kernel of different sizes is used to extract the phrase-level semantics of each word, and the set of phrase-level information is projected into word features through a fully connected layer, i.e. ;

[0019] (2) Image feature extraction: a Swin-Transformer pre-training model is used to extract image features. Swin-Transformer uses a shift window mechanism and hierarchical feature representation, and the feature representation is:

[0020]

[0021] wherein is the region feature matrix, is the number of regions in the image, is the dimension of the extracted region features;

[0022] (3): Frequency domain feature extraction: a CNN-based subnetwork is used to extract features from the frequency domain. The spatial domain image is converted to the frequency domain through discrete cosine transform (DCT) to obtain 64 vectors , and all vectors are sampled to a fixed size of 250; in order to facilitate parallel computing, 64 250-dimensional vectors are combined into a feature matrix , and then the feature matrix is input into the CNN-based network; the CNN-based network includes a main network and two branch networks; the main network uses a traditional frequency domain feature extraction network, the first layer uses 32 channels with a size of 3*3, the second uses 64 channels with a size of 3*3, and the third layer uses 128 channels with a size of 3*3; After the maximum pooling layer, it is sent to the subnetwork; the subnetwork is designed with a high-frequency and low-frequency branch, which is used to extract features of different attention points; the high-frequency branch network uses a 3*3 convolution kernel to focus on capturing local details; the low-frequency branch network uses a 5*5 convolution kernel to capture the overall structure through a larger receptive field; each convolution block is composed of a two-dimensional convolution layer, batch normalization and ReLU activation function; finally, the image feature representation in the frequency domain is obtained .

[0023] Further, the image-text consistency matching module and the frequency domain fusion module:

[0024] (1) The input of the text-image consistency matching module is text features and image features. The text features are processed by four convolution layers with different kernel sizes to obtain four local and global information with different attention degrees. Then the four features are spliced into one feature, which is further processed by a perception machine to map to the same shared feature space as the image features. After the image features and the text features are aligned, a classifier is used to calculate whether they are similar. The gradient is calculated using the cosine loss function to optimize the model. The loss function is:

[0025]

[0026] wherein, is the final image feature obtained after processing, is the text feature, is the threshold value.

[0027] (2) The input of the frequency domain fusion module is image features and frequency domain features. This module uses a single-head attention mechanism to achieve cross-modal interaction. The image features are processed by a convolution layer to obtain the corresponding Q value, while the frequency domain features are used as K and V values for attention calculation to obtain the fused feature Z.

[0028] The formula is as follows:

[0029]

[0030]

[0031] wherein , , is the relevant weight matrix, is the column number of the Q and K matrices, i.e., the vector dimension. The attention weight is calculated by the attention score to obtain the final fused output feature :

[0032]

[0033]

[0034] wherein is the attention weight matrix, is the value vector.

[0035] Further, the cross-modal attention mechanism realizes two times of fusion:

[0036] The cross-modal attention mechanism adopts a structure similar to that of a standard attention network, but is divided into two branches, and information interaction is achieved by exchanging the key-value pairs of the visual branch and the language branch.

[0037] (1) Cross-modal query-key-value generation mechanism: the visual branch generates a query matrix , simultaneously outputs a visual key matrix , and a value matrix ; the language branch generates a query matrix , simultaneously outputs a language key matrix , and a value matrix .

[0038] (2) Cross-modal attention interaction calculation: using the query matrix of the visual branch , the key matrix of the language branch , the attention matching weight is calculated by “dot product + scaling + Softmax”; then the value matrix of the language branch is weighted and summed through the weight, realizing the attention and fusion of the visual modality to the language modality information;

[0039] Similarly, the language branch also completes cross-modal attention calculation by its own query , the key matrix , and the value matrix of the visual branch, realizing bidirectional information interaction between modalities.

[0040] (3) Residual connection and layer normalization feature preservation and stabilization mechanism: the results of cross-modal attention calculation are connected with the original input features of the corresponding modality to preserve the basic information of the original features; then the features after residual connection are subjected to layer normalization;

[0041] (4) Feature expression enhancement mechanism of feedforward network: the features after residual connection + layer normalization are input into the feedforward neural network to enhance the expression ability of the model to complex cross-modal correlation through nonlinear transformation;

[0042] (5) Secondary residual connection and layer normalization: through secondary residual and normalization, the feature integrity is further ensured, the output distribution is stabilized, and the visual features fused with text information , the text features fused with visual information are obtained.

[0043] (6) Finally, the visual features and the text features are spliced to realize the fusion of text features and image features, and then the final feature C is obtained.

[0044] Further, the false news detection:

[0045] The multi-modal fusion feature As input, the probability of the authenticity of the news is predicted by two layers of full connection layers of the classifier, and the calculation formula is as follows:

[0046]

[0047]

[0048] wherein, , respectively represent the weight matrix and the bias of the input layer to the hidden layer, is the output of the first layer full connection layer; , respectively are the weight and the bias of the hidden layer to the output layer, is the final binary classification probability.

[0049] The application adopting the above technical scheme has the following advantages:

[0050] 1. The image space-frequency domain feature joint extraction architecture is designed, the Swin-Transformer and the CNN architecture are integrated, the image space domain structure information and the frequency domain artifact features are captured respectively, the blank of the insufficient utilization of the frequency domain information of the tampered image by the existing model is filled, and more comprehensive support is provided for the image feature detection of the fake news.

[0051] 2. The multi-layer cross-modal multi-head attention fusion network architecture is proposed, the semantic association of the text and the image is gradually enhanced through multi-level attention interaction, different modal features are fully exchanged in the depth fusion process, and the foundation is provided for cross-modal semantic understanding.

[0052] 3. In particular, the text-image consistency matching module is introduced, the semantic alignment of the text and the image is explicitly optimized by means of shared frozen features, the inconsistency of the text-image semantics in the fake news can be accurately captured, and the recognition performance of the model on this kind of cheating mode is further improved.

[0053] 4. Compared with the existing multi-modal fake news detection method, the model of the application fully excavates the multi-dimensional features of the image space domain and the frequency domain, and strengthens the semantic association through deep cross-modal interaction and consistency matching, and performs better in the detection accuracy and robustness of the fake news. BRIEF DESCRIPTION OF DRAWINGS

[0054] The present application can be further illustrated by the non-limiting embodiments shown in the accompanying drawings. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be considered as limiting the scope, and other related drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0055] Figure 1 The overall flowchart in the present application is shown in the accompanying drawings.

[0056] Figure 2 The overall network model architecture diagram in the present application is shown in the accompanying drawings.

[0057] Figure 3 The frequency domain feature extractor diagram in the present application is shown in the accompanying drawings.

[0058] Figure 4 The frequency domain fusion module diagram in the present application is shown in the accompanying drawings.

[0059] Figure 5 The text-image consistency matching module diagram in the present application is shown in the accompanying drawings.

[0060] Figure 6 The cross-modal attention mechanism diagram in the present application is shown in the accompanying drawings. DETAILED DESCRIPTION

[0061] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that in the accompanying drawings or description, similar or identical parts are denoted by the same reference numerals, and the implementation not shown or described in the accompanying drawings is in the form known to those skilled in the art. In the description of the present application, the terms "first", "second", etc. are only used for differentiation and cannot be understood as indicating or implying relative importance.

[0062] Please refer to Figures 1-6 The present embodiment provides a false news detection method combining attention and text-image consistency matching, Figure 1 The specific implementation flowchart is shown in the accompanying drawings, and the method comprises:

[0063] Step (1): Use Swin-Transformer and BERT pre-training models and CNN-based network models to extract image features, text features and frequency domain feature representations, respectively;

[0064] As shown in Figure 2 , the image of the news sample is input into the Swin-Transformer image feature extractor, and the image is input into the CNN subnetwork after Fourier transform to extract the frequency domain feature, and the text is input into the BERT feature extractor;

[0065] Specifically, text feature extraction: the text feature extractor BERT is adopted, BERT is based on the Transformer architecture, which can capture long-distance context associations within a sentence using self-attention mechanisms, which allow the model to focus on any position in the sentence, not just local context. The text content of the news is represented as a sequence of word groups, denoted as where is the number of words in the tweet, and each word is segmented by a pre-prepared vocabulary, and BERT is used to obtain the required aggregated sequence representation as a text feature, denoted as:

[0066]

[0067] where is the token sequence input to the BERT model, is the word embedding matrix. Different sizes of one-dimensional convolution kernels are used to extract the phrase-level semantics of each word. Then, this set of phrase-level information is projected into word features through a fully connected layer, i.e. .

[0068] Image feature extraction: the Swin-Transformer pre-training model is used to extract image features, which combines the local information capture ability of traditional CNN and the global dependency capture ability based on Transformer through the shift window mechanism and hierarchical feature representation, reducing the computational complexity, and the feature representation is:

[0069]

[0070] where is the region feature matrix, is the number of regions in the image, is the dimension of the extracted region features.

[0071] Frequency domain feature extraction: as shown in Figure 3 , a CNN-based subnetwork is used to extract features from the frequency domain, and the spatial domain image is converted from the spatial domain to the frequency domain through discrete cosine transform (DCT), obtaining 64 vectors , and all vectors are sampled to a fixed size of 250. In order to facilitate parallel computing, the 64 250-dimensional vectors are combined into a feature matrix , and then input the feature matrix into a CNN-based network; the CNN-based network is composed of a main network and two branch networks; the main network adopts a conventional frequency domain feature extraction network, the first layer uses a 32-channel 1, 3*3 convolution kernel, the second uses a 64-channel 32, 3*3 convolution kernel, and the third layer uses a 128-channel 64, 3*3 convolution kernel, and is sent to the subnetwork through the maximum pooling layer. The subnetwork is designed with a high-frequency and low-frequency branch, which is used to extract features of different attention points. The high-frequency branch network uses a 3*3 convolution kernel to focus on capturing local details. The low-frequency branch network uses a 5*5 convolution kernel to capture the overall structure through a larger receptive field. Each convolution block is composed of a two-dimensional convolution layer, batch normalization and ReLU activation function. Finally, the frequency domain features are obtained .

[0072] Step (2): The image features and frequency domain features extracted are fused by using a frequency domain fusion module to fully fuse the image low-frequency and high-frequency features and the semantics of the original image.

[0073] As shown in Figure 4 , the input of the frequency domain fusion module is the image features and the frequency domain features. This module achieves cross-modal interaction with the help of a single-head attention mechanism, processes the image features through a convolution layer to obtain the corresponding Q value, and uses the frequency domain features as the K and V values for attention calculation to obtain the fused features Z;

[0074] The formula is as follows:

[0075]

[0076]

[0077] wherein , , is the relevant weight matrix, is the column number of the Q and K matrices, i.e., the vector dimension. The attention weight is calculated through the attention score to obtain the final fused output feature :

[0078]

[0079]

[0080] wherein is the attention weight matrix, is the value vector.

[0081] Step (3): The consistency of the text features and the image features is deeply mined by using the graphic-text consistency matching module, so as to enhance the ability of the model to identify fake news.

[0082] As shown in Figure 5 , the inputs are text features and image features respectively. The text features are processed by four convolutional layers with different kernel sizes, so as to obtain four local and global information with different attention degrees. Then, the four features are spliced into one feature, and then are processed by a perception machine for dimension reduction, and are mapped to the same shared feature space as the image features. After the image features and the text features are aligned, whether they are similar is calculated by using a classifier, and the gradient is calculated by using a cosine loss function, so as to optimize the model.

[0083] The loss function is:

[0084]

[0085] wherein, is the final image feature obtained after processing, is the text feature, is a threshold value.

[0086] Through the above method, the model can explicitly learn the semantic relationship between the text and the image, and the features of the matching samples are pulled in, the consistency is enhanced, and the unmatching samples are pulled away, so that the model has better distinguishability.

[0087] Step (4): The image features and the text features are deeply fused twice by using a multi-layer cross-modal attention mechanism.

[0088] As shown in Figure 6 , the cross-modal attention mechanism adopts a structure similar to the standard attention network, but is divided into two branches. Information interaction is achieved by exchanging the key-value pairs of the visual branch and the language branch.

[0089] Cross-modal query, key and value generation mechanism: the visual branch generates a query matrix , simultaneously outputs a visual key matrix and a value matrix ; the language branch generates a query matrix , simultaneously outputs a language key matrix and a value matrix .

[0090] Cross-modal attention interaction calculation: the query matrix of the visual branch and the key matrix of the language branch are used.The attention matching weight is calculated by "dot product + scaling + Softmax", and then the value matrix of the language branch is multiplied by the weight The weighted sum is calculated to realize the attention and fusion of visual modal information to language modal information

[0091] Similarly, the language branch also queries its own key matrix and value matrix of the visual branch to complete the cross-modal attention calculation and realize the bidirectional information interaction between modalities.

[0092] Residual connection and layer normalization feature preservation and stabilization mechanism: the result of cross-modal attention calculation is connected with the original input features of the corresponding modality to preserve the basic information of the original features; then the features after residual connection are normalized

[0093] Feature expression enhancement mechanism of feedforward network: the features after residual connection and layer normalization are input into the feedforward neural network to enhance the expression ability of the model to complex cross-modal relationships through nonlinear transformation

[0094] Second residual connection and layer normalization: through the second residual and normalization, the feature integrity is further ensured, the output distribution is stabilized, and the visual features fused with text information and the text features fused with visual information are obtained.

[0095] Finally, the two features are spliced to realize the fusion of text features and image features, and then the final feature is obtained.

[0096] Step (5): false news detection by classifier, taking the multi-modal fusion feature as input, the probability of predicting the authenticity of the news is calculated through two fully connected layers of the classifier, and the calculation formula is as follows:

[0097]

[0098]

[0099] wherein, , and are the weight matrix and bias from the input layer to the hidden layer, respectively , are the weight and bias from the hidden layer to the output layer, respectively is the final binary classification probability.

[0100] Through the technical solutions, the text semantic information and the image semantic information required in the fake news detection can be effectively extracted. Through the frequency domain fusion module, the frequency domain features and the image features are implemented deep semantic fusion, the high frequency semantic information and the low frequency semantic information are distinguished, and the effective identification of the model on the image tampering is enhanced. Secondly, through the image-text consistency matching module, the image features and the text features are mapped to the same semantic space, the model can explicitly learn the semantic relationship between the text and the image, the features of the matching samples are pulled closer to each other, the consistency is enhanced, the features of the non-matching samples are pulled away from each other, and thus the distinguishing ability of the model is improved.

[0101] Finally, through the multi-layer cross-modal multi-head attention fusion network architecture, the deep fusion of the text features and the image features is implemented, so that the model can better learn the cross-modal semantics of the image and the text. Through the cooperative operation of these technical means, the ability of the model to understand, analyze and accurately estimate the complex image and text in the fake news detection is greatly enhanced.

[0102] The above merely describes the embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for fake news detection by combining attention and text-image consistency matching, characterized in that, The method comprises: Step (1): using a Swin-Transformer pre-training model to extract image features, using a BERT pre-training model to extract text features, and using a CNN-based network model to extract frequency domain features; Step (2): aligning the semantics of the image features and the text features through a text-image consistency matching module, and fusing the image features and the frequency domain features through a frequency domain fusion module; Step (3): passing the text features and the fused image and frequency domain features through two consecutive cross-modal attention mechanisms to realize twice feature fusion and obtain the final feature representation; Step (4): inputting the final feature output by step 3 into a classifier to predict the authenticity of the news.

2. The method of claim 1, wherein the method further comprises: The step (1) comprises: (1) text feature extraction: a pre-training model BERT is used to obtain an aggregated sequence representation as a text feature, represented as: X t = Bert(w1, w2, ···, wn) (1) n ) (1) wherein w1, w2, ···, w n is a token sequence input to the BERT model, is a word embedding matrix; a one-dimensional convolution kernel of different sizes is used to extract the phrase-level semantics of each word, and the set of phrase-level information is projected into word features through a fully connected layer, that is, T={t1, t2, ···, t N}∈R N×d ; (2) image feature extraction: a Swin-Transformer pre-training model is used to extract image features, and the Swin-Transformer realizes shift window mechanism and hierarchical feature representation, and the feature representation is: X v = Swin-Trans(R) (2) wherein is a region feature matrix, M is the number of regions in the image, d v is the dimension of the extracted region features; (3): Frequency domain feature extraction: a subnetwork based on CNN is used to extract features from the frequency domain, the spatial domain image is converted to the frequency domain through discrete cosine transform (DCT), then 64 vectors {H0, H1, ···, H 63} are obtained, and all vectors are sampled to a fixed size of 250; in order to facilitate parallel calculation, the 64 250-dimensional vectors are combined into a feature matrix H F ∈R 64 ×250 , and then the feature matrix is input into the network based on CNN; the network based on CNN includes a main network and two branch networks; the main network uses a traditional frequency domain feature extraction network, the first layer uses 32 convolution kernels with a channel number of 1 and a size of 3*3, the second layer uses 64 convolution kernels with a channel number of 32 and a size of 3*3, and the third layer uses 128 convolution kernels with a channel number of 64 and a size of 3*3, and is sent to the subnetwork through the maximum pooling layer; the subnetwork designs a high-frequency and low-frequency branch, which is respectively used to extract features of different attention points; the high-frequency branch network uses 3*3 convolution kernels and focuses on capturing local details; the low-frequency branch network uses 5*5 convolution kernels to capture the overall structure through a larger receptive field; each convolution block is composed of a two-dimensional convolution layer, batch normalization and ReLU activation function; finally, the image feature representation X f ∈R d×l in the frequency domain is obtained.

3. The method of claim 2, wherein the method further comprises: The semantic alignment is realized by using a text-image consistency matching module, and the feature fusion is realized by using a frequency domain fusion module, which comprises: The input of the text-image consistency matching module is the text feature and the image feature, the text feature is processed by four convolutional layers with different kernel sizes to obtain four local features with different attention degrees and a global feature, then the four features are spliced into one feature, and then a perception machine is used for dimension reduction processing to map to the same shared feature space as the image feature; after the image feature and the text feature are aligned, a classifier is used to calculate whether they are similar, and a cosine loss function is used to calculate the gradient to optimize the model, and the loss function is: wherein E v ' is the final image feature obtained after processing, E t ' is the text feature, and d is a threshold value. The frequency domain fusion module inputs the image feature and the frequency domain feature, and realizes cross-modal interaction by using a single-head attention mechanism, processes the image feature by a convolutional layer to obtain the corresponding Q value, and uses the frequency domain feature as the K and V values for attention calculation to obtain the fused feature Z; The formula is as follows: Q = W q X v K = W k X f V = W V X f (4) where W q , W k , W v are the relevant weight matrices, d k is the number of columns of the Q, K matrices, i.e., the vector dimension; the attention weights are computed by the attention score, and the final fused output feature E v is obtained: m = softmax(Attn) (6) E v =∑m·V (7) Where m is the attention weight matrix, and V is the value vector.

4. The method of claim 3, wherein the method further comprises: The text feature and the fused feature Z are subjected to twice cross-modal attention mechanisms to obtain the final feature representation C, which comprises: The cross-modal attention mechanism adopts a structure similar to the standard attention network, but is divided into two branches, and information interaction is realized by exchanging the key-value pairs of the visual branch and the language branch; (1) Cross-modal query-key-value generation mechanism: the visual branch generates a query matrix Q v , while outputting a visual key matrix K v and a value matrix V v ; the language branch generates a query matrix Q t , while outputting a language key matrix K t and a value matrix V t ; (2) Cross-modal attention interaction calculation: use the query matrix Q of the visual branch v , the key matrix K of the language branch t Calculate the attention matching weight by "dot product + scaling + Softmax"; then perform weighted sum on the value matrix V of the language branch t to realize the attention and fusion of the visual modality to the language modality information; Similarly, the language branch also queries Q by itself t The key matrix K v And the value matrix V v Complete cross-modal attention calculation, realize the bidirectional information interaction between modal (3) residual connection and layer normalization feature reservation and stabilization mechanism: the result of the cross-modal attention calculation is subjected to residual connection with the original input feature of the corresponding mode to reserve the basic information of the original feature; and then the feature after the residual connection is subjected to layer normalization; (4) feature expression enhancement mechanism of the feedforward network: the feature after the residual connection and the layer normalization is input into the feedforward neural network to enhance the expression ability of the model to the cross-modal complex correlation relationship through nonlinear transformation; (5) Secondary residual connection and layer normalization: Through secondary residual and normalization, further ensure the completeness of the features, stabilize the output distribution, and get the visual features H fused with text information V , text features H fused with visual information t ; (6) Finally, the visual feature H V , the text feature H t are spliced to realize the fusion of the text feature and the image feature, and then the final feature C is obtained.

5. The method of claim 4, wherein the method further comprises: which comprises: The multi-modal fusion feature C is taken as an input, and a probability of the news authenticity is predicted through two full connection layers of the classifier, and a calculation formula is as follows: F = Relu(W1C + b1) (8) p = softmax(W2F + b2) (9) Wherein, W1 and b1 respectively represent a weight matrix and a bias from the input layer to the hidden layer, F is an output of the first full connection layer; W2 and b2 are respectively a weight and a bias from the hidden layer to the output layer, and p is a final binary classification probability.

Images