A multi-modal fake news detection method and device

Abstract

The application discloses a multi-modal false news detection method and device, and relates to the field of multi-modal cognitive computing. The method is used for providing in-depth mining and analysis of news information, realizing semantic enhancement of multi-modal information, and mining and application of news sentiment features. According to the BERT model, the first image caption information is subjected to feature coding to obtain a first caption representation and a first empirical representation. A first text representation is subjected to a self-attention mechanism, a full connection layer and averaging to obtain a sentiment representation of comprehensive analysis of the first text representation by all experts and a sentiment tendency evaluation value of comprehensive analysis of the first text representation by all experts. According to the attention weights of a plurality of different sentiment news processors and the first news processing features, news aggregation features for true or false classification are obtained. According to the news aggregation features for true or false classification and a false news detector, a prediction label of the first news is obtained.

Description

Technical Field

[0001] This invention belongs to the field of multimodal cognitive computing, and more specifically relates to a method and apparatus for detecting multimodal false news. Background Technology

[0002] With the rapid development and widespread adoption of social networks, we have entered an era of information overload. Social networks, with their convenience and immediacy, have become the primary platform for the public to obtain information, exchange ideas, and share viewpoints. However, this has also brought concerns about the authenticity and credibility of information on social networks. In this environment of information overload, false information often spreads rapidly through social networks, causing significant harm to individuals, organizations, and even society as a whole. At the same time, faced with the massive flow of information, the efficiency and accuracy of manual review are far from satisfactory. Therefore, technologies for automated detection of false information have become particularly important.

[0003] Early methods for detecting fake news focused on the extraction and use of manually extracted features. These features mainly included: article structure and semantic features, language style of the text, user attributes involved in dissemination, and dissemination patterns. By extracting these deep features, classification models could be built using algorithms such as Naive Bayes, Support Vector Machines, and Decision Trees to detect fake news. Although these manually extracted features could improve the performance of fake news detection, manually extracting valuable features required a significant amount of time and effort. Furthermore, manually extracted features largely depended on specific social media platforms, making it difficult to effectively generalize across different domains, languages, and topics.

[0004] With the development of deep learning and the improvement of computing power, current fake news detection tasks are mainly divided into context-based methods and content-based methods. Context-based methods rely on the information propagation structure and account information to detect fake news. However, they have poor realism, require a large amount of basic information or the construction of a graph propagation structure, and the data collection and processing are quite complex.

[0005] Content-based methods mainly include two types: unimodal and multimodal. Unimodal methods primarily utilize single text content or image information for fake news detection. However, unimodal methods do not consider the interaction, supplementation, and fusion of multimodal information, resulting in low detection efficiency and limited application scenarios. Therefore, multimodal methods have received increasing attention.

[0006] In multimodal misinformation detection methods, researchers initially used Long Short-Term Memory (LSTM) networks to automatically extract and fuse multimodal features of news, and also introduced social comment features to improve the detection performance. Later, methods based on event and semantic invariance were proposed, improving detection accuracy by removing specific event features while retaining shared features between events. Next, researchers introduced variational autoencoders to reconstruct data from multimodal representations, thereby promoting the model's learning of correlations between modalities. During the research process, pre-trained models were gradually used for data preprocessing to extract deep features, improving detection accuracy. Later, researchers successively introduced entities in images and text, as well as related external knowledge, to enrich content and enhance semantics. Furthermore, the widespread dissemination of misinformation is inseparable from the emotional responses it evokes in readers; sentiment features have also been used to help detect misinformation. With the emergence of the contrastive learning-based language-image pre-trained model CLIP, ideas such as image-text mismatch, multimodal ambiguity learning, and image-text similarity have been widely applied, further influencing and improving the alignment and fusion of multimodal information, and achieving excellent results in multimodal misinformation detection.

[0007] In recent years, more multimodal models have emerged, providing efficient and accurate detection tools for the task of detecting fake news. Innovative works such as contrastive learning, co-attention mechanisms, perceptual behavior simulation, interpretability, and causal reasoning have also emerged, leading to unprecedented advancements in multimodal fake news detection. However, existing methods for images only use pixel-level or entity-level information, suitable for detecting image content tampering but failing to understand the context and semantics of the image content; for text, they only consider the original news content, ignoring the large-scale model's mining and analysis of external empirical information; this easily leads to low alignment and poor fusion effects between cross-modal features, resulting in low detection accuracy. Furthermore, using the same processor for news of different sentiment types simultaneously causes a decrease in detection accuracy, and the phenomenon of differences in the distribution of positive and negative sentiment between true and false news has not been addressed. Summary of the Invention

[0008] This invention provides a method and apparatus for detecting multimodal false news, which is used to deeply mine and analyze news information, and realize semantic enhancement of multimodal information as well as the discovery and application of news sentiment features.

[0009] This invention provides a multimodal false news detection method, comprising:

[0010] The first image caption information is obtained by feature encoding of each word in the first image caption information according to the BERT model, and the first empirical information is obtained by feature encoding of each word in the first empirical information according to the BERT model; wherein, the first image caption information corresponds to the first image information included in the first news selected from the dataset, and the first empirical information corresponds to the first text information included in the first news selected from the dataset.

[0011] The first text representation, the first empirical representation, the first image representation, and the first subtitle representation are sequentially passed through a common attention block composed of two encoders to obtain the first empirically enhanced text representation, the first text-enhanced empirical representation, the first subtitle-enhanced image representation, and the first image-enhanced subtitle representation. The three types of multimodal features obtained based on a fully connected mapping function are then concatenated to obtain the first multimodal semantic enhancement representation. Specifically, the first text information and the first image information included in the first news item are respectively feature-encoded to obtain the first text representation and the first image representation. The three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced image representation, the first enhanced text representation and the first enhanced image representation, and the first text-enhanced empirical representation and the first image-enhanced subtitle representation.

[0012] The first text representation and the first image representation are concatenated and mapped with the reinforcement representation through a self-attention mechanism to obtain the first unimodal text representation and the first unimodal image representation in sequence.

[0013] The first text representation, based on a BiLSTM network, self-attention mechanism, fully connected layer, and averaging, yields the sentiment representation and sentiment tendency evaluation value of all experts' comprehensive analysis of the first text representation. Based on the sentiment representations of all experts' comprehensive analysis of the first text representation and the sentiment representations of all experts' comprehensive analysis of the first image representation, the overall sentiment representation of the first news is obtained. Based on the sentiment tendency evaluation values ​​of all experts' comprehensive analysis of the first text representation and the sentiment tendency evaluation values ​​of all experts' comprehensive analysis of the first image representation, the overall positive sentiment probability of the first news is obtained. The overall positive sentiment probability of the first news and the news truthfulness / falseness label are used to obtain the first sentiment inference detection loss function based on the cross-entropy loss function.

[0014] The first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation, and the first overall sentiment representation of the news are concatenated and fused to obtain the overall content feature of the first news. Based on the overall content feature of the first news and different sentiment news processors, a first news processing feature is obtained. Based on the overall sentiment representation of the first news and a gate control unit, attention weights for multiple different sentiment news processors are obtained. Based on the attention weights of multiple different sentiment news processors and the first news processing feature, a news aggregation feature for true / false classification is obtained. Based on the news aggregation feature for true / false classification and a false news detector, a predicted label for the first news is obtained.

[0015] Preferably, the first text representation and the first empirical representation are processed by a common attention block composed of two encoders to obtain a first empirically enhanced text representation, specifically including:

[0016] The first text representation and the first empirical representation included in the first news are mapped to the same dimension; the output of single-head attention and the combined output of multi-head attention are determined by the following formula:

[0017]

[0018] MultiHead(T i ,R i ,R i = concat(h1,h2,...,h) m )·W O

[0019] The combined output of the multi-head attention and the first text representation are concatenated, and then normalized and processed through a fully connected layer to obtain the first empirically reinforced text representation as shown below:

[0020]

[0021] h′=Norm(T i +MultiHead(T i ,R i ,R i ))

[0022] Among them, h i T represents the output vector of the j-th head under single-head attention. i R represents the first text representation. i This represents the first empirical representation. This represents the query projection matrix corresponding to the j-th head. This represents the key projection matrix corresponding to the j-th head. This represents the projection matrix of the value corresponding to the j-th head. d k Let d represent the dimensions of the first textual representation and the first empirical representation projected onto the same space. h This represents the dimension of each head output feature. m represents the number of heads to focus on, MultiHead(T) i ,R i ,R i W represents the overall output of the multi-head attention mechanism. O Represents the weight matrix. `concat(·)` represents the vector concatenation operation, and `h′` represents the normalization operation on the output of the first paper representation and the overall multi-head attention mechanism. represents the first empirical reinforcement text representation, Norm(·) represents the normalization layer, and FFN(·) represents the fully connected network.

[0023] Preferably, the step of concatenating the three types of multimodal features obtained based on the fully connected mapping function to obtain the first multimodal semantic enhancement representation specifically includes:

[0024] The first empirical reinforced text representation and the first subtitle reinforced image representation are concatenated to obtain the first multimodal feature; the first reinforced representation, including the first reinforced text representation and the first reinforced image representation, are concatenated to obtain the second multimodal feature; the first text reinforced empirical representation and the first image reinforced subtitle representation are concatenated to obtain the third multimodal feature; wherein, the first news item includes the first text information and the first image information, which are used to obtain the first reinforced representation based on a pre-trained multimodal large model;

[0025] The first multimodal feature, the second multimodal feature, and the third multimodal feature are concatenated to obtain the first multimodal semantic enhancement representation as shown below:

[0026] M ter =θ·MLP M (concat(M1,M2,M3))

[0027]

[0028] Among them, M ter This represents the first multimodal semantic enhancement representation, concat(·) represents the vector concatenation operation, and θ represents the graph-text similarity. MLP M (·) denotes the fully connected mapping function for multimodal features, M1 represents the first multimodal feature, M2 represents the second multimodal feature, M3 represents the third multimodal feature, and σ1(·), σ2(·), and σ3(·) represent the fully connected mapping functions respectively. This indicates that the first empirical evidence reinforces the textual representation. The first subtitle indicates that the image representation is enhanced. This represents the first enhanced text representation. This represents the first enhanced image representation. This indicates that the first text reinforces the empirical representation. This indicates that the first image enhances the caption representation.

[0029] Preferably, the first text representation is concatenated and mapped with the enhanced representation through a self-attention mechanism to obtain the following first unimodal text representation:

[0030]

[0031] The image representation is concatenated and mapped with the enhanced representation through a self-attention mechanism to obtain the first unimodal image representation as shown below:

[0032]

[0033] Among them, T tra This represents the first unimodal text representation. T represents the first enhanced text representation. i Represents the first text representation, concat(·) represents the vector concatenation operation, and SelfAttention(T) represents the first text representation. i ) indicates performing a self-attention operation on the first text representation, MLP T (·) is the fully connected mapping function for the first unimodal text feature, V tra V represents the first single-modal image representation. i Representing the first image representation, SelfAttention(V) i This indicates that a self-attention operation is performed on the representation of the first image. This represents the first enhanced image representation, MLP V (·) denotes the fully connected mapping function for the first unimodal image features.

[0034] Preferably, the first text representation is based on a BiLSTM network, a self-attention mechanism, a fully connected layer, and averaging to obtain the sentiment representation of all experts' comprehensive analysis of the first text representation and the sentiment tendency evaluation value of all experts' comprehensive analysis of the first text representation, specifically including:

[0035] The first text representation, based on a BiLSTM network, yields the following hidden states:

[0036]

[0037] The hidden state is mapped using a self-attention network and a fully connected layer to obtain the sentiment representation and sentiment tendency evaluation value of the first text representation, respectively. Based on the average of the sentiment representations and sentiment tendency evaluation values ​​of multiple first text representations, the sentiment representation and sentiment tendency evaluation value of all experts after comprehensive analysis of the first text representation are obtained.

[0038]

[0039] in, This represents the hidden state of the first text representation included in the i-th news item. express Forward hidden state, express Backward hidden state, This represents the sentiment assessment value of the k-th expert on the first text representation included in the v-th news item. Let e ​​represent the sentiment representation of the k-th expert's evaluation of the first text representation included in the i-th news item. T This represents the sentiment representation after comprehensive analysis of the first text representation by all experts. AVG represents the sentiment assessment scores of all experts after comprehensive analysis of the first text representation, and W represents the averaging operation. T b represents the learnable sentence parameters. T This represents the learnable offset parameter.

[0040] Preferably, the overall positive sentiment probability of the first news item, the overall sentiment representation of the first news item, and the loss function for the first sentiment inference detection are as follows:

[0041]

[0042] E = MLP E (concat(λe T ,(1-λ)e V ))

[0043]

[0044] Among them, y e This indicates the overall positive sentiment probability of the first news item. This represents the sentiment assessment value of all experts after comprehensive analysis of the first text representation. E represents the overall sentiment assessment of the first news story based on a comprehensive analysis of the first image by all experts. T This represents the sentiment representation after comprehensive analysis of the first text representation by all experts. VThis represents the sentiment representation of all experts' comprehensive analysis of the first image, λ represents the weighting factor, and concat(·) represents the vector concatenation operation. MLP E (·) represents a fully connected mapping function for sentiment features, L emo Let represent the loss function for the first sentiment inference detection, and y represent the truth or falsehood label of the news.

[0045] Preferably, the overall content characteristics of the first news item are as follows:

[0046] M = [T] tra V tra M ter ;E]

[0047] The attention weights of the various emotional news processors are shown below:

[0048] a = [a1, a2, ..., a x ] = softmax(G(E; θ) G ))

[0049] The news aggregation features used for true / false classification:

[0050]

[0051] m x =H x (M;θ x )

[0052] Where M represents the overall content characteristics of the first news item, and T... tra V represents the first unimodal text representation. tra M represents the first single-modal image representation. ter Let E represent the first multimodal semantic enhancement representation, G represent the first overall news sentiment representation, and θ represent the gate control unit. G This represents the parameters of the control unit, where 'a' represents the parameters derived from [a1, a2, ..., a...]. x A vector composed of ], a x M represents the expression weight corresponding to the x-th sentiment type processor. all Let m represent the news aggregation features used for true / false classification. x H represents the primary news processing characteristic. x Let θ represent the news processor with the x-th sentiment type. x This represents the parameters in the corresponding processor.

[0053] Preferably, the prediction tag for the first news item is as follows:

[0054]

[0055] in, The W tag indicates a prediction of first news. f and b f M represents the trainable parameters. all This represents the news aggregation feature used for true / false classification, and softmax(·) represents the mapping function.

[0056] Preferably, after obtaining the prediction tag for the first news item, the method further includes:

[0057] Based on the predicted label of the first news item, the inspection loss function for all news items is obtained through the cross-entropy loss function, as shown below:

[0058]

[0059] The final loss function is obtained based on the inspection loss function of all the news items, the first sentiment inference detection loss function, and the parameter factors:

[0060] L final =γL emo +L fnd

[0061] Among them, L emo Let L represent the loss function for first sentiment inference detection. fnd Let γ represent the loss function for news inspection, and let γ represent the parameter factor.

[0062] This invention provides a multimodal false news detection device, comprising:

[0063] The first obtaining unit is used to encode the features of each word included in the first image caption information according to the BERT model to obtain the first caption representation, and to encode the features of each word included in the first empirical information according to the BERT model to obtain the first empirical representation; wherein, the first image caption information corresponds to the first image information included in the first news selected from the dataset, and the first empirical information corresponds to the first text information included in the first news selected from the dataset.

[0064] The second obtaining unit is used to sequentially pass the first text representation, the first empirical representation, the first image representation, and the first subtitle representation through a common attention block composed of two encoders to obtain a first empirically enhanced text representation, a first text-enhanced empirical representation, a first subtitle-enhanced image representation, and a first image-enhanced subtitle representation; and concatenate the three types of multimodal features obtained based on a fully connected mapping function to obtain a first multimodal semantic enhancement representation; wherein, the first text information and the first image information included in the first news are respectively processed by feature encoding to obtain the first text representation and the first image representation; the three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced image representation, the first enhanced text representation and the first enhanced image representation, and the first text-enhanced empirical representation and the first image-enhanced subtitle representation;

[0065] The third obtaining unit is used to concatenate and map the first text representation and the first image representation with the reinforcement representation through a self-attention mechanism, respectively, to obtain the first unimodal text representation and the first unimodal image representation in sequence.

[0066] The fourth unit is used to obtain the sentiment representation of the first text representation based on a BiLSTM network, a self-attention mechanism, a fully connected layer, and averaging, as well as the sentiment tendency evaluation value of the comprehensive analysis of the first text representation by all experts. Based on the sentiment representation of the first text representation and the sentiment representation of the first image representation by all experts, the unit obtains the overall sentiment representation of the first news. Based on the sentiment tendency evaluation value of the first text representation and the sentiment tendency evaluation value of the first image representation by all experts, the unit obtains the overall positive sentiment probability of the first news. The overall positive sentiment probability of the first news and the news truthfulness / falseness label are used to obtain the first sentiment inference detection loss function based on the cross-entropy loss function.

[0067] The fifth obtaining unit is used to concatenate and fuse the first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation, and the first overall news sentiment representation to obtain the first overall news content feature; obtain the first news processing feature based on the first overall news content feature and different sentiment news processors; obtain the attention weights of multiple different sentiment news processors based on the first overall news sentiment representation and the gate control unit; obtain the news aggregation feature for true / false classification based on the attention weights of multiple different sentiment news processors and the first news processing feature; and obtain the predicted label of the first news based on the news aggregation feature for true / false classification and the false news detector.

[0068] This invention provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the multimodal false news detection method described above.

[0069] This invention provides a computer-readable storage medium, characterized in that it stores a computer program, which, when executed by a processor, causes the processor to perform the multimodal false news detection method described above.

[0070] This invention provides a multimodal method and apparatus for detecting fake news. The method uses a BERT model to enhance the knowledge of the first text representation and the first image representation, introducing a first caption representation and a first empirical representation, thus addressing the problem of insufficient data mining and analysis in existing methods. Utilizing co-attention and self-attention mechanisms, it obtains a first empirically enhanced text representation, a first text-enhanced empirical representation, a first caption-enhanced image representation, and a first image-enhanced caption representation, respectively. Combining these multiple representations achieves multimodal features. These multimodal features can complement and enhance the semantics of the first unimodal text representation and the first unimodal image representation, thereby solving the problem of low detection accuracy caused by low cross-modal feature similarity and poor fusion effects. Furthermore, an expert network is designed to determine the overall positive sentiment probability and overall sentiment representation of the first news story. A sentiment inference loss is constructed to assist in the fake news detection task, taking into account the differences in sentiment tendencies between true and false news. A balance learning module is designed to allow the model to perform specialized processing based on news sentiment type, improving generality and detection accuracy, increasing the accuracy of multimodal fake news detection on social networks and the applicability of the discrimination method in reality. This method can deeply mine and analyze news information, realize semantic enhancement of multimodal information, and discover and apply the emotional features of news. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a schematic diagram of a multimodal false news detection method provided in an embodiment of the present invention;

[0073] Figure 2 This is a schematic diagram of a multimodal false news detection device provided in an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] Figure 1 This is a schematic diagram of a multimodal false news detection method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method mainly includes the following steps:

[0076] Step 101: Encode the features of each word included in the first image caption information according to the BERT model to obtain the first caption representation; encode the features of each word included in the first empirical information according to the BERT model to obtain the first empirical representation; wherein, the first image caption information corresponds to the first image information included in the first news selected from the dataset, and the first empirical information corresponds to the first text information included in the first news selected from the dataset.

[0077] Step 102: The first text representation, the first empirical representation, the first image representation, and the first subtitle representation are sequentially passed through a common attention block composed of two Transformer encoders to obtain a first empirically enhanced text representation, a first text-enhanced empirical representation, a first subtitle-enhanced image representation, and a first image-enhanced subtitle representation. The three types of multimodal features obtained based on the fully connected mapping function are concatenated to obtain a first multimodal semantic enhancement representation. The first text information and the first image information included in the first news item are respectively feature-encoded to obtain the first text representation and the first image representation. The three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced image representation, the first enhanced text representation and the first enhanced image representation, and the first text-enhanced empirical representation and the first image-enhanced subtitle representation.

[0078] Step 103: The first text representation and the first image representation are concatenated and mapped with the reinforcement representation through a self-attention mechanism to obtain the first unimodal text representation and the first unimodal image representation in sequence.

[0079] Step 104: The first text representation is obtained based on a BiLSTM network, a self-attention mechanism, a fully connected layer, and averaging to obtain the sentiment representation of all experts' comprehensive analysis of the first text representation and the sentiment tendency evaluation value of all experts' comprehensive analysis of the first text representation; the overall sentiment representation of the first news is obtained based on the sentiment representation of all experts' comprehensive analysis of the first text representation and the sentiment representation of all experts' comprehensive analysis of the first image representation; the overall positive sentiment probability of the first news is obtained based on the sentiment tendency evaluation value of all experts' comprehensive analysis of the first text representation and the sentiment tendency evaluation value of all experts' comprehensive analysis of the first image representation; the overall positive sentiment probability of the first news and the news truthfulness label are used to obtain the first sentiment inference detection loss function based on the cross-entropy loss function;

[0080] Step 105: The first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation, and the first overall news sentiment representation are concatenated and fused to obtain the overall content feature of the first news. The first news processing feature is obtained based on the overall content feature of the first news and different sentiment news processors. The attention weights of multiple different sentiment news processors are obtained based on the overall sentiment representation of the first news and the gate control unit. The news aggregation feature for true / false classification is obtained based on the attention weights of multiple different sentiment news processors and the first news processing feature. The predicted label of the first news is obtained based on the news aggregation feature for true / false classification and the false news detector.

[0081] In practical applications, fake news detection is often formulated as a binary classification problem. Multimodal social media news can be defined as:

[0082] N={T,V}(1)

[0083] Where T represents text information, V represents image information, and N represents news.

[0084] Specifically, the truth or falsehood label for fake news detection can be represented by y. Specifically, when y = 1, the fake news detection result indicates the news is true; when y = 0, the fake news detection result indicates the news is false.

[0085] In practical applications, when the multimodal fake news detection model is denoted by M, the goal of this task is to use the multimodal fake news detection model to determine the veracity of news, and the resulting predicted labels can be used... This indicates that, specifically, for the i-th news item N... i ={T i V i The detection process can be represented as:

[0086]

[0087] Among them, T i V represents the i-th text message. i M(N) represents the information of the i-th image. i ) represents the detection of the i-th news item by the multimodal false news detection model.

[0088] In this embodiment of the invention, the preprocessing process and method are illustrated using the processing of two datasets, Weibo and Twitter, as examples.

[0089] Specifically, it is assumed that all news posts in the Weibo dataset were published on Weibo within a set time period, and all authentic news items have been verified by official news agencies. The dataset contains 4779 authentic news items, 4749 false news items, and 9528 images. The news includes text, images, and social background. In practical applications, the data was cleaned, removing news items without text or images, and for news items with multiple images, only one image was retained. The training and test sets were divided according to the format given by the official data provider.

[0090] The Twitter dataset is a well-known multimodal dataset for detecting fake news, consisting of news articles from multiple events. The number of images in this dataset is much smaller than the number of samples, meaning an image may be shared by multiple examples. The Twitter dataset contains 8720 real news articles, 7448 fake news articles, and 514 images. In practical applications, the dataset is processed to remove symbols such as "@", "#", and "http:", emojis, and hyperlinks from the text. Only news articles containing both text and images are retained. The training and test sets are split according to the official dataset specifications.

[0091] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture. It learns the language representation of each word and has achieved good results in many downstream NLP (Natural Language Processing) tasks. In this embodiment of the invention, the pre-trained BERT model is used to encode the features of each word in the first text information corresponding to the first news article, resulting in the first text representation, as shown below:

[0092] T i ={w1,w2,…,w m} (3)

[0093] Among them, T i Let m represent the number of words in the first text representation, w1 represent the first word, and w represent the number of words in the first text representation. mThis represents the m-th word. d w This represents the dimension of the word embedding.

[0094] In practical applications, d w The dimension of the vector is determined by the number of dimensions; different dimensions mean that it can contain different levels of semantic information. Let's assume the word embedding dimension is d. w =768, meaning the hidden state vector for each word has 768 dimensions, which encodes rich semantic, syntactic, and other linguistic information about the word. m This represents the hidden state of the corresponding word in the last output layer under the BERT model.

[0095] The Swin Transformer is a model that further improves upon the Transformer by incorporating hierarchical and localization concepts, achieving state-of-the-art performance on many computer vision tasks.

[0096] In this embodiment of the invention, the Swing Transformer is used to perform feature encoding on the first image information corresponding to the first news item to obtain the first image representation, as shown below:

[0097] V i ={v1,v2,…,v n} (4)

[0098] Among them, V i This represents the i-th image. w and h represent the width and height of the i-th image, respectively, and n represents the number of selected regions in the i-th image. d v This represents the dimension of the embedding of the i-th image.

[0099] In practical applications, v n d represents the hidden state vector corresponding to a certain region in the last output layer of the i-th image in the Swing Transformer. v The dimensionality of the image feature vector is specified, which affects the richness and detail of the information contained in the image features. For example, if the embedding dimension of the i-th image is d... v =1024 means that the feature vector corresponding to each region is 1024-dimensional. This high-dimensional vector is used to carry the rich visual feature information such as texture and shape of each region of the image.

[0100] The key role of the CLIP large model is to connect the semantic information of image and text information. It can learn the correspondence between image and text information, acting like a bridge between images and related text, facilitating the matching of image and text information. In this embodiment of the invention, the pre-trained CLIP model can generate highly aligned text and image representations.

[0101] Specifically, based on First News N i ={T i V i In other words, news articles that contain both text and images can be used to obtain, based on the pre-trained CLIP model, globally similar, and representation-enhanced first-level text and image representations, as shown below:

[0102]

[0103] in, This represents the first enhanced text representation. T represents the first enhanced image representation. i V represents the first text representation. i This represents the first image. d c N represents the embedding dimension of the first text information and the first image information after CLIP enhancement. i This indicates the i-th news item.

[0104] In practical applications, the CLIP-enhanced embedding dimension of the first text and image information facilitates more comprehensive and effective multimodal information fusion and analysis operations. For example, d c =512, meaning that the vector dimensions of the first text information and the first image information are both 512-dimensional. They are features enhanced by CLIP, which makes the first text information and the first image information better correspond and associate at the semantic level.

[0105] BLIP is a pre-trained multimodal large model for unified visual language understanding and generation, which has rich language and visual knowledge. In the embodiments of this invention, BLIP can be used to obtain descriptive information of image scenes and semantics.

[0106] In step 101, the first image information corresponding to the first news story is processed through the BLIP model to obtain the first image caption information. This first image caption information corresponds to the first image information selected from the dataset; that is, the first image caption information can describe the scene and semantics expressed by the first image in the first news story. Further, the first image caption information is input into the BERT model. The BERT model encodes the features of each word included in the first image caption information, resulting in the first caption representation shown below:

[0107] P i ={p1,p2,…,p z} (6)

[0108] Among them, P i Let z represent the first caption representation of the i-th image, and z represent the number of words included in the caption information of the first image. p z d represents the hidden state of the last output layer of the BERT model corresponding to a word. p This represents the dimension of the word embedding.

[0109] For example, if the caption of a sunset picture at the beach contains multiple words such as "golden," "sunset," and "shining," the number of words is the value of z, while p z Specifically, this refers to the hidden state vector of the last word in the last output layer of the BERT model when processing the caption information of the first image. This vector carries the semantic and other related features of the word after being learned by the model. p This specifies the dimension of each word embedding vector; different dimensions mean that word features can be characterized with varying degrees of richness.

[0110] Furthermore, ChatGPT, through training with a large amount of data in its knowledge base, is able to understand the content of the first text information and mine relevant empirical information on the internet. ChatGPT can provide inferences about the authenticity of the first text information corresponding to the first news story.

[0111] Specifically, after mining the first text information corresponding to the first news story using ChatGPT, the first empirical information can be obtained. This first empirical information corresponds to the first text information selected from the dataset; that is, the first empirical information can demonstrate the reasoning basis for the truthfulness or falsity of the first text information included in the first news story. Further, the first empirical information is input into the BERT model. Through feature encoding of each word included in the first empirical information by the BERT model, the following first empirical representation can be obtained:

[0112] R i ={r1,r2,…,r u} (7)

[0113] Among them, R i Let u represent the first empirical representation, and r represent the number of words contained in the first empirical information. u This represents the hidden state vector of the last word contained in the first empirical information in the last output layer of the BERT model. d r This represents the dimension of the word embedding.

[0114] In practical applications, the co-attention mechanism can effectively enhance the interaction between information. Its main structure is similar to that of the Transformer encoder. The attention module is a crucial component, capable of capturing the global constraints of all items in the sequence. The co-attention mechanism plays a key role in this process; its structure is similar to the Transformer encoder, with the attention module being particularly important as it captures the global constraints of all items in the input sequence. Simply put, it analyzes the relationships and importance between elements from a holistic perspective. Its input consists of three matrices: a query matrix Q, a key matrix K, and a value matrix V. These matrices interact, and attention weights are derived based on the similarity between the query matrix Q and the key matrix K, determining which part of the value matrix V is more important.

[0115] In step 102, the first text representation and the first empirical representation are mapped to the same dimension. Then, the first text representation and the first empirical representation are used as input to the encoder. Attention weights are calculated based on the cosine similarity between the first text representation and the first empirical representation to determine which part of the first empirical representation is more important and can be used as supplementary information to the first text representation. The entire multi-head attention process is as follows:

[0116]

[0117] MultiHead(T i ,R i ,R i = concat(h1,h2,...,h) m )·W O (9)

[0118] Among them, h i Let represent the output vector of the j-th head under single-head attention. In this embodiment of the invention, it can also be expressed as the output vector of the j-th head under single-head attention from the first empirical representation R. i The semantic information obtained related to the first text representation, T i R represents the first text representation. i This represents the first empirical representation. This represents the query projection matrix corresponding to the j-th head. This represents the key projection matrix corresponding to the j-th head. This represents the projection matrix of the value corresponding to the j-th head. d k Let d represent the dimensions of the first textual representation and the first empirical representation projected onto the same space. h This represents the dimension of each head output feature. m represents the number of heads to focus on, MultiHead(T) i ,R i ,R i W represents the overall output of the multi-head attention mechanism. O Represents the weight matrix. concat(·) represents the vector concatenation operation.

[0119] Next, the key parts of the first empirical representation are concatenated and fused with the first text representation. After normalization and a fully connected layer, the output is the first empirical reinforced text representation:

[0120] h′=Norm(T i +MultiHead(T i ,R i ,R i (10)

[0121]

[0122] Where h′ represents the normalization operation performed on the output of the first paper representation and the overall multi-head attention mechanism. represents the first empirical reinforcement text representation, Norm(·) represents the normalization layer, and FFN(·) represents the fully connected network.

[0123] It should be noted that, in this embodiment of the invention, the other encoder in the shared attention block is responsible for supplementing and enhancing the textual representation with the empirical evidence. That is, the first textual representation and the first empirical representation pass through the shared attention block composed of two Transformer encoders, where the query matrix Q = R... i The bond matrix K = T i Value matrix V = T i The first text reinforcement empirical representation is obtained by using an algorithm similar to the above formulas (8)-(11).

[0124] Similarly, the first image representation and the first subtitle representation can also be obtained through the algorithm related to the above formulas (8)-(11), when the query matrix Q = P i The bond matrix K = V i Value matrix V = V i When the query matrix Q = V, the first subtitle enhanced image representation is obtained; iThe bond matrix K = P i Value matrix V = P i At that time, the first image is enhanced with subtitle representation.

[0125] Specifically, the first empirical reinforced text representation, the first text reinforced empirical representation, the first caption reinforced image representation, and the first image reinforced caption representation are as follows:

[0126]

[0127] in, This indicates that the first empirical evidence reinforces the textual representation. The first subtitle indicates that the image representation is enhanced. This indicates that the first text reinforces the empirical representation. This indicates that the first image enhances the caption representation, T i R represents the first text representation. i V represents the first empirical subtitle representation. i P represents the first image representation. i Represents the first subtitle representation, CoAttention(T) i ,R i CoAttention(V) represents the coordination and processing of the first text representation and the first empirical representation. i ,P i This indicates that the first image representation and the first subtitle representation will be coordinated.

[0128] In this embodiment of the invention, three multimodal features, respectively fused from text-related information and image-related information, are proposed to achieve deep multimodal semantic enhancement. Specifically, the first multimodal feature is obtained by concatenating the first empirically enhanced text representation and the first subtitle enhanced image representation; the second multimodal feature is obtained by concatenating the first enhanced text representation and the first enhanced image representation included in the first enhanced representation; and the third multimodal feature is obtained by concatenating the first text-enhanced empirical representation and the first image-enhanced subtitle representation. The three multimodal features are as follows:

[0129]

[0130] Where M1 represents the first multimodal feature, M2 represents the second multimodal feature, M3 represents the third multimodal feature, and σ1(·), σ2(·) and σ3(·) represent fully connected mapping functions, respectively.

[0131] Next, the three types of multimodal features will be concatenated and then mapped using the fully connected mapping function MLP for multimodal features. M (.) and the introduction of image-text similarity We adjust the weight relationship between the multimodal representation and the unimodal representation to obtain the final knowledge-based first multimodal semantic enhancement representation, as shown below:

[0132] M ter =θ·MLP M (concat(M1,M2,M3)) (14)

[0133] Among them, M ter This represents the first multimodal semantic enhancement representation, concat(·) represents the vector concatenation operation, and θ represents the graph-text similarity. MLP M (·) denotes the fully connected mapping function for multimodal features.

[0134] It should be noted that existing research suggests that mismatches between the text and images in news articles are highly likely to be fake news. Therefore, this invention introduces the concept of text-image similarity, which can be used to adjust the weighting relationship between multimodal and unimodal representations. Higher text-image similarity indicates that multimodal information is more important, while lower similarity indicates that unimodal information is more important.

[0135] In step 103, for the first text representation corresponding to the first news story, a self-attention mechanism is first used to learn the internal relationships between words in the first text representation, as well as the importance of each word itself. For example, in a news story, the self-attention mechanism can discover that some keywords may be more important. In a first text representation such as "In this sports event, an athlete broke the world record, an amazing achievement," words related to "broken the world record" may be assigned higher importance weights after calculation by the self-attention mechanism because they carry core semantic information.

[0136] After completing the self-attention operation, the result can be concatenated with the first enhanced text representation generated by CLIP, and then mapped and learned through a fully connected mapping function of unimodal text features to finally obtain the first unimodal text representation. In this invention, the first unimodal text representation is represented by the following formula:

[0137]

[0138] Among them, T tra This represents the first unimodal text representation. T represents the first enhanced text representation. i Represents the first text representation, concat(·) represents the vector concatenation operation, and SelfAttention(T) represents the first text representation. i This indicates that a self-attention operation is performed on the first text representation, MLP. T(·) is the fully connected mapping function for the first unimodal text feature.

[0139] Accordingly, for the first image representation corresponding to the first news story, a self-attention mechanism is first used to learn the internal connections between different regions of the first image, as well as the importance of each region itself. For example, in a landscape image, after processing by the self-attention mechanism, the region containing the main subject (such as the mountain peak in the center of the image) may be judged as more important, that is, given a higher weight, while the relatively less important peripheral regions will have a lower weight. This highlights the differences in importance and correlation between the regions within the image.

[0140] The self-attention operation learns the internal connections and importance between different regions of the first image. The result of this self-attention operation can be concatenated and mapped with the first enhanced image representation generated by CLIP, ultimately yielding the first unimodal image representation as shown below:

[0141]

[0142] Among them, V tra V represents the first single-modal image representation. i Represents the first image representation, SelfAttentoion(V) i This indicates that a self-attention operation is performed on the representation of the first image. This represents the first enhanced image representation, MLP V (·) denotes the fully connected mapping function for the first unimodal image features.

[0143] In this embodiment of the invention, it is initially assumed that the sentiment tendencies in real news and false news may be inconsistent. By performing sentiment statistics on Weibo and Twitter datasets respectively, it can be found that there are significant differences in the sentiment tendencies of real news and false news in the Weibo and Twitter datasets. Specifically, false news expresses more negative sentiments, while real news expresses more positive sentiments. Utilizing these characteristics can assist in the detection of false news. In this embodiment of the invention, a sentiment analysis module is constructed to extract sentiment features from news and analyze the sentiment tendencies of the news.

[0144] In step 104, the sentiment analysis module is used to extract the sentiment features of the first news story and analyze its sentiment tendency. Specifically, the first news story's first text representation and first image representation are input into the sentiment analysis module, which can then learn the sentiment features of the news text and image. K sentiment evaluation experts are set up to perform sentiment learning on the first text representation and the first image representation, respectively. Each evaluation expert consists of a BiLSTM network, a self-attention network, and a fully connected network.

[0145] The following example uses the first text representation, which is input into a network composed of K sentiment assessment experts for processing. Each expert first contains a BiLSTM. After the first text representation is input into the BiLSTM network, due to the characteristics of each BiLSTM network, each network unit corresponds to a word embedding, which enables each word in the first text representation to interact with its surrounding words, that is, to learn the sentiment features of the text by considering the contextual information of each word.

[0146] Each unit corresponds to a word embedding in the first text representation. Each word interacts with its surrounding words, and the sentiment features of the text are learned through contextual information to obtain the hidden state of the first text representation of the first news item after passing through the BiLSTM network:

[0147]

[0148] in, This represents the hidden state of the first text representation of the i-th news item after passing through the BiLSTM network. express Forward hidden state, express Backward hidden state, [;] represents vector concatenation operation.

[0149] In practical applications, It integrates the first text representation of the first news item from front to back (forward hidden state) ) and from back to front (backward hidden state) The information obtained from the analysis is used to integrate the forward hidden state and the backward hidden state through vector concatenation operations.

[0150] Furthermore, the output of the BiLSTM network... Then, the text is input into a self-attention network. The self-attention mechanism highlights the parts with more pronounced emotional features based on the calculated attention weights, further refining the key emotional information in the text, and obtaining the emotional representation of the first news article, which is evaluated by the experts.

[0151]

[0152] in, This represents the sentiment representation of the k-th expert's evaluation of the first text representation included in the i-th news item.

[0153] Then By mapping through fully connected layers, a classification task of positive and negative sentiment is performed to obtain the sentiment tendency evaluation values ​​of the first text representations included in the first news story, as set by experts:

[0154]

[0155] in, W represents the sentiment assessment score of the k-th expert on the first text representation included in the i-th news item. This score reflects the degree to which the expert judges the first text representation to present a positive or negative sentiment; here, 1 represents positive sentiment and 0 represents negative sentiment. T b represents the learnable sentence parameters. T The learnable offset parameter is represented by softmax(·), which is a mapping function for calculating sentiment scores. It transforms the input into a form that conforms to a probability distribution, so that the output can be interpreted as the probability of belonging to different sentiment categories (positive or negative).

[0156] Next, since there are K such emotion assessment experts, it is necessary to assess multiple... and multiple By averaging the results, we can obtain the sentiment representations and sentiment tendency assessments of all experts after the comprehensive analysis of the first text representations:

[0157]

[0158] Among them, e T This represents the sentiment representation after comprehensive analysis of the first text representation by all experts. AVG represents the sentiment assessment scores of all experts after comprehensive analysis of the first text representation, and it indicates the averaging operation. This represents the sentiment representation evaluated by the first expert on the first text representation included in the i-th news item. This represents the sentiment assessment value of the first expert on the first text representation included in the i-th news item.

[0159] Accordingly, based on formulas (17)-(21) in step 104 above, the emotional representation of all experts based on the comprehensive analysis of the first image representation and the emotional tendency evaluation value of all experts based on the comprehensive analysis of the first image representation can be obtained, which will not be elaborated here.

[0160] Furthermore, to balance the importance of the emotions expressed by the first text representation and the first image representation in the overall sentiment of the news, a parameter λ is introduced in this embodiment of the invention to adjust the sentiment representation and sentiment tendency evaluation value after the comprehensive analysis of text and images, respectively. Specifically, the overall sentiment representation of the first news is obtained based on the sentiment representations of all experts' comprehensive analysis of the first text representation and the sentiment representations of all experts' comprehensive analysis of the first image representation; the overall positive sentiment probability of the first news is obtained based on the sentiment tendency evaluation values ​​of all experts' comprehensive analysis of the first text representation and the sentiment tendency evaluation values ​​of all experts' comprehensive analysis of the first image representation.

[0161]

[0162] E = MLP E (concat(λe T ,(1-λ)e V )) (twenty three)

[0163] Where λ represents the weighting factor, y e This indicates the overall positive sentiment probability of the first news item. This represents the sentiment assessment value of all experts after comprehensive analysis of the first text representation. E represents the overall sentiment assessment of the first news story based on a comprehensive analysis of the first image by all experts. T This represents the sentiment representation after comprehensive analysis of the first text representation by all experts. V This represents the sentiment representation of all experts' comprehensive analysis of the first image. `concat(·)` represents the vector concatenation operation. (MLP) E (·) represents a fully connected mapping function for sentiment features.

[0164] In practical applications, λ is a weighting factor between 0 and 1 that adjusts the relative importance of textual and visual sentiment. When λ approaches 1, it means that textual sentiment is given more weight when judging the overall sentiment of the news; when λ approaches 0, it means that visual sentiment is more important in the news.

[0165] In practical applications, it is considered that false news tends to convey negative emotions, while true news tends to convey positive emotions. In this embodiment of the invention, it is assumed that news expressing negative emotions is false, and news expressing positive emotions is true, thus constructing an emotion inference loss.

[0166] Specifically, y is used to represent the truthfulness or falsehood of news, where y=1 indicates true news and y=0 indicates false news. e The labels representing the predicted truth or falsehood of news obtained through sentiment inference are then used to calculate the first sentiment inference detection loss function for all news items using the cross-entropy loss function:

[0167] L emo =∑-[ylog(y e )+(1-y)log(1-y e )] (twenty four)

[0168] Among them, L emo Let represent the loss function for the first sentiment inference detection, and y represent the truth or falsehood label of the news.

[0169] In practical applications, the cross-entropy loss function is often used to measure the degree of difference between the predicted result and the true label. Here, it measures the difference between the news truth prediction result obtained by sentiment inference and the actual truth situation. By minimizing the first sentiment inference detection loss function, the performance of the sentiment analysis module is optimized so that it can more accurately judge the truth of the news based on sentiment features.

[0170] News articles within the same sentiment category share certain commonalities, while news articles across different sentiment categories exhibit certain differences. Previous studies, ignoring sentiment categories and applying the same processing procedure to all news articles, have resulted in a decrease in overall detection accuracy. This invention addresses this by constructing a balance learning module.

[0171] In step 105, the first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation, and the first overall news sentiment representation obtained in the above steps are concatenated and fused to obtain the following overall news content features:

[0172] M = [T] tra V tra M ter ;E] (25)

[0173] Where M represents the overall content characteristics of the first news item, and T... tra V represents the first unimodal text representation. tra M represents the first single-modal image representation. ter E represents the first multimodal semantic enhancement representation, and E represents the first overall sentiment representation of the news.

[0174] Furthermore, X news processors with different sentiment types were set up. Based on the overall content characteristics of the first news and the news processors with different sentiments, the following first news processing characteristics were obtained:

[0175] m x =H x (M;θ x (26)

[0176] Among them, H x Let θ represent the news processor with the x-th sentiment type. x This represents the parameter in the corresponding processor, m x The processor H represents the xth emotion type. x The first news processing features are mapped to the overall content features M of the first news.

[0177] At the same time, attention weights for multiple different emotional news processors are obtained based on the overall sentiment representation of the first news report and the gate control unit:

[0178] a = [a1, a2, ..., a x] = softmax(G(E; θ) G (27)

[0179] Where 'a' represents the expression composed of [a1, a2, ..., a... x A vector composed of ], a x Let G represent the expression weight corresponding to the x-th sentiment type processor, and θ represent the gate control unit. G These represent the parameters of the control unit.

[0180] Furthermore, based on the attention weights of multiple different sentiment news processors and the first news processing features, the final news aggregation features used for truth / falsehood classification are obtained:

[0181]

[0182] Among them, M all a represents the news aggregation feature used for true / false classification. x m represents the expression weight corresponding to the x-th sentiment type processor. x Let x represent the news processing features obtained by the x-th sentiment type processor, where x = 1, ..., X.

[0183] Furthermore, the news aggregation features used for truth / falsehood classification are fed into the false news detector and mapped to the binary classification result through a fully connected layer, ultimately obtaining the predicted label for the first news item:

[0184]

[0185] in, The predicted tag for news item i, W f and b f M represents the trainable parameters. all This represents the news aggregation feature used for true / false classification, and softmax(·) represents the mapping function.

[0186] It should be noted that, in this embodiment of the invention, y is used to represent the real tag of the news. This represents the predicted label of the first news item obtained through news aggregation features. Then, the cross-entropy loss function is used to determine the detection loss function for all news items as shown below:

[0187]

[0188] In this embodiment, the predicted label of the first news item in the main classification task and the loss function of the first sentiment inference detection in sentiment analysis are combined. In addition, a parameter factor γ is introduced to adjust the ratio between the two, resulting in the following final loss function:

[0189] L final =γLemo +L fnd (31)

[0190] Among them, L emo Let L represent the loss function for first sentiment inference detection. fnd Let γ represent the loss function for news inspection, and let γ represent the parameter factor, where the value of γ is between 0 and 1.

[0191] In practical applications, each training session selects a batch of samples for training, and training stops when the training loss converges or the number of training iterations reaches the maximum set value.

[0192] This invention provides a multimodal method and apparatus for detecting fake news. The method uses a BERT model to enhance the knowledge of the first text representation and the first image representation, introducing a first caption representation and a first empirical representation, thus addressing the problem of insufficient data mining and analysis in existing methods. Utilizing co-attention and self-attention mechanisms, it obtains a first empirically enhanced text representation, a first text-enhanced empirical representation, a first caption-enhanced image representation, and a first image-enhanced caption representation, respectively. Combining these multiple representations achieves multimodal features. These multimodal features can complement and enhance the semantics of the first unimodal text representation and the first unimodal image representation, thereby solving the problem of low detection accuracy caused by low cross-modal feature similarity and poor fusion effects. Furthermore, an expert network is designed to determine the overall positive sentiment probability and overall sentiment representation of the first news story. A sentiment inference loss is constructed to assist in the fake news detection task, taking into account the differences in sentiment tendencies between true and false news. A balance learning module is designed to allow the model to perform specialized processing based on news sentiment type, improving generality and detection accuracy, increasing the accuracy of multimodal fake news detection on social networks and the applicability of the discrimination method in reality. This method can deeply mine and analyze news information, realize semantic enhancement of multimodal information, and discover and apply the emotional features of news.

[0193] Based on the same inventive concept, this invention provides a multimodal false news detection device. Since the principle by which this device solves the technical problem is similar to that of a multimodal false news detection method, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be described again.

[0194] Figure 2 This is a schematic diagram of a multimodal false news detection device provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the device includes: a first obtaining unit 201, a second obtaining unit 202, a third obtaining unit 203, a fourth obtaining unit 204, and a fifth obtaining unit 205.

[0195] The first obtaining unit 201 is used to perform feature encoding on each word included in the first image caption information according to the BERT model to obtain the first caption representation, and to perform feature encoding on each word included in the first empirical information according to the BERT model to obtain the first empirical representation; wherein, the first image caption information corresponds to the first image information included in the first news selected from the dataset, and the first empirical information corresponds to the first text information included in the first news selected from the dataset.

[0196] The second obtaining unit 202 is used to sequentially pass the first text representation, the first empirical representation, the first image representation, and the first subtitle representation through a common attention block composed of two encoders to obtain a first empirically enhanced text representation, a first text-enhanced empirical representation, a first subtitle-enhanced image representation, and a first image-enhanced subtitle representation; and concatenate the three types of multimodal features obtained based on a fully connected mapping function to obtain a first multimodal semantic enhancement representation; wherein, the first text information and the first image information included in the first news are respectively processed by feature encoding to obtain the first text representation and the first image representation; the three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced image representation, the first enhanced text representation and the first enhanced image representation, and the first text-enhanced empirical representation and the first image-enhanced subtitle representation;

[0197] The third obtaining unit 203 is used to concatenate and map the first text representation and the first image representation with the reinforcement representation through a self-attention mechanism, respectively, to obtain the first unimodal text representation and the first unimodal image representation in sequence.

[0198] The fourth unit 204 is used to obtain the sentiment representation of the first text representation based on a BiLSTM network, a self-attention mechanism, a fully connected layer, and averaging, as well as the sentiment tendency evaluation value of the comprehensive analysis of the first text representation by all experts. Based on the sentiment representation of the comprehensive analysis of the first text representation by all experts and the sentiment representation of the first image representation by all experts, the overall sentiment representation of the first news is obtained. Based on the sentiment tendency evaluation value of the comprehensive analysis of the first text representation by all experts and the sentiment tendency evaluation value of the comprehensive analysis of the first image representation by all experts, the overall positive sentiment probability of the first news is obtained. The overall positive sentiment probability of the first news and the news truthfulness / falseness label are used to obtain the first sentiment inference detection loss function based on the cross-entropy loss function.

[0199] The fifth unit 205 is used to concatenate and fuse the first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation, and the first overall news sentiment representation to obtain the first overall news content feature; obtain the first news processing feature based on the first overall news content feature and different sentiment news processors; obtain the attention weights of multiple different sentiment news processors based on the first overall news sentiment representation and the gate control unit; obtain the news aggregation feature for true / false classification based on the attention weights of multiple different sentiment news processors and the first news processing feature; and obtain the predicted label of the first news based on the news aggregation feature for true / false classification and the false news detector.

[0200] It should be understood that the units included in the above-described multimodal false news detection device are merely a logical division based on the functions implemented by the device. In practical applications, the units can be superimposed or split. Furthermore, the functions implemented by the multimodal false news detection device provided in this embodiment correspond one-to-one with the multimodal false news detection method provided in the above-described embodiment. The more detailed processing flow implemented by the device has been described in detail in the first embodiment of the method described above, and will not be described in detail here.

[0201] Another embodiment of the present invention provides a computer device, the computer device including: a processor and a memory; the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device executes each step of the multimodal false news detection method in the method flow shown in the above method embodiment.

[0202] Another embodiment of the present invention provides a computer-readable storage medium storing computer instructions that, when executed on a computer device, cause the computer device to perform each step of the multimodal false news detection method in the method flow shown in the above method embodiment.

[0203] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0204] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for multi-modal fake news detection, the method comprising: Comprise: According to the BERT model, the feature encoding of each word included in the first image caption information is carried out, and the first caption representation is obtained. According to the BERT model, the feature encoding of each word included in the first evidence information is carried out, and the first evidence representation is obtained; wherein the first image caption information corresponds to the first picture information included in the first news selected from the data set, and the first evidence information corresponds to the first text information included in the first news selected from the data set; The first text representation and the first evidence representation, the first picture representation and the first caption representation pass through the co-attention block composed of two encoders in turn to obtain the first evidence enhanced text representation, the first text enhanced evidence representation, the first caption enhanced picture representation and the first picture enhanced caption representation; The three types of multi-modal features obtained based on the full connection mapping function are spliced to obtain the first multi-modal semantic enhanced representation; wherein the first text information and the first picture information included in the first news are respectively subjected to feature encoding to obtain the first text representation and the first picture representation; The three types of multi-modal features respectively include the first evidence enhanced text representation and the first caption enhanced picture representation, the first enhanced text representation and the first enhanced picture representation, the first text enhanced evidence representation and the first picture enhanced caption representation; The first text representation and the first picture representation pass through the self-attention mechanism, splice and map with the enhanced representation respectively to obtain the first single-modal text representation and the first single-modal picture representation in turn; The first text representation is based on BiLSTM network, self-attention mechanism, full connection layer and average to obtain the sentiment representation of the comprehensive analysis of the first text representation by all experts and the sentiment tendency evaluation value of the comprehensive analysis of the first text representation by all experts; The first news overall sentiment representation is obtained according to the sentiment representation of the comprehensive analysis of the first text representation by all experts and the sentiment representation of the comprehensive analysis of the first picture representation by all experts; The positive emotion probability of the first news overall is obtained according to the sentiment tendency evaluation value of the comprehensive analysis of the first text representation by all experts and the sentiment tendency evaluation value of the comprehensive analysis of the first picture representation by all experts; The positive emotion probability of the first news overall and the news true or false label are based on the cross entropy loss function to obtain the first sentiment reasoning detection loss function; The first single-modal text representation, the first single-modal picture representation, the first multi-modal semantic enhanced representation and the first news overall sentiment representation are spliced and fused to obtain the first news overall content feature, and the first news processing feature is obtained according to the first news overall content feature and different emotional news processors; The attention weights of multiple different emotional news processors are obtained according to the first news overall sentiment representation and the gate control unit; The news aggregation feature for true or false classification is obtained according to the attention weights of multiple different emotional news processors and the first news processing feature, and the prediction label of the first news is obtained according to the news aggregation feature for true or false classification and the false news detector.

2. The method of claim 1, wherein, The first text representation and the first empirical representation are obtained through a co-attention block composed of two encoders to obtain a first empirical enhanced text representation, specifically comprising: The first text representation and the first empirical representation included in the first news are mapped to the same dimension; the output of single-head attention and the integrated output of multi-head attention of the first text representation and the first empirical representation are determined by the following formula: MultiHead(T i ,R i ,R i )=concat(h1,h2,...,h m )·W O The integrated output of the multi-head attention and the first text representation are spliced to obtain the first empirical enhanced text representation as shown below through normalization and a fully connected layer: h' = Norm(T i + MultiHead(T i , R i , R i )) wherein h i represents the output vector of the jth head under single-head attention, T i represents the first text representation, R i represents the first empirical representation, W j q represents the query projection matrix corresponding to the jth head, represents the key projection matrix corresponding to the jth head, W j v represents the value projection matrix corresponding to the jth head, d k represents the dimension after the first text representation and the first empirical representation are projected into the same space, d h represents the dimension of each head output feature, m represents the number of heads of attention, MultiHead(T i ,R i ,R i ) represents the output result of the overall multi-head attention mechanism, W O represents the weight matrix, concat(·) represents the vector concatenation operation, h' represents the normalization operation on the first text representation and the output result of the overall multi-head attention mechanism, represents the first empirical enhanced text representation, Norm(·) represents the normalization layer, and FFN(·) represents the fully connected network.

3. The method of claim 1, wherein, The three types of multi-modal features obtained based on the fully connected mapping function are spliced to obtain a first multi-modal semantic enhancement representation, specifically comprising: The first empirical enhanced text representation and the first caption enhanced picture representation are spliced to obtain a first multi-modal feature; the first text representation and the first picture representation included in the first enhanced representation are spliced to obtain a second multi-modal feature; the first text representation and the first picture representation included in the first text enhanced empirical representation are spliced to obtain a third multi-modal feature; wherein the first text information and the first picture information included in the first news are obtained based on the first enhanced representation of the pre-trained multi-modal large model; The first multi-modal feature, the second multi-modal feature, and the third multi-modal feature are spliced to obtain the first multi-modal semantic enhancement representation as shown below: M ter = θ MLP M (concat(M1, M2, M3)) wherein M ter denotes the first multi-modal semantic enhanced representation, concat(·) denotes a vector concatenation operation, and θ denotes a text-image similarity, MLP M denotes a fully connected mapping function of the multi-modal feature, M1 denotes a first multi-modal feature, M2 denotes a second multi-modal feature, M3 denotes a third multi-modal feature, σ1(·), σ2(·), and σ3(·) respectively denote fully connected mapping functions, denotes a first empirical reinforcement text representation, denotes a first caption reinforcement image representation, denotes a first reinforcement text representation, denotes a first reinforcement image representation, denotes a first text reinforcement empirical representation, denotes a first image reinforcement caption representation.

4. The method of claim 1, wherein, The first text representation is spliced and mapped with the enhanced representation through a self-attention mechanism to obtain the first single-modal text representation as shown below: The picture representation is spliced and mapped with the enhanced representation through a self-attention mechanism to obtain the first single-modal picture representation as shown below: wherein, T tra represents the first single-modal text representation, represents the first reinforced text representation, T i represents the first text representation, concat(·) represents a vector concatenation operation, SelfAttention(T i ) represents a self-attention operation on the first text representation, MLP T (·) is a fully connected mapping function of the first single-modal text feature, V tra represents the first single-modal picture representation, V i represents the first picture representation, SelfAttention(V i ) represents a self-attention operation on the first picture representation, represents the first reinforced picture representation, MLP V (·) represents a fully connected mapping function of the first single-modal picture feature.

5. The method of claim 1, wherein, The first text representation is based on a BiLSTM network, a self-attention mechanism, a fully connected layer, and an average to obtain a sentiment representation of a comprehensive analysis of the first text representation by all experts and a sentiment tendency evaluation value of a comprehensive analysis of the first text representation by all experts, specifically comprising: The first text representation is based on a BiLSTM network to obtain the hidden state as shown below: The hidden state is mapped according to a self-attention network and a fully connected layer to obtain a sentiment representation of the first text representation evaluation and a sentiment tendency evaluation value of the first text representation, respectively, and the sentiment representation of the first text representation evaluation and the sentiment tendency evaluation value of the first text representation are averaged, respectively, to obtain a sentiment representation of a comprehensive analysis of the first text representation by all experts and a sentiment tendency evaluation value of a comprehensive analysis of the first text representation by all experts: wherein, denotes a hidden state of a first text representation included in the i-th news, denotes a forward hidden state, denotes a backward hidden state, denotes a sentiment tendency evaluation value of the k-th expert on the first text representation included in the i-th news, denotes a sentiment representation evaluated by the k-th expert on the first text representation included in the i-th news, e T denotes a sentiment representation after comprehensive analysis of the first text representation by all experts, denotes a sentiment tendency evaluation value after comprehensive analysis of the first text representation by all experts, AVG denotes an averaging operation, W T denotes a learnable sentence parameter, b T denotes a learnable bias parameter.

6. The method of claim 1, wherein, The overall positive sentiment probability of the first news, the overall sentiment representation of the first news, and the first sentiment reasoning detection loss function are as follows: E = MLP E (concat(λe T ,(1-λ)e V )) L emo =∑-[ylog(y e )+(1-y)log(1-y e )] where y e represents the positive sentiment probability of the first news as a whole, represents the sentiment tendency evaluation value of the first text representation after comprehensive analysis by all experts, represents the sentiment tendency evaluation value of the first picture representation after comprehensive analysis by all experts, E represents the sentiment representation of the first news as a whole, e T represents the sentiment representation of the first text representation after comprehensive analysis by all experts, e V represents the sentiment representation of the first picture representation after comprehensive analysis by all experts, λ represents a weight factor, concat(·) represents a vector connection operation, MLP E (·) represents a fully connected mapping function of sentiment features, L emo represents the first sentiment reasoning detection loss function, y represents the true or false label of the news.

7. The method of claim 1, wherein, The overall content feature of the first news is as follows: M = [T tra ; V tra ; M ter ; E] The attention weights of the plurality of different sentiment news processors are as follows: a = [a1, a2,..., an]T x ] = softmax(G(E; θ G )) The news aggregation feature for true and false classification: m x = H x (M; θ x ) wherein M represents a first news overall content feature, T tra represents a first single-modal text representation, V tra represents a first single-modal picture representation, M ter represents a first multi-modal semantic enhanced representation, E represents a first news overall sentiment representation, G represents a gate control unit, θ G represents a parameter of the control unit, a represents a vector composed of [a1, a2, …, a x ], a x represents an expression weight corresponding to the xth sentiment type processor, M all represents a news aggregation feature for true or false classification, m x represents a first news processing feature, H x represents a news processor of the xth sentiment type, θ x represents a parameter in the corresponding processor.

8. The method of claim 1, wherein, The predicted label of the first news is as follows: wherein, represents a predicted label of the first news, W f and b f represents a trainable parameter, M all represents a news aggregation feature for true-false classification, and softmax(·) represents a mapping function.

9. The method of claim 1, wherein, After obtaining the predicted label of the first news, it further comprises: According to the predicted label of the first news, the inspection loss function of all news is obtained through a cross-entropy loss function as follows: The final loss function is obtained according to the all-news inspection loss function, the first sentiment reasoning detection loss function and a parameter factor: L final = γL emo + L fnd wherein L emo represents the first sentiment reasoning detection loss function, L fnd represents the news checking loss function, and γ represents a parameter factor.

10. A multi-modal fake news detection apparatus, characterized in that, Comprise: The first obtaining unit is configured to obtain first caption representation by performing feature encoding on each word included in first image caption information according to a BERT model, and obtain first evidence representation by performing feature encoding on each word included in first evidence information according to the BERT model; wherein the first image caption information corresponds to first picture information included in first news selected from a data set, and the first evidence information corresponds to first text information included in the first news; The second obtaining unit is configured to obtain first evidence reinforced text representation, first text reinforced evidence representation, first caption reinforced picture representation and first picture reinforced caption representation by sequentially passing the first text representation and the first evidence representation, the first picture representation and the first caption representation through a co-attention block composed of two encoders; and obtain first multi-modal semantic enhanced representation by splicing three types of multi-modal features obtained based on a full connection mapping function; wherein the first text information and the first picture information included in the first news are subjected to feature encoding to obtain the first text representation and the first picture representation; the three types of multi-modal features respectively comprise the first evidence reinforced text representation and the first caption reinforced picture representation, the first text reinforced representation and the first picture reinforced representation, and the first text reinforced evidence representation and the first picture reinforced caption representation; The third obtaining unit is configured to sequentially obtain first single-modal text representation and first single-modal picture representation by splicing and mapping the first text representation and the first picture representation through a self-attention mechanism and the reinforced representation respectively; The fourth obtaining unit is configured to obtain sentiment representation and sentiment tendency evaluation value of comprehensive analysis of the first text representation by the all experts based on a BiLSTM network, a self-attention mechanism, a full connection layer and averaging; obtain first news overall sentiment representation according to the sentiment representation of comprehensive analysis of the first text representation by the all experts and the sentiment representation of comprehensive analysis of the first picture representation by the all experts; and obtain positive sentiment probability of the first news overall according to the sentiment tendency evaluation value of comprehensive analysis of the first text representation by the all experts and the sentiment tendency evaluation value of comprehensive analysis of the first picture representation by the all experts; and obtain a first sentiment reasoning detection loss function based on a cross-entropy loss function according to the positive sentiment probability of the first news overall and a news true or false label. The fifth obtaining unit is configured to splice and fuse the first single-modal text representation, the first single-modal picture representation, the first multi-modal semantic enhanced representation, and the first overall sentiment of the news to obtain a first overall content feature of the news, and obtain a first news processing feature according to the first overall content feature of the news and different sentiment news processors; obtain attention weights of the different sentiment news processors according to the first overall sentiment of the news and a gate control unit; obtain a news aggregation feature for fake news classification according to the attention weights of the different sentiment news processors and the first news processing feature, and obtain a predicted label of the first news according to the news aggregation feature for fake news classification and an untruthful news detector.

Images