Multi-modal unreal news detection method and device

Abstract

The invention discloses a multi-modal unreal news detection method and device, and relates to the field of multi-modal cognitive calculation. The method is used for deeply mining and analyzing news information, realizing semantic enhancement of multi-modal information and mining and applying news emotion features. The method comprises the following steps: performing feature coding on each word included in first image subtitle information according to a BERT model to obtain a first subtitle representation, and performing feature coding on each word included in first empirical information according to the BERT model to obtain a first empirical representation; the first text representation is based on a BiLSTM network, a self-attention mechanism, a full connection layer and averaging to obtain emotion representations of all experts on comprehensive analysis of the first text representation and emotion tendency evaluation values of all the experts on the comprehensive analysis of the first text representation; obtaining a first emotion reasoning detection loss function based on a cross entropy loss function according to the positive emotion probability of the first news whole body and the news authenticity label; according to the attention weights of the multiple different emotion news processors and the first news processing features, news aggregation features used for true and false classification are obtained, and according to the news aggregation features used for true and false classification and an unreal news detector, a prediction label of the first news is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal cognitive computing, and more specifically relates to a multimodal false news detection method and device. Background Art

[0002] With the rapid development and popularization of social networks, we have entered an era of information explosion. With its convenient and instant features, social networks have become the main platform for the public to obtain information, exchange ideas and share opinions. However, this has led to concerns about the authenticity and credibility of information on social networks. In this information overload environment, false information often spreads rapidly through social networks, causing great harm to individuals, organizations and even the entire society. At the same time, faced with massive information flows, the efficiency and accuracy of manual review are difficult to be satisfactory. Therefore, the technology of automatically detecting false information has become particularly important.

[0003] Early methods for detecting fake news focused on the extraction and use of manual features. These features mainly include: article structure and content semantic features, language style of text information, user attributes involved in dissemination, and dissemination rules. By extracting these deep features and building classification models using algorithms such as naive Bayes, support vector machines, and decision trees, false news detection can be achieved. Although these manually extracted features can improve the performance of fake news detection, manually extracting valuable features requires a lot of time and effort. In addition, manually extracted features are largely dependent on specific social platforms and are difficult to effectively generalize in different fields, languages, and topics.

[0004] With the development of deep learning and the improvement of computing power, the current fake news detection tasks are mainly divided into context-based methods and content-based methods. Context-based methods rely on the propagation structure of information and account information to detect false news. However, they are less realistic, require a lot of basic information or need to build a graph propagation structure, and the collection and processing of data are very complex.

[0005] Content-based methods mainly include single-modal and multi-modal methods. Single-modal methods mainly use single text content or image information to detect false news. However, single-modal methods do not consider the interactive complementation and fusion of multi-modal information, resulting in low detection efficiency and limited application scenarios. Therefore, multi-modal methods have received more and more attention.

[0006] In the multimodal false news detection method, researchers initially used long short-term memory networks to automatically extract and fuse the multimodal features of news. In addition, social comment features were introduced to improve the detection effect of false news. Later, methods based on events and semantic invariance were proposed, which improved the detection accuracy by eliminating specific event features and retaining shared features between events. Then, researchers introduced variational autoencoders to reconstruct data from multimodal representations, thereby promoting the correlation between model learning modalities. In the process of research, pre-trained models were gradually used to pre-process data, extract deep features, and improve the detection accuracy. Later, researchers gradually introduced entities in pictures and texts and external knowledge related to entities to enrich content and perform semantic enhancement. In addition, the widespread dissemination of false news is also inseparable from the emotional response it causes in readers, and emotional features are also used to help detect false news. With the emergence of CLIP, a language-image pre-training model based on contrastive learning, ideas such as image-text mismatch relationship, multimodal ambiguity learning, and image-text similarity have been widely used, which has further influenced and improved the alignment and fusion of multimodal information, and the multimodal false news detection task has also achieved excellent results.

[0007] In recent years, more multimodal models have been produced, providing efficient and accurate detection tools for the task of false news detection. Innovative work such as contrastive learning, co-attention mechanism, perceptual behavior simulation, explainability, causal reasoning, etc. has also been produced, and the development of multimodal false news detection has also achieved unprecedented improvement. However, the existing methods only use pixel-level or entity-level information for images, which is suitable for detecting tampering of image content but does not understand the scene and semantics of the image content; when targeting text, only the original content of the news is considered, ignoring the mining and analysis of external empirical information by large models; it is easy to cause low alignment between cross-modal features, poor fusion effect, and low detection accuracy. Moreover, using the same processor for news of different emotional types at the same time will cause a decrease in detection accuracy, and the difference in the distribution of positive and negative emotions between true and false news has not been applied. Summary of the invention

[0008] The embodiment of the present invention provides a multimodal false news detection method and device for in-depth mining and analysis of news information, achieving semantic enhancement of multimodal information and discovery and application of news sentiment features.

[0009] The embodiment of the present invention provides a multimodal false news detection method, comprising:

[0010] Performing feature encoding on each word included in the first image subtitle information according to the BERT model to obtain a first subtitle representation, and performing feature encoding on each word included in the first empirical information according to the BERT model to obtain a first empirical representation; wherein the first image subtitle information corresponds to first picture information included in the first news selected from the data set, and the first empirical information corresponds to first text information included in the first news selected from the data set;

[0011] The first text representation and the first empirical representation, the first picture representation and the first subtitle representation are sequentially passed through a common attention block composed of two encoders to obtain a first empirically enhanced text representation, a first text-enhanced empirical representation, a first subtitle-enhanced picture representation and a first picture-enhanced subtitle representation; the three types of multimodal features obtained based on the fully connected mapping function are concatenated to obtain a first multimodal semantically enhanced representation; wherein the first text information and the first picture information included in the first news are respectively feature encoded to obtain the first text representation and the first picture representation; the three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced picture representation, the first enhanced text representation and the first enhanced picture representation, and the first text-enhanced empirical representation and the first picture-enhanced subtitle representation;

[0012] The first text representation and the first image representation are respectively concatenated and mapped with the enhanced representation through a self-attention mechanism to obtain a first unimodal text representation and a first unimodal image representation in turn;

[0013] The first text representation is based on a BiLSTM network, a self-attention mechanism, a fully connected layer and an average to obtain a sentiment representation of the first text representation comprehensively analyzed by all experts and a sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts; the overall sentiment representation of the first news is obtained according to the sentiment representation of the first text representation comprehensively analyzed by all experts and the sentiment representation of the first picture representation comprehensively analyzed by all experts; the overall positive sentiment probability of the first news is obtained according to the sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts and the sentiment tendency evaluation value of the first picture representation comprehensively analyzed by all experts; the overall positive sentiment probability of the first news and the news truth or falsehood label are used to obtain a first sentiment reasoning detection loss function based on a cross entropy loss function;

[0014] The first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation and the first news overall sentiment representation are spliced ​​and fused to obtain the first news overall content feature, and the first news processing feature is obtained according to the first news overall content feature and different sentiment news processors; the attention weights of multiple different sentiment news processors are obtained according to the first news overall sentiment representation and the gate control unit; the news aggregation feature for true or false classification is obtained according to the attention weights of multiple different sentiment news processors and the first news processing feature, and the prediction label of the first news is obtained according to the news aggregation feature for true or false classification and the false news detector.

[0015] Preferably, the first text representation and the first empirical representation are subjected to a co-attention block consisting of two encoders to obtain a first empirically enhanced text representation, specifically comprising:

[0016] The first text representation and the first empirical representation included in the first news are mapped to the same dimension; the first text representation and the first empirical representation determine the output of the single-head attention and the comprehensive output of the multi-head attention through the following formula:

[0017]

[0018] MultiHead(T i ,R i ,R i )=concat(h1,h2,...,h m )·W O

[0019] The comprehensive output of the multi-head attention is concatenated with the first text representation, and the first empirically enhanced text representation is obtained through normalization and a fully connected layer as shown below:

[0020]

[0021] h′=Norm(T i +MultiHead(T i ,R i ,R i ))

[0022] Among them, h i represents the output vector of the jth head under single-head attention, T i represents the first text representation, R i represents the first empirical representation, represents the query projection matrix corresponding to the j-th head, represents the key projection matrix corresponding to the j-th head, represents the value projection matrix corresponding to the j-th head, d k represents the dimension after the first text representation and the first empirical representation are projected into the same space, d h Represents the dimension of each head output feature, m represents the number of attention heads, MultiHead(T i ,R i ,R i ) represents the output result of the overall multi-head attention mechanism, W O represents the weight matrix, concat(·) represents the vector concatenation operation, h′ represents the normalization operation of the output results of the first text representation and the overall multi-head attention mechanism, represents the first empirically enhanced text representation, Norm(·) represents the normalization layer, and FFN(·) represents the fully connected network.

[0023] Preferably, the step of concatenating the three types of multimodal features obtained based on the fully connected mapping function to obtain a first multimodal semantically enhanced representation specifically includes:

[0024] The first empirical enhanced text representation and the first subtitle enhanced picture representation are spliced ​​to obtain a first multimodal feature; the first enhanced text representation and the first enhanced picture representation included in the first enhanced representation are spliced ​​to obtain a second multimodal feature; the first text enhanced empirical representation and the first picture enhanced subtitle representation are spliced ​​to obtain a third multimodal feature; wherein the first text information and the first picture information included in the first news are used to obtain the first enhanced representation based on a pre-trained multimodal large model;

[0025] The first multimodal feature, the second multimodal feature and the third multimodal feature are concatenated to obtain the first multimodal semantic enhancement representation shown below:

[0026] M ter =θ·MLP M (concat(M1,M2,M3))

[0027]

[0028] Among them, M ter represents the first multimodal semantic enhancement representation, concat(·) represents the vector concatenation operation, θ represents the image-text similarity, MLP M (·) represents the fully connected mapping function of the multimodal feature, M1 represents the first multimodal feature, M2 represents the second multimodal feature, M3 represents the third multimodal feature, σ1(·), σ2(·) and σ3(·) represent the fully connected mapping function, represents the first empirically enhanced text representation, Indicates that the first subtitle strengthens the image representation, represents the first enhanced text representation, represents the first enhanced image representation, Indicates that the first text strengthens the empirical representation, Indicates the first picture enhanced caption representation.

[0029] Preferably, the first text representation is concatenated and mapped with the enhanced representation through a self-attention mechanism to obtain a first unimodal text representation as shown below:

[0030]

[0031] The image representation is concatenated and mapped with the enhanced representation through a self-attention mechanism to obtain a first unimodal image representation as shown below:

[0032]

[0033] Among them, T tra represents the first unimodal text representation, represents the first enhanced text representation, T i represents the first text representation, concat(·) represents the vector concatenation operation, SelfAttention(T i ) represents the self-attention operation on the first text representation, MLP T (·) is the fully connected mapping function of the first unimodal text feature, V tra represents the first unimodal image representation, V i Represents the first image representation, SelfAttention (V i ) represents the self-attention operation on the first image representation, represents the first enhanced image representation, MLP V (·) represents the fully connected mapping function of the first unimodal image features.

[0034] Preferably, the first text representation is based on a BiLSTM network, a self-attention mechanism, a fully connected layer and an average to obtain a sentiment representation of the first text representation comprehensively analyzed by all experts and a sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts, specifically including:

[0035] The first text representation is based on the BiLSTM network to obtain the following hidden state:

[0036]

[0037] The hidden state is mapped according to the self-attention network and the fully connected layer to obtain the sentiment representation of the first text representation evaluation and the sentiment tendency evaluation value of the first text representation respectively. Based on the average of the sentiment representations evaluated by multiple first text representations and the sentiment tendency evaluation values ​​of multiple first text representations, the sentiment representation after comprehensive analysis of the first text representation by all experts and the sentiment tendency evaluation value after comprehensive analysis of the first text representation by all experts are obtained:

[0038]

[0039] in, represents the hidden state of the first text representation included in the i-th news, express The forward hidden state, express Backward hidden state, represents the sentiment evaluation value of the kth expert on the first text representation included in the vth news, represents the sentiment representation of the k-th expert's evaluation of the first text representation included in the i-th news, e T represents the sentiment representation after comprehensive analysis of the first text representation by all experts, represents the sentiment tendency evaluation value after comprehensive analysis of the first text representation by all experts, AVG represents the average operation, and W T represents the learnable sentence parameters, b T represents the learnable offset parameter.

[0040] Preferably, the positive emotion probability of the first news as a whole, the first news as a whole emotion representation and the first emotion reasoning detection loss function are as follows:

[0041]

[0042] E=MLP E (concat(λe T ,(1-λ)e V ))

[0043]

[0044] Among them, y e represents the overall positive sentiment probability of the first news, It represents the sentiment evaluation value after comprehensive analysis of the first text representation by all experts. represents the sentiment evaluation value of the first picture representation after comprehensive analysis by all experts, E represents the overall sentiment representation of the first news, and e T represents the sentiment representation after comprehensive analysis of the first text representation by all experts, e Vrepresents the sentiment representation of the first picture by all experts, λ represents the weight factor, concat(·) represents the vector concatenation operation, and MLP E (·) represents the fully connected mapping function of sentiment features, L emo represents the first sentiment reasoning detection loss function, and y represents the true or false label of the news.

[0045] Preferably, the overall content features of the first news are as follows:

[0046] M=[T tra ; V tra ;M ter ; E]

[0047] The attention weights of the different emotional news processors are as follows:

[0048] a=[a1,a2,…,a x ]=softmax(G(E;θ G ))

[0049] The news aggregation features used for true or false classification are:

[0050]

[0051] m x =H x (M;θ x )

[0052] Among them, M represents the overall content characteristics of the first news, T tra represents the first unimodal text representation, V tra represents the first unimodal image representation, M ter represents the first multimodal semantic enhancement representation, E represents the first news overall sentiment representation, G represents the gate control unit, θ G represents the parameters of the control unit, a represents the combination of [a1, a2, …, a x ] is a vector composed of x represents the expression weight corresponding to the xth emotion type processor, M all represents the news aggregation features used for true or false classification, m x represents the first news processing feature, H x represents the news processor of the xth sentiment type, θ x Represents the parameters in the corresponding processor.

[0053] Preferably, the predicted label of the first news is as follows:

[0054]

[0055] in, represents the predicted label of the first news, W f and b f represents the trainable parameters, M all represents the aggregated features of news for true or false classification, and softmax(·) represents the mapping function.

[0056] Preferably, after obtaining the prediction label of the first news, the method further includes:

[0057] According to the predicted label of the first news, the inspection loss function of all news is obtained through the cross entropy loss function as shown below:

[0058]

[0059] The final loss function is obtained according to the inspection loss function of all news, the first sentiment reasoning detection loss function and the parameter factor:

[0060] L final =γL emo +L fnd

[0061] Among them, L emo represents the first sentiment inference detection loss function, L fnd represents the news inspection loss function, and γ represents the parameter factor.

[0062] The embodiment of the present invention provides a multi-modal false news detection device, comprising:

[0063] A first obtaining unit is used to perform feature encoding on each word included in the first image subtitle information according to the BERT model to obtain a first subtitle representation, and to perform feature encoding on each word included in the first empirical information according to the BERT model to obtain a first empirical representation; wherein the first image subtitle information corresponds to first picture information included in the first news selected from the data set, and the first empirical information corresponds to first text information included in the first news selected from the data set;

[0064] The second obtaining unit is used for sequentially passing the first text representation and the first empirical representation, the first picture representation and the first subtitle representation through a common attention block composed of two encoders to obtain a first empirically enhanced text representation, a first text-enhanced empirical representation, a first subtitle-enhanced picture representation and a first picture-enhanced subtitle representation; splicing the three types of multimodal features obtained based on the fully connected mapping function to obtain a first multimodal semantically enhanced representation; wherein the first text information and the first picture information included in the first news are respectively feature encoded to obtain the first text representation and the first picture representation; the three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced picture representation, the first enhanced text representation and the first enhanced picture representation, and the first text-enhanced empirical representation and the first picture-enhanced subtitle representation;

[0065] A third obtaining unit is used to concatenate and map the first text representation and the first image representation with the enhanced representation through a self-attention mechanism, respectively, to obtain a first unimodal text representation and a first unimodal image representation in turn;

[0066] The fourth obtaining unit is used for obtaining the sentiment representation of the first text representation comprehensively analyzed by all experts and the sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts based on the BiLSTM network, the self-attention mechanism, the fully connected layer and the averaging of the first text representation; obtaining the overall sentiment representation of the first news according to the sentiment representation comprehensively analyzed by all experts on the first text representation and the sentiment representation comprehensively analyzed by all experts on the first picture representation; obtaining the overall positive sentiment probability of the first news according to the sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts and the sentiment tendency evaluation value of the first picture representation comprehensively analyzed by all experts; obtaining the first sentiment reasoning detection loss function based on the positive sentiment probability of the first news as a whole and the news truth or falsehood label based on the cross entropy loss function;

[0067] The fifth obtaining unit is used to splice and fuse the first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation and the first news overall sentiment representation to obtain the first news overall content feature, and obtain the first news processing feature according to the first news overall content feature and different sentiment news processors; obtain the attention weights of multiple different sentiment news processors according to the first news overall sentiment representation and the gate control unit; obtain the news aggregation feature for true or false classification according to the attention weights of multiple different sentiment news processors and the first news processing feature, and obtain the predicted label of the first news according to the news aggregation feature for true or false classification and the false news detector.

[0068] An embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the multimodal false news detection method described above.

[0069] An embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored therein, and when the computer program is executed by a processor, the processor executes the multimodal false news detection method described above.

[0070] The embodiment of the present invention provides a multimodal false news detection method and device. The method uses a BERT model to perform knowledge enhancement on a first text representation and a first image representation, introduces a first subtitle representation and a first empirical representation, and solves the problem of insufficient data mining and analysis in existing methods. The first empirical enhanced text representation, the first text enhanced empirical representation, the first subtitle enhanced image representation, and the first image enhanced subtitle representation are obtained by using a common attention mechanism and a self-attention mechanism. The multimodal feature is combined with the above multiple representations to realize multimodal features. The multimodal feature and the first unimodal text representation and the first unimodal image representation can complement interaction and semantic enhancement, thereby solving the current problem of low detection accuracy caused by low similarity and poor fusion effect of cross-modal features. Furthermore, an expert network is designed to determine the overall positive emotion probability of the first news and the overall emotional representation of the first news, and an emotional reasoning loss is constructed to assist the false news detection task in combination with the difference in emotional tendencies in true and false news. A balanced learning module is designed to allow the model to perform special processing according to the news emotional type to improve the versatility and detection accuracy, thereby increasing the accuracy of multimodal false news detection on social networks and the applicability of the discrimination method in reality. This method can conduct in-depth mining and analysis of news information, realize semantic enhancement of multimodal information, and discover and apply the emotional characteristics of news. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 A schematic flow chart of a multimodal false news detection method provided by an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of the structure of a multimodal false news detection device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Figure 1 A flowchart of a multimodal false news detection method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method mainly includes the following steps:

[0076] Step 101, feature encoding each word included in the first image subtitle information according to the BERT model to obtain a first subtitle representation, and feature encoding each word included in the first empirical information according to the BERT model to obtain a first empirical representation; wherein the first image subtitle information corresponds to the first picture information included in the first news selected from the data set, and the first empirical information corresponds to the first text information included in the first news selected from the data set;

[0077] Step 102: the first text representation and the first empirical representation, the first picture representation and the first subtitle representation are sequentially passed through a common attention block composed of two Transformer encoders to obtain a first empirically enhanced text representation, a first text-enhanced empirical representation, a first subtitle-enhanced picture representation and a first picture-enhanced subtitle representation; the three types of multimodal features obtained based on the fully connected mapping function are concatenated to obtain a first multimodal semantically enhanced representation; wherein the first text information and the first picture information included in the first news are respectively feature encoded to obtain the first text representation and the first picture representation; the three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced picture representation, the first enhanced text representation and the first enhanced picture representation, and the first text-enhanced empirical representation and the first picture-enhanced subtitle representation;

[0078] Step 103: The first text representation and the first image representation are respectively concatenated and mapped with the enhanced representation through a self-attention mechanism to obtain a first unimodal text representation and a first unimodal image representation in turn;

[0079] Step 104, the first text representation obtains the sentiment representation of the comprehensive analysis of the first text representation by all experts and the sentiment tendency evaluation value of the comprehensive analysis of the first text representation by all experts based on the BiLSTM network, the self-attention mechanism, the fully connected layer and the averaging; the overall sentiment representation of the first news is obtained according to the sentiment representation of the comprehensive analysis of the first text representation by all experts and the sentiment representation of the comprehensive analysis of the first picture representation by all experts; the overall positive sentiment probability of the first news is obtained according to the sentiment tendency evaluation value of the comprehensive analysis of the first text representation by all experts and the sentiment tendency evaluation value of the comprehensive analysis of the first picture representation by all experts; the first sentiment reasoning detection loss function is obtained based on the positive sentiment probability of the first news as a whole and the news truth or falsehood label based on the cross entropy loss function;

[0080] Step 105, splicing and fusing the first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation and the first news overall sentiment representation to obtain the first news overall content feature, and obtaining the first news processing feature according to the first news overall content feature and different sentiment news processors; obtaining the attention weights of multiple different sentiment news processors according to the first news overall sentiment representation and the gate control unit; obtaining the news aggregation feature for true or false classification according to the attention weights of multiple different sentiment news processors and the first news processing feature, and obtaining the predicted label of the first news according to the news aggregation feature for true or false classification and the false news detector.

[0081] In practical applications, fake news detection is usually formulated as a binary classification problem. Multimodal social media news can be defined as:

[0082] N={T,V}(1)

[0083] Among them, T represents text information, V represents picture information, and N represents news.

[0084] Specifically, the true or false label corresponding to fake news detection can be represented by y. Specifically, when y=1, the fake news detection result indicates that the news is true; when y=0, the fake news detection result indicates that the news is false.

[0085] In practical applications, when the multimodal false news detection model is represented by M, the goal of this task is to use the multimodal false news detection model to judge the truth of news. The predicted labels can be used Specifically, for the i-th news N i ={T i ,V i The detection process can be expressed as:

[0086]

[0087] Among them, T i represents the i-th text information, V i Represents the i-th picture information, M(N i ) represents the detection of the i-th news by the multimodal false news detection model.

[0088] In the embodiment of the present invention, the processing process of two data sets, Weibo and Twitter, is taken as an example to illustrate the process and method of preprocessing.

[0089] Specifically, it is assumed that all news posts in the Weibo dataset are published on Weibo within a set time period, and all true news are verified by official news agencies. The dataset contains 4779 true news, 4749 false news, and 9528 pictures. News includes text, pictures, and social background. In specific applications, the data is cleaned, news without text or pictures is removed, and only one picture is retained for news with multiple pictures. The training set and test set are divided according to the official data.

[0090] The Twitter dataset is a well-known multimodal dataset for false news detection, consisting of news from multiple events. The number of images in this dataset is much smaller than the number of samples, which means that one image may be shared by multiple examples. The Twitter dataset contains 8720 real news and 7448 false news and 514 pictures. In specific applications, the dataset is processed to remove symbols such as "@", "#", "http:", emoticons, and hyperlinks in the text. Only news that contains both text and images is retained, and the training set and test set are divided according to the official data.

[0091] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture. It learns the language representation of each word and has achieved good results in many NLP (Natural Language Processing) downstream tasks. In an embodiment of the present invention, a pre-trained BERT model is used to perform feature encoding on each word included in the first text information corresponding to the first news to obtain a first text representation, which is specifically shown as follows:

[0092] T i ={w1,w2,…,w m} (3)

[0093] Among them, T i represents the i-th text representation, m represents the number of words included in the first text information, w1 represents the first word, and w mrepresents the mth word, d w Represents the dimensionality of word embeddings.

[0094] In practical applications, d w Determines the dimension of the vector. Different dimensions mean that it can contain different levels of semantic information. Assume that the word embedding dimension is d w =768, that is, the dimension of the hidden state vector corresponding to each word is 768, which can encode the rich semantic, grammatical and other language-level information of the word. m It indicates the hidden state of the corresponding word in its last output layer under the processing of the BERT model.

[0095] Swin Transformer is a model that further improves the Transformer by adopting hierarchical and localized ideas, and achieves state-of-the-art performance on many computer vision tasks.

[0096] In the embodiment of the present invention, Swin Transformer is used to perform feature encoding on the first picture information corresponding to the first news to obtain the first picture representation, which is specifically as follows:

[0097] V i ={v1,v2,…,v n} (4)

[0098] Among them, V i represents the i-th image representation, w and h represent the width and height of the i-th image, respectively, and n represents the number of selected areas of the i-th image. d v represents the dimension of the i-th image embedding.

[0099] In practical applications, v n represents the hidden state vector corresponding to a certain area in the last output layer of the i-th image in the Swin Transformer, d v Specifies the dimension of the image feature vector, which affects the richness and refinement of the information contained in the image features. If the i-th image embedding dimension d v =1024, which means that the feature vector corresponding to each region is 1024-dimensional. Such high-dimensional vectors are used to carry rich visual feature information such as texture and shape of each region of the image.

[0100] The key role of the CLIP model is to connect the semantic information of image information and text information. It can learn the correspondence between image information and text information, just like building a bridge between the image and the text related to it, which facilitates the matching of image information and text information. In the embodiment of the present invention, the pre-trained CLIP model can generate highly aligned text information and image information representation.

[0101] Specifically, based on the first news N i ={T i ,V i}, that is, news with both text information and image information, based on the pre-trained CLIP model, we can obtain the global, high-similarity, and representation-enhanced first enhanced text representation and first enhanced image representation, as shown below:

[0102]

[0103] in, represents the first enhanced text representation, represents the first enhanced image representation, T i represents the first text representation, V i represents the first image representation, d c represents the embedding dimension of the first text information and the first image information after CLIP enhancement, N i Indicates the i-th news.

[0104] In practical applications, the embedding dimensions of the first text information and the first image information after CLIP enhancement facilitate subsequent more comprehensive and effective multimodal information fusion and analysis operations. c =512, that is, the vector dimensions of the first text information and the first picture information are both 512 dimensions, which are the features enhanced by CLIP, so that the first text information and the first picture information are better corresponded and associated at the semantic level.

[0105] BLIP is a pre-trained multimodal large model that unifies visual language understanding and generation, and has rich language and visual knowledge. In an embodiment of the present invention, description information of image scenes and semantics can be obtained through BLIP.

[0106] In step 101, after the first picture information corresponding to the first news passes through the BLIP model, the first image subtitle information can be obtained. The first image subtitle information here corresponds to the first picture information selected from the data set, that is, the first image subtitle information can describe the scene and semantics expressed by the first picture in the first news; further, the first image subtitle information is input into the BERT model, and the BERT model is used to perform feature encoding on each word included in the first image subtitle information, and the first subtitle representation shown below can be obtained:

[0107] P i ={p1,p2,…,p z} (6)

[0108] Among them, P i represents the first subtitle representation of the i-th image, z represents the number of words included in the first image subtitle information, p z represents the hidden state of the last output layer of the BERT model corresponding to the word, d p Represents the dimensionality of word embeddings.

[0109] For example, if the caption of a sunset picture contains multiple words such as "golden", "sunset", and "sprinkled on", the number of words is the value of z, and p z Specifically, it refers to the hidden state vector of the last word in the last output layer when the BERT model processes the caption information of the first image. This vector carries the semantic and other related features of the word after the model has learned it. p It specifies the dimension size of each word embedding vector. Different dimensions mean that the characteristics of the word can be characterized from different degrees of richness.

[0110] Furthermore, ChatGPT can understand the content of the first text information and mine relevant empirical information on the Internet through training with a large amount of data in the knowledge base. ChatGPT can provide an inference basis for the truth or falsity of the first text information corresponding to the first news.

[0111] Specifically, after the first text information corresponding to the first news is mined through ChatGPT, the first empirical information can be obtained. The first empirical information here has a corresponding relationship with the first text information selected from the data set, that is, the first empirical information can show the reasoning basis for the truth or falsity of the first text information included in the first news. Further, the first empirical information is input into the BERT model, and the BERT model is used to perform feature encoding on each word included in the first empirical information, and the first empirical representation shown below can be obtained:

[0112] R i ={r1,r2,…,r u} (7)

[0113] Among them, R i represents the first empirical representation, u represents the number of words contained in the first empirical information, and r u Represents the hidden state vector of the last word in the last output layer of the BERT model contained in the first empirical information, d r Represents the dimensionality of word embeddings.

[0114] In practical applications, the co-attention mechanism can effectively help enhance the interaction between information. Its main structure is similar to that of the Transformer encoder. The attention module is an important component, which can capture the global constraints of all items in the sequence. The co-attention mechanism plays a key role in this process. Its structure is similar to that of the Transformer encoder, among which the attention module is particularly important, which can capture the global constraints of all items in the input sequence. In simple terms, it can analyze the correlation and importance of each element from a holistic perspective. Its input consists of three matrices: the query matrix Q, the key matrix K, and the value matrix V. They interact with each other, and the attention weights are obtained according to the similarity between the query matrix Q and the key matrix K to determine which part of the value matrix V is more important.

[0115] In step 102, the first text representation and the first empirical representation are mapped to the same dimension, and then the first text representation and the first empirical representation are used as inputs of the encoder, and the attention weight is calculated according to the cosine similarity of the first text representation and the first empirical representation to determine which part of the first empirical representation is more important and can be used as supplementary information for the first text representation. The entire multi-head attention process is as follows:

[0116]

[0117] MultiHead(T i ,R i ,R i )=concat(h1,h2,...,h m )·W O (9)

[0118] Among them, h i represents the output vector of the j-th head under single-head attention. In the embodiment of the present invention, it can also be expressed as the j-th head under single-head attention from the first empirical representation R i The semantic information related to the first text representation obtained in T i represents the first text representation, R i represents the first empirical representation, represents the query projection matrix corresponding to the j-th head, represents the key projection matrix corresponding to the j-th head, represents the value projection matrix corresponding to the j-th head, d k represents the dimension after the first text representation and the first empirical representation are projected into the same space, d h Represents the dimension of each head output feature, m represents the number of attention heads, MultiHead(T i ,R i ,R i ) represents the output result of the overall multi-head attention mechanism, W O represents the weight matrix, concat(·) represents a vector concatenation operation.

[0119] Then, the important part of the first empirical representation is concatenated and fused with the first text representation, and the first empirical enhanced text representation is obtained after normalization and full connection layer output:

[0120] h′=Norm(T i +MultiHead(T i ,R i ,R i )) (10)

[0121]

[0122] Among them, h′ represents the normalization operation of the output results of the first text representation and the overall multi-head attention mechanism, represents the first empirically enhanced text representation, Norm(·) represents the normalization layer, and FFN(·) represents the fully connected network.

[0123] It should be noted that, in the embodiment of the present invention, another encoder in the co-attention block is responsible for the supplementary enhancement of the text to the empirical evidence, that is, the first text representation and the first empirical representation are passed through a co-attention block composed of two Transformer encoders, in which the query matrix Q = R i , key matrix K = T i , value matrix V = T i , through an algorithm similar to the above formula (8)-formula (11), the first text enhanced empirical representation is obtained.

[0124] Similarly, the first image representation and the first subtitle representation can also be obtained by the algorithm related to the above formula (8)-formula (11), when the query matrix Q = P i , key matrix K = V i , value matrix V = V i When the query matrix Q = V i, key matrix K = P i , value matrix V = P i When , the first picture enhanced subtitle representation is obtained.

[0125] Specifically, the first empirically enhanced text representation, the first text-enhanced empirical representation, the first subtitle-enhanced picture representation, and the first picture-enhanced subtitle representation are as follows:

[0126]

[0127] in, represents the first empirically enhanced text representation, Indicates that the first subtitle strengthens the image representation, Indicates that the first text strengthens the empirical representation, represents the first image enhanced caption representation, T i represents the first text representation, R i represents the first empirical subtitle representation, V i represents the first image representation, P i represents the first subtitle representation, CoAttention(T i ,R i ) indicates that the first text representation and the first empirical representation are coordinated, CoAttention(V i ,P i ) indicates that the first image representation and the first subtitle representation are coordinated.

[0128] In an embodiment of the present invention, three multimodal features are proposed, which are respectively fused from text-related information and image-related information, to achieve deep multimodal semantic enhancement. Specifically, the first empirical enhanced text representation and the first subtitle enhanced image representation are spliced ​​to obtain the first multimodal feature; the first enhanced text representation and the first enhanced image representation included in the first enhanced representation are spliced ​​to obtain the second multimodal feature; the first text enhanced empirical representation and the first image enhanced subtitle representation are spliced ​​to obtain the third multimodal feature. The three multimodal features are specifically shown as follows:

[0129]

[0130] Among them, M1 represents the first multimodal feature, M2 represents the second multimodal feature, M3 represents the third multimodal feature, and σ1(·), σ2(·) and σ3(·) represent fully connected mapping functions respectively.

[0131] Next, the three types of multimodal features are concatenated and passed through the fully connected mapping function MLP of multimodal features. M (.) and introduce image-text similarity To adjust the weight relationship between the multimodal representation and the unimodal representation, the final knowledge-based first multimodal semantic enhanced representation is obtained, which is shown as follows:

[0132] M ter =θ·MLP M (concat(M1,M2,M3)) (14)

[0133] Among them, M ter represents the first multimodal semantic enhancement representation, concat(·) represents the vector concatenation operation, θ represents the image-text similarity, MLP M (·) represents the fully connected mapping function of multimodal features.

[0134] It should be noted that existing research believes that when the text and image content of news do not match, it is likely to be fake news. For this reason, the concept of image-text similarity is introduced in the embodiment of the present invention, and the weight relationship between multimodal representation and unimodal representation can be adjusted through image-text similarity. When the image-text similarity is high, it means that multimodal information is more important. When the image-text similarity is low, it means that unimodal information is more important.

[0135] In step 103, for the first text representation corresponding to the first news, the self-attention mechanism is first used to learn the internal connections between the words included in the first text representation and the importance of each word itself. For example, in a piece of news, the self-attention mechanism can be used to find that some key words may be more important. For example, in the first text representation of "In this sports event, an athlete broke the world record, which is an amazing achievement", the words related to "breaking the world record" may be assigned a higher importance weight after calculation by the self-attention mechanism, because they carry core semantic information.

[0136] After completing the self-attention operation, the result can be concatenated with the first enhanced text representation generated by CLIP, and then mapped and learned through a fully connected mapping function of a unimodal text feature, and finally the first unimodal text representation is obtained. In the implementation of the present invention, the first unimodal text representation is expressed by the following formula:

[0137]

[0138] Among them, T tra represents the first unimodal text representation, represents the first enhanced text representation, T i represents the first text representation, concat(·) represents the vector concatenation operation, SelfAttention(T i ) represents the self-attention operation on the first text representation, MLP T(·) is the fully connected mapping function of the first unimodal text feature.

[0139] Accordingly, for the first picture representation corresponding to the first news, the self-attention mechanism is first used to learn the internal connections between different regions of the first picture and the importance of each region itself. For example, in a landscape picture, after being processed by the self-attention mechanism, the area containing the main scenery (such as the mountain in the center of the picture) may be judged as more important, that is, it is given a higher weight, while the weight of the relatively less important area on the edge will be lower, thus highlighting the importance difference and correlation between the internal regions of the picture.

[0140] After self-attention, the internal connections and importance between different regions of the first image are learned. The result of this self-attention operation can be spliced ​​and mapped with the first enhanced image representation generated by CLIP, and finally the first unimodal image representation is obtained as shown below:

[0141]

[0142] Among them, V tra represents the first unimodal image representation, V i Represents the first image representation, SelfAttentoion(V i ) represents the self-attention operation on the first image representation, represents the first enhanced image representation, MLP V (·) represents the fully connected mapping function of the first unimodal image features.

[0143] In the embodiment of the present invention, it is preliminarily assumed that there may be inconsistency in the sentiment tendency between real news and false news. By performing sentiment statistics on Weibo and Twitter datasets respectively, it can be found that there are obvious differences in the sentiment tendency between real news and false news in Weibo datasets and Twitter datasets. Among them, false news expresses more negative emotions, while real news expresses more positive emotions, and the above characteristics can assist in false news detection. In the embodiment of the present invention, a sentiment analysis module is constructed to extract the sentiment features of news and analyze the sentiment tendency of news.

[0144] In step 104, the sentiment analysis module is used to extract the sentiment features of the first news and analyze the sentiment tendency of the first news, that is, the first text representation and the first picture representation included in the first news are input into the sentiment analysis module, and the sentiment analysis module can be used to learn the sentiment features of the news text and the news picture. K sentiment evaluation experts are set here to perform sentiment learning on the first text representation and the first picture representation respectively, and each evaluation expert is composed of a BiLSTM network, a self-attention network and a fully connected network.

[0145] Taking the first text representation as an example, the first text representation is input into a network composed of K sentiment assessment experts for processing. Each expert first contains a BiLSTM. After the first text representation is input into the BiLSTM network, due to the characteristics of each BiLSTM network, each network unit corresponds to a word embedding, which can interact each word included in the first text representation with its surrounding words, that is, considering the context information of each word to learn the sentiment characteristics of the text.

[0146] Each unit corresponds to a word embedding included in the first text representation. Each word interacts with its surrounding words, and the sentiment characteristics of the text are learned through context information. The hidden state of the first text representation included in the first news after passing through the BiLSTM network is obtained:

[0147]

[0148] in, represents the hidden state of the first text representation of the i-th news after passing through the BiLSTM network, express The forward hidden state, express Backward hidden state, [;] represents vector concatenation operation.

[0149] In practical applications, It comprehensively represents the first text included in the first news from front to back (forward hidden state ) and from back to front (backward hidden state ) analyzes the information obtained and integrates the forward hidden state and the backward hidden state through a vector connection operation.

[0150] Furthermore, the output of the BiLSTM network Then input into the self-attention network, the self-attention mechanism will highlight the parts with more distinct emotional features according to the calculated attention weights, further refine the key emotional information in the text, and obtain the emotional representation of the first text representation evaluation of the first news included by the set experts:

[0151]

[0152] in, represents the sentiment representation evaluated by the k-th expert on the first text representation included in the i-th news.

[0153] Then Through the fully connected layer, the classification task of positive and negative emotions is performed to obtain the emotional tendency evaluation value of the first text representation included in the first news by the set experts:

[0154]

[0155] in, W represents the sentiment tendency evaluation value of the kth expert on the first text representation included in the i-th news. This score reflects the degree to which the expert judges that the first text representation presents positive or negative sentiment. Here, 1 represents positive sentiment and 0 represents negative sentiment. T represents the learnable sentence parameters, b T represents a learnable offset parameter, and softmax(·) is a mapping function for calculating the sentiment score, which can transform the input into a form that conforms to the probability distribution so that the output result can be interpreted as the probability of belonging to different sentiment categories (positive or negative).

[0156] Next, since there are K such sentiment assessment experts, we need to and multiple By performing averaging processing, we can obtain the sentiment representation after comprehensive analysis of the first text representation by all experts and the sentiment tendency evaluation value after comprehensive analysis of the first text representation by all experts:

[0157]

[0158] Among them, e T represents the sentiment representation after comprehensive analysis of the first text representation by all experts, It represents the sentiment tendency evaluation value after comprehensive analysis of the first text representation by all experts, and AVG represents the average operation. represents the sentiment representation of the first expert's evaluation of the first text representation included in the i-th news, Represents the sentiment tendency evaluation value of the first expert on the first text representation included in the i-th news.

[0159] Accordingly, based on formula (17)-formula (21) in the above step 104, according to the first picture representation, the emotional representation of the first picture representation comprehensively analyzed by all experts and the emotional tendency evaluation value after the comprehensive analysis of the first picture representation by all experts can be obtained, which will not be repeated here.

[0160] Furthermore, in order to balance the importance of the emotions expressed by the first text representation and the first picture representation in the overall emotion of the news, a parameter λ is introduced in the embodiment of the present invention to adjust the emotion representation and the emotion tendency evaluation value after the comprehensive analysis of the picture and text, respectively. Specifically, the overall emotion representation of the first news is obtained according to the emotion representation of the comprehensive analysis of the first text representation by all experts and the emotion representation of the comprehensive analysis of the first picture representation by all experts; the overall positive emotion probability of the first news is obtained according to the emotion tendency evaluation value of the comprehensive analysis of the first text representation by all experts and the emotion tendency evaluation value of the comprehensive analysis of the first picture representation by all experts:

[0161]

[0162] E=MLP E (concat(λe T ,(1-λ)e V )) (twenty three)

[0163] Among them, λ represents the weight factor, y e represents the overall positive sentiment probability of the first news, It represents the sentiment evaluation value after comprehensive analysis of the first text representation by all experts. represents the sentiment evaluation value of the first picture representation after comprehensive analysis by all experts, E represents the overall sentiment representation of the first news, and e T represents the sentiment representation after comprehensive analysis of the first text representation by all experts, e V represents the sentiment representation of the first picture by all experts, concat(·) represents the vector concatenation operation, MLP E (·) represents the fully connected mapping function of sentiment features.

[0164] In practical applications, λ is a weight factor between 0 and 1 that adjusts the importance of text sentiment and image sentiment. When λ approaches 1, it means that text sentiment is more important when judging the overall sentiment of the news; when λ approaches 0, it means that image sentiment in the news is more important.

[0165] In practical applications, considering that false news tends to convey negative emotions, and true news tends to convey positive emotions, in the embodiment of the present invention, it is assumed that news expressing negative emotions are all false, and news expressing positive emotions are all true, and sentiment reasoning loss is constructed.

[0166] Specifically, y is used to represent the truth or falsity label of news, y = 1 represents true news, y = 0 represents false news, and y e Represents the news truth or falsehood prediction label obtained by sentiment reasoning, and then uses the cross entropy loss function to calculate the first sentiment reasoning detection loss function of all news:

[0167] L emo =∑-[ylog(y e )+(1-y)log(1-y e )] (twenty four)

[0168] Among them, L emo represents the first sentiment reasoning detection loss function, and y represents the true or false label of the news.

[0169] In practical applications, the cross entropy loss function is often used to measure the difference between the predicted results and the true labels. Here, it is used to measure the difference between the news truth prediction results obtained by sentiment reasoning and the actual truth. By minimizing the first sentiment reasoning detection loss function, the performance of the sentiment analysis module is optimized so that it can more accurately judge the truth of news based on sentiment characteristics.

[0170] News of the same emotional category have some commonalities, and news of different emotional categories have some differences. Previous studies ignored emotional categories and used the same processing for all news, which would result in a decrease in overall detection accuracy. The embodiment of the present invention constructs a balanced learning module.

[0171] In step 105, the first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation and the first news overall sentiment representation obtained in the above steps are spliced ​​and fused to obtain the first news overall content feature as shown below:

[0172] M=[T tra ; V tra ;M ter ; E] (25)

[0173] Among them, M represents the overall content characteristics of the first news, T tra represents the first unimodal text representation, V tra represents the first unimodal image representation, M ter represents the first multimodal semantic enhanced representation, and E represents the first news overall sentiment representation.

[0174] Furthermore, X news processors of different emotion types are set, and the first news processing features shown below are obtained according to the overall content features of the first news and the news processors of different emotions:

[0175] m x =H x (M;θ x ) (26)

[0176] Among them, H x represents the news processor of the xth sentiment type, θ x Represents the parameters in the corresponding processor, m x The processor H representing the xth emotion type x The first news processing feature is mapped to the first news overall content feature M.

[0177] At the same time, the attention weights of multiple different emotional news processors are obtained according to the overall emotional representation of the first news and the gate control unit:

[0178] a=[a1,a2,…,a x]=softmax(G(E;θ G )) (27)

[0179] Among them, a represents [a1, a2, …, a x ] is a vector composed of x represents the expression weight corresponding to the xth emotion type processor, G represents the gate control unit, θ G Indicates the parameters of the control unit.

[0180] Furthermore, the final news aggregation feature for true or false classification is obtained according to the attention weights of multiple different emotional news processors and the first news processing features:

[0181]

[0182] Among them, M all represents the news aggregation features used for true or false classification, a x represents the expression weight corresponding to the xth emotion type processor, m x represents the news processing features obtained by the xth sentiment type processor, x=1,…,X.

[0183] Furthermore, the news aggregation features used for true and false classification are sent to the false news detector and mapped to the binary classification results through the fully connected layer, and finally the predicted label of the first news is obtained:

[0184]

[0185] in, represents the predicted label of the i-th news, W f and b f represents the trainable parameters, M all represents the aggregated features of news for true or false classification, and softmax(·) represents the mapping function.

[0186] It should be noted that, in the embodiment of the present invention, y is used to represent the real label of the news. represents the predicted label of the first news obtained by the news aggregation features, and then the cross entropy loss function is used to determine the detection loss function of all news as shown below:

[0187]

[0188] In this embodiment, the predicted label of the first news in the main classification task and the first sentiment reasoning detection loss function in sentiment analysis are combined, and a parameter factor γ is introduced to adjust the proportional relationship between the two, and the final loss function shown below is obtained:

[0189] L final =γLemo +L fnd (31)

[0190] Among them, L emo represents the first sentiment inference detection loss function, L fnd represents the news inspection loss function, γ represents the parameter factor, and the value of γ is between 0 and 1.

[0191] In practical applications, a batch of samples is selected for training each time, and training is stopped when the training loss converges or the number of training times reaches the maximum set value.

[0192] The embodiment of the present invention provides a multimodal false news detection method and device. The method uses a BERT model to perform knowledge enhancement on a first text representation and a first image representation, introduces a first subtitle representation and a first empirical representation, and solves the problem of insufficient data mining and analysis in existing methods. The first empirical enhanced text representation, the first text enhanced empirical representation, the first subtitle enhanced image representation, and the first image enhanced subtitle representation are obtained by using a common attention mechanism and a self-attention mechanism. The multimodal feature is combined with the above multiple representations to realize multimodal features. The multimodal feature and the first unimodal text representation and the first unimodal image representation can complement interaction and semantic enhancement, thereby solving the current problem of low detection accuracy caused by low similarity and poor fusion effect of cross-modal features. Furthermore, an expert network is designed to determine the overall positive emotion probability of the first news and the overall emotional representation of the first news, and an emotional reasoning loss is constructed to assist the false news detection task in combination with the difference in emotional tendencies in true and false news. A balanced learning module is designed to allow the model to perform special processing according to the news emotional type to improve the versatility and detection accuracy, thereby increasing the accuracy of multimodal false news detection on social networks and the applicability of the discrimination method in reality. This method can conduct in-depth mining and analysis of news information, realize semantic enhancement of multimodal information, and discover and apply the emotional characteristics of news.

[0193] Based on the same inventive concept, an embodiment of the present invention provides a multimodal false news detection device. Since the principle of solving the technical problem of the device is similar to that of a multimodal false news detection method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0194] Figure 2 A schematic diagram of the structure of a multi-modal false news detection device provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the device includes: a first obtaining unit 201, a second obtaining unit 202, a third obtaining unit 203, a fourth obtaining unit 204 and a fifth obtaining unit 205.

[0195] A first obtaining unit 201 is used to perform feature encoding on each word included in the first image subtitle information according to the BERT model to obtain a first subtitle representation, and perform feature encoding on each word included in the first empirical information according to the BERT model to obtain a first empirical representation; wherein the first image subtitle information corresponds to the first picture information included in the first news selected from the data set, and the first empirical information corresponds to the first text information included in the first news selected from the data set;

[0196] The second obtaining unit 202 is used for sequentially passing the first text representation and the first empirical representation, the first picture representation and the first subtitle representation through a common attention block composed of two encoders to obtain a first empirically enhanced text representation, a first text-enhanced empirical representation, a first subtitle-enhanced picture representation and a first picture-enhanced subtitle representation; splicing the three types of multimodal features obtained based on the fully connected mapping function to obtain a first multimodal semantically enhanced representation; wherein the first text information and the first picture information included in the first news are respectively feature encoded to obtain the first text representation and the first picture representation; the three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced picture representation, the first enhanced text representation and the first enhanced picture representation, and the first text-enhanced empirical representation and the first picture-enhanced subtitle representation;

[0197] A third obtaining unit 203 is used to concatenate and map the first text representation and the first image representation with the enhanced representation through a self-attention mechanism, respectively, to obtain a first unimodal text representation and a first unimodal image representation in sequence;

[0198] The fourth obtaining unit 204 is used for obtaining, based on the BiLSTM network, the self-attention mechanism, the fully connected layer and the averaging, the sentiment representation of the first text representation comprehensively analyzed by all experts and the sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts; obtaining the overall sentiment representation of the first news according to the sentiment representation of the first text representation comprehensively analyzed by all experts and the sentiment representation of the first picture representation comprehensively analyzed by all experts; obtaining the overall positive sentiment probability of the first news according to the sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts and the sentiment tendency evaluation value of the first picture representation comprehensively analyzed by all experts; obtaining the first sentiment reasoning detection loss function based on the positive sentiment probability of the first news as a whole and the news truth or falsehood label based on the cross entropy loss function;

[0199] The fifth obtaining unit 205 is used to splice and fuse the first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation and the first news overall sentiment representation to obtain the first news overall content feature, and obtain the first news processing feature according to the first news overall content feature and different sentiment news processors; obtain the attention weights of multiple different sentiment news processors according to the first news overall sentiment representation and the gate control unit; obtain the news aggregation feature for true or false classification according to the attention weights of multiple different sentiment news processors and the first news processing feature, and obtain the predicted label of the first news according to the news aggregation feature for true or false classification and the false news detector.

[0200] It should be understood that the units included in the above multimodal false news detection device are only logical divisions based on the functions implemented by the device. In actual applications, the above units can be superimposed or split. In addition, the functions implemented by the multimodal false news detection device provided in this embodiment correspond to the multimodal false news detection method provided in the above embodiment. The more detailed processing flow implemented by the device has been described in detail in the above method embodiment 1 and will not be described in detail here.

[0201] Another embodiment of the present invention also provides a computer device, which includes: a processor and a memory; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes each step of the multimodal false news detection method in the method flow shown in the above method embodiment.

[0202] Another embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer device, the computer device executes each step of the multimodal false news detection method in the method flow shown in the above method embodiment.

[0203] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0204] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multimodal false news detection method, characterized in that: include: Performing feature encoding on each word included in the first image subtitle information according to the BERT model to obtain a first subtitle representation, and performing feature encoding on each word included in the first empirical information according to the BERT model to obtain a first empirical representation; wherein the first image subtitle information corresponds to first picture information included in the first news selected from the data set, and the first empirical information corresponds to first text information included in the first news selected from the data set; The first text representation and the first empirical representation, the first picture representation and the first subtitle representation are sequentially passed through a common attention block composed of two encoders to obtain a first empirically enhanced text representation, a first text-enhanced empirical representation, a first subtitle-enhanced picture representation and a first picture-enhanced subtitle representation; the three types of multimodal features obtained based on the fully connected mapping function are concatenated to obtain a first multimodal semantically enhanced representation; wherein the first text information and the first picture information included in the first news are respectively feature encoded to obtain the first text representation and the first picture representation; the three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced picture representation, the first enhanced text representation and the first enhanced picture representation, and the first text-enhanced empirical representation and the first picture-enhanced subtitle representation; The first text representation and the first image representation are respectively concatenated and mapped with the enhanced representation through a self-attention mechanism to obtain a first unimodal text representation and a first unimodal image representation in turn; The first text representation is based on a BiLSTM network, a self-attention mechanism, a fully connected layer and an average to obtain a sentiment representation of the first text representation comprehensively analyzed by all experts and a sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts; the overall sentiment representation of the first news is obtained according to the sentiment representation of the first text representation comprehensively analyzed by all experts and the sentiment representation of the first picture representation comprehensively analyzed by all experts; the overall positive sentiment probability of the first news is obtained according to the sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts and the sentiment tendency evaluation value of the first picture representation comprehensively analyzed by all experts; the overall positive sentiment probability of the first news and the news truth or falsehood label are used to obtain a first sentiment reasoning detection loss function based on a cross entropy loss function; The first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation and the first news overall sentiment representation are spliced ​​and fused to obtain the first news overall content feature, and the first news processing feature is obtained according to the first news overall content feature and different sentiment news processors; the attention weights of multiple different sentiment news processors are obtained according to the first news overall sentiment representation and the gate control unit; the news aggregation feature for true or false classification is obtained according to the attention weights of multiple different sentiment news processors and the first news processing feature, and the prediction label of the first news is obtained according to the news aggregation feature for true or false classification and the false news detector.

2. The method according to claim 1, characterized in that The first text representation and the first empirical representation are subjected to a co-attention block consisting of two encoders to obtain a first empirical enhanced text representation, specifically comprising: The first text representation and the first empirical representation included in the first news are mapped to the same dimension; the first text representation and the first empirical representation determine the output of the single-head attention and the comprehensive output of the multi-head attention through the following formula: MultiHead(T i ,R i ,R i )=concat(h1,h2,...,h m )·W O The comprehensive output of the multi-head attention is concatenated with the first text representation, and the first empirically enhanced text representation is obtained through normalization and a fully connected layer as shown below: h′=Norm(T i +MultiHead(T i ,R i ,R i )) Among them, h i represents the output vector of the jth head under single-head attention, T i represents the first text representation, R i represents the first empirical representation, W j q represents the query projection matrix corresponding to the jth head, W j k represents the key projection matrix corresponding to the jth head, W j v represents the value projection matrix corresponding to the j-th head, d k represents the dimension after the first text representation and the first empirical representation are projected into the same space, d h Represents the dimension of each head output feature, m represents the number of attention heads, MultiHead(T i ,R i ,R i ) represents the output result of the overall multi-head attention mechanism, W O represents the weight matrix, concat(·) represents the vector concatenation operation, h′ represents the normalization operation of the output results of the first text representation and the overall multi-head attention mechanism, represents the first empirically enhanced text representation, Norm(·) represents the normalization layer, and FFN(·) represents the fully connected network.

3. The method according to claim 1, characterized in that The three types of multimodal features obtained based on the fully connected mapping function are concatenated to obtain a first multimodal semantic enhancement representation, specifically including: The first empirical enhanced text representation and the first subtitle enhanced picture representation are spliced ​​to obtain a first multimodal feature; the first enhanced text representation and the first enhanced picture representation included in the first enhanced representation are spliced ​​to obtain a second multimodal feature; the first text enhanced empirical representation and the first picture enhanced subtitle representation are spliced ​​to obtain a third multimodal feature; wherein the first text information and the first picture information included in the first news are used to obtain the first enhanced representation based on a pre-trained multimodal large model; The first multimodal feature, the second multimodal feature and the third multimodal feature are concatenated to obtain the first multimodal semantic enhancement representation shown below: M ter =θ·MLP M (concat(M1,M2,M3)) Among them, M ter represents the first multimodal semantic enhancement representation, concat(·) represents the vector concatenation operation, θ represents the image-text similarity, MLP M (·) represents the fully connected mapping function of the multimodal feature, M1 represents the first multimodal feature, M2 represents the second multimodal feature, M3 represents the third multimodal feature, σ1(·), σ2(·) and σ3(·) represent the fully connected mapping function, represents the first empirically enhanced text representation, Indicates that the first subtitle strengthens the image representation, represents the first enhanced text representation, represents the first enhanced image representation, Indicates that the first text strengthens the empirical representation, Indicates the first picture enhanced caption representation.

4. The method according to claim 1, characterized in that The first text representation is concatenated and mapped with the enhanced representation through a self-attention mechanism to obtain a first unimodal text representation as shown below: The image representation is concatenated and mapped with the enhanced representation through a self-attention mechanism to obtain a first unimodal image representation as shown below: Among them, T tra represents the first unimodal text representation, represents the first enhanced text representation, T i represents the first text representation, concat(·) represents the vector concatenation operation, SelfAttention(T i ) represents the self-attention operation on the first text representation, MLP T (·) is the fully connected mapping function of the first unimodal text feature, V tra represents the first unimodal image representation, V i Represents the first image representation, SelfAttention (V i ) represents the self-attention operation on the first image representation, represents the first enhanced image representation, MLP V (·) represents the fully connected mapping function of the first unimodal image features.

5. The method according to claim 1, characterized in that The first text representation is based on the BiLSTM network, the self-attention mechanism, the fully connected layer and the averaging to obtain the sentiment representation of the comprehensive analysis of the first text representation by all experts and the sentiment tendency evaluation value of the comprehensive analysis of the first text representation by all experts, specifically including: The first text representation is based on the BiLSTM network to obtain the following hidden state: The hidden state is mapped according to the self-attention network and the fully connected layer to obtain the sentiment representation of the first text representation evaluation and the sentiment tendency evaluation value of the first text representation, and the sentiment representations of the multiple first text representation evaluations and the sentiment tendency evaluation values ​​of the multiple first text representation evaluations are averaged to obtain the sentiment representation after comprehensive analysis of the first text representation by all experts and the sentiment tendency evaluation value after comprehensive analysis of the first text representation by all experts: in, represents the hidden state of the first text representation included in the i-th news, express The forward hidden state, express Backward hidden state, represents the sentiment evaluation value of the kth expert on the first text representation included in the vth news, represents the sentiment representation of the k-th expert's evaluation of the first text representation included in the i-th news, e T represents the sentiment representation after comprehensive analysis of the first text representation by all experts, represents the sentiment tendency evaluation value after comprehensive analysis of the first text representation by all experts, AVG represents the average operation, and W T represents the learnable sentence parameters, b T represents the learnable offset parameter.

6. The method according to claim 1, characterized in that The positive sentiment probability of the first news as a whole, the first news overall sentiment representation and the first sentiment reasoning detection loss function are as follows: And=MLP E (concat(λe T ,(1-λ)e V )) L emo =∑-[ylog(y e )+(1-y)log(1-y e )] Among them, y e represents the overall positive sentiment probability of the first news, It represents the sentiment evaluation value after comprehensive analysis of the first text representation by all experts. represents the sentiment evaluation value of the first picture representation after comprehensive analysis by all experts, E represents the overall sentiment representation of the first news, and e T represents the sentiment representation after comprehensive analysis of the first text representation by all experts, e V represents the sentiment representation of the first picture by all experts, λ represents the weight factor, concat(·) represents the vector concatenation operation, and MLP E (·) represents the fully connected mapping function of sentiment features, L emo represents the first sentiment reasoning detection loss function, and y represents the true or false label of the news.

7. The method according to claim 1, characterized in that The overall content characteristics of the first news are as follows: M=[T tra ;V tra ;M ter ;E] The attention weights of the different emotional news processors are as follows: a\[a1,a2,…,a x ]=softmax(G(E;θ G )) The news aggregation features used for true or false classification are: m x =H x (M;θ x ) Among them, M represents the overall content characteristics of the first news, T tra represents the first unimodal text representation, V tra represents the first unimodal image representation, M ter represents the first multimodal semantic enhancement representation, E represents the first news overall sentiment representation, G represents the gate control unit, θ G represents the parameters of the control unit, a represents the combination of [a1, a2, …, a x ] is a vector composed of x represents the expression weight corresponding to the xth emotion type processor, M all represents the news aggregation features used for true or false classification, m x represents the first news processing feature, H x represents the news processor of the xth sentiment type, θ x Represents the parameters in the corresponding processor.

8. The method according to claim 1, characterized in that The predicted labels for the first news are as follows: in, represents the predicted label of the first news, W f and b f represents the trainable parameters, M all represents the aggregated features of news for true or false classification, and softmax(·) represents the mapping function.

9. The method according to claim 1, characterized in that After obtaining the prediction label of the first news, the method further includes: According to the predicted label of the first news, the inspection loss function of all news is obtained through the cross entropy loss function as shown below: The final loss function is obtained according to the inspection loss function of all news, the first sentiment reasoning detection loss function and the parameter factor: THE final =γL emo +L fnd Among them, L emo represents the first sentiment inference detection loss function, L fnd represents the news inspection loss function, and γ represents the parameter factor.

10. A multi-modal false news detection device, characterized in that: include: A first obtaining unit is used to perform feature encoding on each word included in the first image subtitle information according to the BERT model to obtain a first subtitle representation, and to perform feature encoding on each word included in the first empirical information according to the BERT model to obtain a first empirical representation; wherein the first image subtitle information corresponds to first picture information included in the first news selected from the data set, and the first empirical information corresponds to first text information included in the first news selected from the data set; The second obtaining unit is used for sequentially passing the first text representation and the first empirical representation, the first picture representation and the first subtitle representation through a common attention block composed of two encoders to obtain a first empirically enhanced text representation, a first text-enhanced empirical representation, a first subtitle-enhanced picture representation and a first picture-enhanced subtitle representation; splicing the three types of multimodal features obtained based on the fully connected mapping function to obtain a first multimodal semantically enhanced representation; wherein the first text information and the first picture information included in the first news are respectively feature encoded to obtain the first text representation and the first picture representation; the three types of multimodal features respectively include the first empirically enhanced text representation and the first subtitle-enhanced picture representation, the first enhanced text representation and the first enhanced picture representation, and the first text-enhanced empirical representation and the first picture-enhanced subtitle representation; A third obtaining unit is used for concatenating and mapping the first text representation and the first image representation with the enhanced representation through a self-attention mechanism, respectively, to obtain a first unimodal text representation and a first unimodal image representation in sequence; The fourth obtaining unit is used for obtaining the sentiment representation of the first text representation comprehensively analyzed by all experts and the sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts based on the BiLSTM network, the self-attention mechanism, the fully connected layer and the averaging of the first text representation; obtaining the overall sentiment representation of the first news according to the sentiment representation comprehensively analyzed by all experts on the first text representation and the sentiment representation comprehensively analyzed by all experts on the first picture representation; obtaining the overall positive sentiment probability of the first news according to the sentiment tendency evaluation value of the first text representation comprehensively analyzed by all experts and the sentiment tendency evaluation value of the first picture representation comprehensively analyzed by all experts; obtaining the first sentiment reasoning detection loss function based on the positive sentiment probability of the first news as a whole and the news truth or falsehood label based on the cross entropy loss function; The fifth obtaining unit is used to splice and fuse the first unimodal text representation, the first unimodal image representation, the first multimodal semantic enhancement representation and the first news overall sentiment representation to obtain the first news overall content feature, and obtain the first news processing feature according to the first news overall content feature and different sentiment news processors; obtain the attention weights of multiple different sentiment news processors according to the first news overall sentiment representation and the gate control unit; obtain the news aggregation feature for true or false classification according to the attention weights of multiple different sentiment news processors and the first news processing feature, and obtain the predicted label of the first news according to the news aggregation feature for true or false classification and the false news detector.

Images