Multimodal false information detection system guided by empathy theory

Abstract

The present invention provides a multimodal false information detection system guided by empathy theory. The system is implemented by a multimodal false information detection model guided by empathy theory. The model is based on two components of empathy: cognitive empathy and emotional empathy. By integrating empathy theory, the accuracy of multimodal false information detection is improved. The cognitive empathy reasoning layer simulates the degree of cognitive empathy by comparing the consistency of comments and news semantics. The emotional empathy reasoning layer simulates the degree of emotional empathy by performing similarity analysis on news text emotions and comment emotions. Finally, the design of the fusion reasoning layer integrates cognitive empathy and emotional empathy to judge whether empathy is generated for news content, thereby improving the accuracy of false information detection.

Description

Technical Field

[0001] The present invention relates to the field of information detection technology under the field of artificial intelligence technology, and more specifically, to a multimodal false information detection system guided by empathy theory. Background Art

[0002] The spread of fake news through social media will pose a destructive threat to the sustainable development of society. False information not only damages public trust and disrupts social order, but also hinders the normal operation and progress of society. Most of the existing research on fake news detection methods regards fake news detection as a classification task. At present, based on the main features used by the classification model, from the perspective of methods, fake news detection can be roughly divided into three categories: content-based detection methods, social environment-based detection methods, and knowledge-based detection methods.

[0003] Among them, content-based fake news detection methods aim to extract various semantic features from news content and detect the authenticity of news through these features. There are some linguistic differences between fake news and real news, and fake news can be detected by distinguishing the language style of real and fake news texts. For example, fake news is more subjective than real news, and the first and second persons are used more frequently in fake news. Fake news contains more words that can be used for exaggeration (such as subjects, hyperfinite words, and modal adverbs), while real news often uses concrete (such as numbers), objective (such as third person), and positive words; the writing style of the author of fake news will be more extreme than that of real news. However, by extracting a complete set of content features from real and fake news, including the total number of words in the news, content length, number of capitalized words, special symbols, number of opening sentences, and offensiveness, the importance of the features is listed through experiments. It can be found that the total number of words, content length, and number of capitalized words have a greater impact on the discrimination of real news. Therefore, abbreviations and total number of words have a greater impact on the detection results of fake news.

[0004] The above-mentioned content-based fake news detection methods can discover the language features of true and false news, but fake news sometimes deliberately imitates the writing style of real news to mislead readers. Since content-based fake news detection methods cannot distinguish the feature differences between such fake news and real news, in order to solve this problem, social environment-based detection methods make full use of hidden information as auxiliary data, such as social background information and propagation paths in social networks. For example, Shu et al. explored the relationship between user data and fake news on social media, and used users' social participation as an auxiliary for false information detection. Shu et al. proposed a framework to simulate the ternary relationship between news publishers, news articles, and users, extract effective features from the participation behavior of news publishers and readers, and then capture the interaction between them. Studies have shown that the use of social background information can not only improve the effect of fake news detection, but also effectively predict it in the early stage. Another research direction detects fake news by simulating the propagation path of fake news in the network. For example, Monti et al. found that the propagation method is an important feature of fake news, surpassing other aspects such as news content, user data, and social behavior. Raza et al. proposed a fake news detection framework based on the Transformer architecture, which includes an encoder and a decoder part: the encoder part is used to learn the representation of fake news data, and the decoder part is used to predict future behavior based on past observations. The model utilizes the characteristics of news content and social context to improve classification accuracy.

[0005] Knowledge (KB)-based fake news detection detects the authenticity of news by verifying fake news and facts, so it is also called fact checking. Fact checking can be divided into two categories: manual verification and automatic verification. Manual methods use domain expert knowledge or crowdsourcing methods, which are highly accurate but inefficient and cannot meet the needs of the big data era. Therefore, automatic verification methods using natural language processing and machine learning techniques have become a hot area of ​​research. Fact checking first needs to construct a knowledge base or knowledge graph from the network through knowledge extraction, and then compare and verify the fake news with the knowledge base or knowledge graph to determine the authenticity of the news. For example, Pan et al. used knowledge graphs to detect fake news based on news content to solve the problem that computational fact checking is not comprehensive enough. Hu et al. developed a heterogeneous graph attention network to learn the context of news representation and encode the semantics of news content. By using an entity comparison network, the context entity representation is compared with the representation derived from the knowledge base (KB) to capture the consistency between the news content and the knowledge base.

[0006] In the above-mentioned problem of false information detection, researchers usually study directions based on news content, knowledge, social networks, etc. These detection methods based on the surface information itself still have the defects of being easily circumvented by counterfeiters and having low accuracy in false information detection. Summary of the invention

[0007] In view of the above-mentioned problems such as poor detection accuracy in the current field of false information detection, the purpose of the present invention is to provide a multimodal false information detection system guided by empathy theory, which uses empathy theory to explain the characteristics and patterns of false information to improve the accuracy of false information detection.

[0008] On the one hand, the present invention provides a multimodal false information detection system based on empathy theory, which is implemented based on a preset multimodal false information detection model based on empathy theory, and includes:

[0009] A feature extraction module is used to extract features of news content and comment content to obtain image semantic features and text semantic features in the news content and comment content;

[0010] An empathy reasoning module, used to simulate the degree of cognitive empathy and the degree of emotional empathy of the news content and the comment content through cognitive empathy reasoning and emotional empathy reasoning respectively based on the image semantic features and the text semantic features;

[0011] The fusion judgment module is used to fuse the degree of cognitive empathy and the degree of emotional empathy, and judge whether the news content and comment content have generated empathy for fake news based on the result of the fusion.

[0012] Among them, an optional solution is that the feature extraction module further includes:

[0013] An image semantic feature extraction unit, used to extract image semantic features from image information in the news content and comment content using a preset ResNet152 residual network; and

[0014] The text semantic feature extraction unit is used to extract text semantic features from the text information in the news content and the comment content using a preset BERT.

[0015] Among them, an optional solution is that the image semantic feature extraction unit further includes:

[0016] An image preprocessing unit, used to adjust the images in the news content and comment content to a preset size to obtain standardized image information;

[0017] The feature acquisition unit is used to acquire the image semantic features in the standard image information using the ResNet152 residual network.

[0018] Among them, an optional solution is that the feature acquisition unit further includes:

[0019] The coded image acquisition unit is used to represent ResNet (I) by separating the last fully connected layer and obtaining the output of the last convolutional layer as the coded image, wherein,

[0020] ResNet(I)={r i |r i ∈R 2048 ,i=1,2,...,R}

[0021] Among them, each r i is a 2048-dimensional vector representing an area on the image; image I is represented by: ResNet(I)∈R 2048×R ;

[0022] A linear transformation unit, used to project the encoded image representation ResNet (I) into: E G =W r ResNet(I)

[0023] in, is a trainable parameter, An encoded representation of the image content.

[0024] Among them, an optional solution is to also include a comment screening module for selecting a preset number of the most representative comments by calculating the difference between each comment and other comment scaffolds as a reference for cognitive empathy reasoning and emotional empathy reasoning.

[0025] Among them, an optional scheme is that the cognitive empathy reasoning includes: encoding the text information and image information in the news content and comment content based on a preset self-attention network; and inputting the encoded features into a preset cross-attention module for semantic interaction.

[0026] Among them, an optional solution is that the self-attention network applies a multi-head self-attention mechanism to respectively learn the global dependencies of the text semantic features and image semantic features extracted by the feature extraction layer at all positions; wherein,

[0027] Given a query Q, key K, and value Z, the formula for scaled dot product attention is:

[0028]

[0029] Among them, d represents the variance, K T Indicates the transposition of key K; sets the text content Image content Project the query, key, and value h times through different linear projections, and then perform scaled dot product attention in parallel on the results after h projections; multi-head attention is formally described as:

[0030]

[0031] O=MultiHead(Q,K,Z)=conc(head1,head2,...,head h )W 0

[0032] in, H=d / h;W o is a learnable parameter, W o ∈R d×d , conc(·) is a series operation, and It is the encoded text semantic features and image semantic features.

[0033] Among them, an optional solution is to capture the global dependency between text and image based on the cross attention mechanism, where

[0034] The encoded text semantic features Set as query, the encoded image semantic features Set as keys and values, guide the multimodal false information detection model guided by empathy theory to focus on consistent image areas through the text semantic features,

[0035]

[0036]

[0037] Among them, all W are trainable parameters, For semantic consistency from news to images.

[0038] Among them, an optional solution is to integrate the degree of cognitive empathy and the degree of emotional empathy, including:

[0039] The interaction between cognitive empathy and affective empathy is simulated by the co-attention block. The input of Co-Attn is <E T ,V>:

[0040]

[0041] Among them, Norm and FFN are normalized system and feedforward network respectively, H co-sg It is the fusion semantics for the interaction blocks of cognitive empathy and affective empathy;

[0042] The softmax function is used for classification output to improve the accuracy of fake news detection.

[0043] Among them, an optional scheme is that the Softmax function is used to issue a prediction of the task learning probability distribution, wherein the global loss forces the preset empathy theory-guided multimodal false information detection model to minimize the cross entropy error of training samples with true label y:

[0044] p=softmax(W p H co-sg +b p )

[0045] Loss=-Σylogp

[0046] Among them, p is the obtained softmax value, W p and b p is a learnable parameter and y is the true value.

[0047] From the above technical scheme, it can be seen that the multimodal false information detection system guided by empathy theory provided by the present invention uses empathy theory to explain the characteristics and patterns of false information, integrates empathy theory based on deep learning algorithms, and improves the accuracy of multimodal false information detection to maintain the network environment. Among them, the present invention is based on the two components of empathy: cognitive empathy and emotional empathy, and designs and implements a multimodal false information detection system guided by empathy theory. Through the cognitive empathy reasoning layer, the consistency of comments and news semantics is compared to simulate the degree of cognitive empathy; through the emotional empathy reasoning layer, the similarity analysis of news text emotions and comment emotions is performed to simulate the degree of emotional empathy; finally, by designing a fusion reasoning layer to integrate cognitive empathy and emotional empathy to determine whether empathy has been generated for the news content, thereby improving the accuracy of false information detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more apparent and easy to understand. In the accompanying drawings:

[0049] Figure 1 Schematic diagram of the framework structure of a multimodal false information detection system guided by empathy theory according to an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the logical structure of a multimodal false information detection model guided by empathy theory according to an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a logical framework of a feature extraction layer according to an embodiment of the present invention;

[0052] Figure 4 Schematic diagram of the logical framework of the cognitive empathy reasoning layer according to an embodiment of the present invention;

[0053] Figure 5 is an example of similarity calculation according to an embodiment of the present invention;

[0054] Figure 6 Schematic diagram of the logical framework of the emotional empathy reasoning layer according to an embodiment of the present invention;

[0055] Figure 7 According to an embodiment of the present invention, <cls>Schematic diagram of the corresponding vector input to the fully connected layer classification;

[0056] Figure 8 A schematic diagram of a logical framework of a fusion reasoning layer according to an embodiment of the present invention;

[0057] Fig. 9 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In the following description, for the purpose of illustration, in order to provide a comprehensive understanding of one or more embodiments, many specific details are set forth. However, it is apparent that these embodiments may also be implemented without these specific details. In other examples, for ease of describing one or more embodiments, known structures and devices are shown in the form of block diagrams.

[0059] In response to the many problems existing in the aforementioned existing false information detection systems, the present invention proposes a solution that integrates psychology or human cognition, using the principles of psychology to comprehensively evaluate the authenticity and credibility of information to improve the accuracy of false information detection.

[0060] In order to better illustrate the technical solution of the present invention, some technical terms involved in the present invention are briefly explained below.

[0061] BERT: Bidirectional Encoder Representation from Transformers, is one of the most popular NLP models. It is based on Transformer and is composed of multiple Transformer Encoders stacked up layer by layer.

[0062] ResNet (Residual Neural Network)-152: A residual network with a depth of 152 layers.

[0063] Scaled Dot-Product Attention is an attention mechanism commonly used in Transformer models. Scaled Dot-Product Attention is used to implement Multi-Head Attention in Transformer models. Specifically, Multi-Head Attention performs linear transformations on the input matrix for multiple heads, then calculates Scaled Dot-Product Attention for each head's transformation result, and finally concatenates the attention results of each head and outputs them through a linear transformation.

[0064] In order to solve the problem of low detection accuracy in existing false information detection schemes that only detect false information from superficial information based on news content, knowledge, social networks, etc., the present invention provides a multimodal false information detection system guided by empathy theory. Based on empathy theory, a preset multimodal false information detection model guided by empathy theory (hereinafter referred to as the model) for false news detection is designed and implemented. First, a feature extraction layer is designed to extract features of news content and comment content. Secondly, a cognitive empathy reasoning layer is designed to calculate the degree of cognitive empathy generated by readers, and an emotional empathy reasoning layer is designed to calculate the degree of emotional empathy generated by readers. Finally, a fusion reasoning layer is designed to fuse cognitive empathy and emotional empathy to determine whether empathy is generated for false news, thereby improving the accuracy of false information detection.

[0065] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0066] It should be noted that the following description of the exemplary embodiments is merely illustrative and is not intended to limit the present invention and its application or use. Technologies and devices known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the technologies and devices should be considered as part of the specification.

[0067] In order to illustrate the multimodal false information detection system based on empathy theory provided by the present invention, Figure 1 , Figure 2 The framework structure of the multimodal false information detection system guided by empathy theory and the model structure adopted in the embodiment of the present invention are respectively exemplarily marked. It should be noted that the drawings and the embodiments described below are only some implementations of the present invention and are not intended to limit the present invention. For ordinary technicians in this field, other drawings and implementations can be obtained based on these drawings and implementations without creative work.

[0068] refer to Figure 1 The multimodal false information detection system 100 provided in this embodiment is implemented based on a preset multimodal false information detection model based on empathy theory, and mainly includes the following modules:

[0069] A feature extraction module 110 is used to extract features of news content and comment content to obtain image semantic features and text semantic features in the news content and comment content;

[0070] An empathy reasoning module 120, configured to simulate the degree of cognitive empathy and the degree of emotional empathy of the news content and the commentary content through cognitive empathy reasoning and emotional empathy reasoning respectively based on the image semantic features and the text semantic features;

[0071] The fusion judgment module 130 is used to fuse the degree of cognitive empathy and the degree of emotional empathy, and judge whether the news content and commentary content have generated empathy for fake news based on the result of the fusion.

[0072] It can be seen from the above scheme that, in order to address the above technical problems existing in the prior art, the present invention mainly realizes multimodal false information detection guided by empathy theory by utilizing empathy theory to explain the characteristics and patterns of false information.

[0073] The above modules will be described in detail below in conjunction with the network structure adopted by the present invention.

[0074] In order to realize the above-mentioned multimodal false information detection system guided by empathy theory, the present invention designs the following based on empathy theory: Figure 2 The multimodal false information detection model based on empathy theory is shown. The multimodal false information detection system based on empathy theory provided by the present invention will be described in more detail below in conjunction with a specific model structure embodiment.

[0075] Specifically, as an example, Figure 2 The overall framework structure diagram of the multimodal false information detection model based on empathy theory according to an embodiment of the present invention is shown. Figure 2 As shown, the multimodal false information detection model 200 guided by empathy theory provided in this embodiment mainly includes four parts: feature extraction layer 210, cognitive empathy reasoning layer 220, emotional empathy reasoning layer 230 and fusion reasoning layer 240. Among them, the feature extraction layer 210 is mainly used to extract the features of news content and commentary content; the cognitive empathy reasoning layer 220 is used to calculate the degree of cognitive empathy generated by readers, and the emotional empathy reasoning layer 230 is used to calculate the degree of emotional empathy generated by readers. Finally, the fusion reasoning layer 240 combines cognitive empathy and emotional empathy to determine whether empathy is generated for fake news, so as to improve the accuracy of false information detection.

[0076] Figures 3 to 7 The logical frameworks of the feature extraction layer 210, cognitive empathy reasoning layer 220, emotional empathy reasoning layer 230 and fusion reasoning layer 240 according to the embodiment of the present invention are shown respectively. Among them, the feature extraction layer 210 corresponds to the feature extraction module of the multimodal false information detection system guided by empathy theory, the cognitive empathy reasoning layer 220 and the emotional empathy reasoning layer 230 correspond to the empathy reasoning module of the multimodal false information detection system guided by empathy theory, and the fusion reasoning layer 240 corresponds to the fusion judgment module of the multimodal false information detection system guided by empathy theory.

[0077] The logical framework of the feature extraction layer 210 is as follows: Figure 3 shown.

[0078] Due to the diversity of information, false information detection currently includes image information detection and text information detection. Therefore, there are two common types of features in this type of problem: one is image semantic features; the other is text semantic features. Figure 3 In the illustrated embodiment, in the feature extraction layer 210, the image semantic features in the news content and the comment content are extracted by the image semantic feature extraction unit using the ResNet152 residual network, and the text semantic features in the news content and the comment content are extracted by the text semantic feature extraction unit using the preset BERT, wherein the text semantic features include the news text and the news comments (excluding images).

[0079] In the process of image semantic feature extraction, for the coding of image processing, considering the different sizes of multimedia news images in social media, in order to reduce the model's dependence on certain attributes in the image, in a specific embodiment of the present invention, the image semantic feature extraction unit further includes an image preprocessing unit and a feature acquisition unit. Among them, the image preprocessing unit first adjusts the size of the images in the news content and comment content to 224×224 pixels, and then the feature acquisition unit uses the ResNet152 residual network to obtain the image semantic features.

[0080] Specifically, as an example, the last fully connected (FC) layer is separated by the encoded image acquisition unit and the output of the last convolutional layer is obtained:

[0081]

[0082] Among them, each r i is a 2048-dimensional vector representing an area on the image, so the image I can be represented as , R represents the vector dimension space. In order to make the visual features have the same dimension as the text semantic features, in a specific embodiment of the present invention, the linear transformation unit projects the encoded image representation ResNet (I) through linear transformation as follows:

[0083] E G = W r ResNet(I) (2)

[0084] in, is a trainable parameter. is the encoded representation of the image content, i.e., the extracted image semantic features, and b is the hidden size of BERT.

[0085] In the process of text semantic feature extraction, the text content sequence is encoded as E X ={x1,x2,...,x N },in is the representation of the i-th token of the pre-trained BERT encoding, and N is the length of the text content sequence. Encoded sequence is the output of the last layer of BERT encoder, i.e., the extracted text semantic features, and b is the hidden size of BERT.

[0086] Since the text semantic features include news text and comment content, for comments, we know how many comments there are under each news, and each comment is generally a text sequence. Therefore, the representation of each comment is the same as that of the news text, expressed using BERT, that is, Among them l Ci is the length of the ith comment. In order to select the most representative comments, in a specific embodiment of the present invention, a comment selection mechanism is designed. The comment screening module is used to extract the first K (the value of K can be limited according to specific detection needs) comments as the most representative comments for the subsequent cognitive empathy reasoning layer 220 and the emotional empathy reasoning layer 230 to perform empathy reasoning. The comment selection mechanism automatically calculates the difference between each comment and other comments. To this end, the comment selection mechanism of this embodiment optimizes an inter-sequence attention matrix Among them, N is the number of comments under a piece of news, and R represents the vector dimension space.

[0087] The entry (m, n) of the inter-sequence attention matrix U is used to save the difference between annotation m and annotation n (1≤m,n≤N,m≠n), which can be formalized as:

[0088]

[0089] in, is the activation function. All W and b are trainable parameters, and ⊙ represents the dot product operator. Therefore, in the present invention, the final choice is to have a high difference C K The top-K representative reviews are used as the reviews for cognitive empathy reasoning and affective empathy reasoning.

[0090] Figure 4 FIG. 2 shows the logical structure of the cognitive empathy reasoning layer 220 according to an embodiment of the present invention. Figure 2 and Figure 4 As shown together, the image semantic features and text semantic features extracted by the feature extraction layer 210 are input into the cognitive empathy reasoning layer 220 for cognitive empathy reasoning.

[0091] Cognitive empathy refers to an individual's ability to understand and share the thought processes of others. This ability enables people to establish deeper emotional connections and better understand the feelings and positions of others. The online disinhibition effect shows that online users express themselves more openly in a less restrictive online environment (Suler, 2004). The focus of cognitive empathy is on understanding. On social media, when users are exposed to a piece of news that they can or cannot understand, some people may honestly express their opinions through actions such as comments. From a machine's perspective, the reader's understanding of the text can be quantified as the semantic similarity between the comment and the news content.

[0092] When people see a piece of news, they usually look at the eye-catching image first, and then look at the text content in detail with the image information. This process can be abstracted as multimodal interaction, which can learn the consistent information between the text content and the image content. To this end, cross-modal alignment is designed in the present invention to simulate this process in news. Specifically, as an example, the text semantic features and image semantic features extracted by the feature extraction layer are first encoded by the self-attention network, and then the encoded features are input into the designed cross-attention module for semantic interaction.

[0093] In a specific embodiment of the present invention, a multi-head self-attention mechanism is applied to learn all global dependencies of text semantic features and image semantic features extracted by the feature extraction layer at all positions. Given a query Q (indicating the content that needs to be paid attention to at the current position), a key K (indicating the content at all positions, used to compare with the query to calculate the attention weight) and a value Z (used to perform weighted averaging of the content at different positions according to the attention weight), the formula for scaling dot product attention is:

[0094]

[0095] Among them, d represents the variance, K T Indicates the transposition of key K; sets the text content In order to capture more valuable information from news in parallel, in this embodiment, the query Q, key K, value Z are projected h times through different linear projections, and then the scaled dot product attention is performed on these results in parallel. Formally, the multi-head attention is described as:

[0096]

[0097] O=MultiHead(Q,K,Z)=conc(head1,head2,...,head h )W 0 (8)

[0098] in, (all where H = d / h), respectively represents the trainable parameters of query Q, key K, and value Z; W 0 are learnable parameters, conc(·) is a concatenation operation. and It is the encoded text semantic features and image semantic features output by the self-attention network.

[0099] The encoded text semantic features and image semantic features are input into the preset cross-attention module (Cross-Attn) for semantic interaction to capture the global dependency between the encoded text semantic features and image semantic features.

[0100] The cross-attention block is a variant of the standard multi-head attention block, which can capture the global dependencies between text and image. In the multimodal false information detection model guided by empathy theory of the present invention, a cross-attention module is developed from the text-image perspective. Specifically, the text-image cross-attention block in this embodiment (as shown in equations (9) and (10)) encodes the semantic features of the text Set as query Q, and transform the image semantic features Set as key K and value Z. In this way, text semantic features can guide the multimodal false information detection model guided by empathy theory to focus on consistent image regions.

[0101]

[0102] in, (all Where H = d / h) are the trainable parameters of query Q, key K, and value Z respectively; W 0 are learnable parameters, For semantic consistency from news text to images.

[0103] In order to discover the audience's views on the authenticity of different modal information of unconfirmed news and explore the semantic consistency (i.e., cognitive empathy) of news and comments, after capturing the global dependency between the encoded text semantic features and image semantic features through the cross-attention module, the similarity analysis (Similarity-analysis) of the comment semantics and the fused multimodal news semantics is further performed to simulate the degree of cognitive empathy. Among them, the fused multimodal news semantics is the embodiment of the global dependency between the encoded text semantic features and image semantic features. By comparing the consistency of the comment semantics and the fused multimodal semantics, cognitive empathy can be judged. Specifically, as an example, the degree of cognitive empathy of the news content and the comment content is finally obtained by calculating the cosine similarity between them. Figure 5 This is an example of similarity calculation according to an embodiment of the present invention.

[0104] like Figure 5 As shown, a represents a vector, (x1, y1) represents vector a, b represents a vector, (x2, y2) represents vector b, θ is the angle between a and b, and the similarity analysis calculation results are as follows:

[0105]

[0106] Similarity analysis can illustrate the similarity between the fused multimodal news semantics and comment semantics.

[0107] Figure 6 FIG. 2 shows the logical structure of the emotional empathy reasoning layer 230 according to an embodiment of the present invention. Figure 6 and Figure 4 As shown in the figure, the text semantic features extracted by the feature extraction layer 210 and the text semantic features with high difference C selected by the comment selection mechanism K The top-K representative reviews are input into the sentiment empathy reasoning layer 230 for sentiment empathy reasoning.

[0108] In sentiment analysis or sentiment recognition tasks, text semantic features are usually superior to non-text semantic features, partly because text semantic features are derived from advanced language models or word embeddings trained on massive data sources, while image semantic features are artificially designed and relatively underdeveloped. Therefore, in the present invention, only the similarity between news text sentiment and comment sentiment is used to simulate the degree of emotional empathy.

[0109] BERT, one of the pre-trained natural language processing models, can also be used for sentiment analysis tasks. The power of the BERT model lies in that it is pre-trained on a large-scale text corpus, thereby learning rich text representations, enabling it to perform well in various natural language processing tasks, including sentiment analysis. BERT converts specific NLP tasks processed by universal models into agnostic NLP tasks under the BERT model. Users can fine-tune the BERT pre-trained model to enable it to handle a variety of NLP tasks, such as: single text classification (such as sentiment analysis), text pair classification (such as natural language inference), question and answer, text tagging (such as named entity recognition), etc. Therefore, in a specific embodiment of the present invention, the BERT model is used for sentiment analysis.

[0110] The sentiment analysis in the present invention is to use the BERT model to perform text classification, which is the task of the sentiment empathy reasoning layer 230. Therefore, it is only necessary to obtain the contextual comprehensive information of the entire comment sentence (' <cls>'), and then connected to a fully connected layer, the classification task can be achieved.

[0111] In this example, the base model Bert_base, Uncased, is used by default, using a 12-layer Transformer encoder block, 768 hidden units, and 12 self-attention heads. Figure 7 According to an embodiment of the present invention, <cls>The corresponding vector is input to the fully connected layer for classification. Figure 7 As shown, we only need to extract the comprehensive context information from the BERT output information' <cls>', and add an external fully connected layer to complete the sentiment analysis (text classification) effect.

[0112] After calculating the news text sentiment and comment sentiment respectively using the above calculation method, the similarity of the two sentiments is calculated using formula (11) to simulate the degree of emotional empathy.

[0113] After obtaining the degree of cognitive empathy and the degree of emotional empathy, a fusion analysis can be performed on the degree of cognitive empathy and the degree of emotional empathy to determine whether empathy has been generated for fake news, and then determine the truthfulness of the information containing the news content and commentary content.

[0114] Figure 8 FIG. 2 shows the logical structure of the fusion reasoning layer 240 according to an embodiment of the present invention. Figure 2 and Figure 8 As shown together, the cognitive empathy degree obtained through the cognitive empathy reasoning layer 220 and the emotional empathy degree obtained through the emotional empathy reasoning layer 230 are input into the fusion reasoning layer 240 for empathy fusion.

[0115] In order to explore the relationship between the two components of empathy (cognitive empathy and affective empathy), in a specific embodiment of the present invention, a collaborative attention block is designed to simulate the interaction between the two empathies. <E T ,V>:

[0116]

[0117] Among them, E T represents the degree of cognitive empathy, V represents the degree of emotional empathy; Norm and FFN are normalized systems and feedforward networks, H co-sg It is the fusion semantics of the interaction blocks of cognitive empathy and emotional empathy. We use the concat function to integrate the two empathies to form a comprehensive fusion representation.

[0118] Finally, the classification output is performed through the softmax function to improve the accuracy of fake news detection. Among them, the Softmax function emits a prediction of the probability distribution of task learning, and the global loss forces the multimodal false information detection model guided by empathy theory to minimize the cross entropy error of training samples with true labels y:

[0119] p=softmax(W p H co-sg +b p ) (16)

[0120] Loss=-∑ylogp (17)

[0121] Among them, p is the obtained softmax value, W p and b p is a learnable parameter and y is the true value.

[0122] like Fig. 9 As shown, the present invention also provides an electronic device, the electronic device comprising:

[0123] at least one processor; and,

[0124] a memory communicatively connected to at least one processor; wherein,

[0125] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can implement the functions of each module in the aforementioned multimodal false information detection system guided by empathy theory.

[0126] It can be understood by those skilled in the art that Fig. 9 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0127] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0128] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0129] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0130] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0131] The multimodal false information detection program 12 based on empathy theory and stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, the following functions can be realized:

[0132] A feature extraction module is used to extract features of news content and comment content to obtain image semantic features and text semantic features in the news content and comment content;

[0133] An empathy reasoning module, used to simulate the degree of cognitive empathy and the degree of emotional empathy of the news content and the comment content through cognitive empathy reasoning and emotional empathy reasoning respectively based on the image semantic features and the text semantic features;

[0134] The fusion judgment module is used to fuse the degree of cognitive empathy and the degree of emotional empathy, and judge whether the news content and comment content have generated empathy for fake news based on the result of the fusion.

[0135] Specifically, the specific implementation system of the processor 10 for the above instructions can refer to Figure 1 The description of the relevant embodiments in the corresponding embodiments will not be repeated here.

[0136] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0137] As described above with reference to the accompanying drawings, the multimodal false information detection system based on empathy theory proposed in accordance with the present invention is described by way of example. However, those skilled in the art should understand that various improvements can be made to the multimodal false information detection system based on empathy theory proposed in the present invention without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the attached claims.< / cls> < / cls> < / cls> < / cls>

Claims

1. A multimodal false information detection system based on empathy theory, characterized in that: The system is implemented based on a preset multimodal false information detection model guided by empathy theory, including: A feature extraction module is used to extract features of news content and comment content to obtain image semantic features and text semantic features in the news content and comment content; An empathy reasoning module is used to simulate the degree of cognitive empathy and the degree of emotional empathy of the news content and the comment content through cognitive empathy reasoning and emotional empathy reasoning respectively based on the image semantic features and the text semantic features; wherein the cognitive empathy refers to the ability of an individual to understand and share the thought process of others, and the degree of cognitive empathy is: after capturing the global dependency between the encoded text semantic features and the image semantic features through the cross-attention module, further similarity analysis is performed on the comment semantics and the fused multimodal news semantics to simulate the degree of cognitive empathy; the degree of emotional empathy is: using the BERT model to perform text classification on the news text and the comment to obtain the news text emotion and the comment emotion, and then simulating the degree of emotional empathy through the similarity between the news text emotion and the comment emotion; A fusion judgment module, used to fuse the degree of cognitive empathy and the degree of emotional empathy, and judge whether the news content and commentary content have generated empathy for fake news based on the result of the fusion; The integration of the degree of cognitive empathy and the degree of emotional empathy includes: The interaction between cognitive empathy and emotional empathy is simulated by the collaborative attention block, and the input of the collaborative attention block is <E T ,V>: Among them, E T represents the degree of cognitive empathy, V represents the degree of emotional empathy, Norm and FFN represent the normalization method and feedforward network respectively, d represents the variance, and H co-sg It is the fusion semantics of the interaction blocks of cognitive empathy and emotional empathy, and the concat function is used to integrate the cognitive empathy and the emotional empathy to form a comprehensive fusion representation; The softmax function is used for classification output to improve the accuracy of fake news detection.

2. The multimodal false information detection system based on empathy theory as claimed in claim 1, characterized in that: The feature extraction module further comprises: An image semantic feature extraction unit, used to extract image semantic features from image information in the news content and comment content using a preset ResNet152 residual network; and The text semantic feature extraction unit is used to extract text semantic features from the text information in the news content and the comment content using a preset BERT.

3. The multimodal false information detection system based on empathy theory as claimed in claim 2, characterized in that: The image semantic feature extraction unit further comprises: An image preprocessing unit, used to adjust the images in the news content and comment content to a preset size to obtain standardized image information; The feature acquisition unit is used to acquire the image semantic features in the standard image information using the ResNet152 residual network.

4. The multimodal false information detection system based on empathy theory as claimed in claim 3, characterized in that: The feature acquisition unit further comprises: The coded image acquisition unit is used to represent ResNet (I) by separating the last fully connected layer and obtaining the output of the last convolutional layer as the coded image, wherein, Among them, each r i is a 2048-dimensional vector representing an area on the image; image I is represented as: R represents the vector dimension space; A linear transformation unit, used to project the encoded image representation ResNet (I) into: E G =W r ResNet(I); in, is a trainable parameter, is the encoded representation of the image content, and b is the hidden size of BERT.

5. The multimodal false information detection system based on empathy theory as claimed in claim 4, characterized in that: It also includes a comment screening module for selecting a preset number of the most representative comments by calculating the difference between each comment and other comment scaffolds as a reference for cognitive empathy reasoning and emotional empathy reasoning.

6. The multimodal false information detection system based on empathy theory as claimed in claim 5, characterized in that: The cognitive empathic reasoning includes: Encoding the text information and image information in the news content and comment content based on a preset self-attention network; The encoded features are input into the preset cross-attention module for semantic interaction.

7. The multimodal false information detection system based on empathy theory as claimed in claim 6, characterized in that: The self-attention network applies a multi-head self-attention mechanism to respectively learn the global dependencies of the text semantic features and image semantic features extracted by the feature extraction layer at all positions; wherein, Given a query Q, key K, and value Z, the formula for scaled dot product attention is: Among them, d represents the variance, K T Indicates the transposition of key K; sets the text content Image content in, is the output of the last BERT encoder layer, N is the length of the text content sequence; query, key, and value are projected h times through different linear projections, and then scaled dot product attention is performed in parallel on the results after h projections; multi-head attention is formally described as: O=MultiHead(Q,K,Z)=conc(head1,head2,...,head h )W 0 in, H=d / h;W 0 are learnable parameters, conc(·) is a concatenation operation. and It is the encoded text semantic features and image semantic features.

8. The multimodal false information detection system based on empathy theory as claimed in claim 7, characterized in that: The global dependencies between text and images are captured based on the cross-attention mechanism, where The encoded text semantic features Set as query, the encoded image semantic features Set as keys and values, guide the multimodal false information detection model guided by empathy theory to focus on consistent image areas through the text semantic features, Among them, all W are trainable parameters, For semantic consistency from news text to images.

9. The multimodal false information detection system based on empathy theory as claimed in claim 8, characterized in that: The softmax function is used to issue predictions of the task learning probability distribution, where the global loss forces the preset empathy-theory-guided multimodal false information detection model to minimize the cross-entropy error of training samples with true labels y: p=softmax(W p H co-sg +b p ) Loss=-∑ylogp Among them, p is the obtained softmax value, W p and b p is a learnable parameter and y is the true value.

Images