A multimodal fake news detection method based on inter-modal conflict

Abstract

The present invention discloses a multimodal fake news detection method based on inter-modal conflict. All word-region pairs between two modalities, i.e., text and image, are divided into matching parts and conflicting parts by similarity scores. The matching parts are aggregated according to cosine similarity, and the conflicting parts are bitwise added to obtain feature representations of the fragment pairs. The representations of the matching parts are then fused. m and the conflicting part c , and classification is performed based on these representations to obtain classification results. For real news, the focus is on clues provided by the matching part, while for fake news, the focus is on clues provided by the conflicting part. This improves the accuracy of the multimodal fake news detection method. In addition, during the training process, the present invention calculates the mean squared error loss function based on the weighted softmax function of the matching part and the conflicting part, and uses it as part of the loss function, thus further improving the accuracy of the modal fake news detection method.

Description

Technical Field

[0001] The present invention belongs to the technical field of fake news detection, and more specifically, relates to a multimodal fake news detection method based on inter-modal conflict. Background Art

[0002] The main goal of Fake News Detection (FND) is to determine the authenticity of a given piece of news content. In the prior art, common fake news detection methods include two paradigms: unimodal and multimodal. Unimodal fake news detection often targets a piece of text. To determine the authenticity of this text, it is necessary to collect social context information about the text, such as the dissemination network and online comments. However, recent research has shown that multimodal fake news spreads more widely and deeply, making the detection of multimodal fake news (generally containing text and images) more meaningful.

[0003] Considering that it is impossible to collect sufficient social context information when fake news is released, existing multimodal fake news detection methods mostly focus on judging the authenticity of news based on the multimodal content of the news. Among them, the multimodal fake news detection method based on the attention mechanism can effectively integrate the fine-grained features between different modalities and achieve better detection results.

[0004] Existing multimodal fake news detection methods based on attention mechanisms typically use a query segment from one modality (such as a word in text or an area in an image) as a query segment. They then calculate its similarity score with all segments from the other modality, selecting segments with high similarity scores (i.e., relevant segments) as complementary information. This allows them to explore the consistency between different modalities and determine the authenticity of news content. However, for fake news detection, conflicts between different modalities are more important, and these conflicts often exist in segments with low similarity scores. This means that the accuracy of existing multimodal fake news detection methods based on attention mechanisms needs to be improved. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a multimodal fake news detection method based on inter-modal conflict to further improve the accuracy of fake news detection.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a multimodal fake news detection method based on inter-modal conflict, which is characterized by comprising the following steps:

[0007] (1) Feature extraction

[0008] A feature encoder is used to extract features from multimodal news. The feature encoder includes a text encoder and an image encoder. Multimodal news contains content in multiple different modalities, where text and images are denoted as W and R respectively.

[0009] For text W, we first use the text encoder to extract the text embedding representation Where N is the number of words in the text W, Represents the features extracted from the i-th word in the text W, which is dimension d W Column vector of , Indicates d W A real number matrix with N rows and N columns, where one column is the feature of a word, is then embedded into the text representation E using a 1D convolutional network with kernel sizes of 1, 2, and 3 and a fully connected layer. W Map to a subspace of dimension d to get word features Among them, t i Features The mapped feature is a d-dimensional column vector;

[0010] For an image R, we first use an image encoder to extract the image region embedding representation Where M represents the number of image regions, Represents the features extracted from the jth image region in image R, with dimension d R Column vector of , Indicates d W A real number matrix with M rows and M columns, where one column represents the feature of an image region. Then, a fully connected layer is used to embed the image region and map it to a subspace of dimension d to obtain the regional feature. Among them, v j Features The mapped feature is a d-dimensional column vector;

[0011] (2) Segmentation

[0012] First, for word features T = {t1, t2, ..., t N} and regional features V={v1,v2,…,v M}, the formula for calculating the cosine similarity of word-region pairs, i.e., fragment pairs, is:

[0013]

[0014] Where superscript T represents transposition, i∈[1,N], j∈[1,M], s ij ∈[-1,1];

[0015] Then, set a hyperparameter λ∈[0,1) as the threshold and transform the similarity matrix S=(s ij ) N×M Split into two parts:

[0016]

[0017] Among them, S m Indicates the matching part, S c Indicates the conflicting part;

[0018] (3) Integration

[0019] For the matching part S m , if there is a cosine similarity s of the i-th row in the similarity matrix S ij , then refer to the attention mechanism to aggregate them according to cosine similarity:

[0020]

[0021] in, represents the word feature representation that contains complementary information of image modalities, and σ represents the softmax function;

[0022] For the conflicting part S c The cosine similarity s ij , the feature representation of the fragment pair is obtained by bitwise addition:

[0023]

[0024] Two multi-layer perceptron networks are used to calculate word feature representations respectively Consistency score And the feature representation c ij Conflict score

[0025]

[0026]

[0027] Among them, MLP represents the multi-layer perceptron network, sigmoid represents the activation function;

[0028] The matching part and the conflicting part including the cosine similarity are fused according to the corresponding scores to obtain the representation z of the matching part m and the conflicting part c :

[0029]

[0030] (4) Classification

[0031] First, for the matching part and the conflict part, the following classifiers are used to learn their weights respectively:

[0032]

[0033] Among them, w m For matching partial weights, w c is the weight of the conflict part, W z and b z is the trainable parameter of the classifier, W z is a row vector of dimension d, b z is the bias, ° represents the weight w of the splicing or matching part m The conflicting weight w c As two elements form a two-dimensional row vector, σ represents the softmax function, and its value w mc Also a two-dimensional row vector, that is, a 1×2 matrix;

[0034] Then, the multimodal representation z of the multimodal news is obtained by adding the weights, and the classification result is obtained

[0035]

[0036]

[0037] in, Representation z of the spliced ​​or matched part m and the conflicting part c As two columns form a d×2 matrix, the obtained multimodal representation z is a 1×d matrix, are the trainable parameters of the classifier, d f ×d matrix, d f ×1 matrix, ReLU is the rectified linear unit, 2×d f The matrix, is a 2×1 matrix, classification results It is a two-dimensional row vector, i.e. a 1×2 matrix, where the first and second elements represent the true and false scores of the multimodal news respectively;

[0038] (5) Training

[0039] For classification results Calculate the cross entropy loss function L1 with the multimodal news label y, and at the same time for the two-dimensional row vector w mc , calculate its difference with the region label y pThe mean square error loss function L2 is used to obtain the loss function L = L1 + L2 of the training classifier. The classifier is trained based on the loss function L and the trainable parameters of the entire model are updated. The label y of the real multimodal news is [1,0], and the corresponding regional label y p is [1,0]; the label y of false multimodal news is [0,1], and the corresponding regional label y p is [0,1];

[0040] (6) Detection

[0041] For multimodal news, process it according to steps (1) to (4) to obtain the classification results If the false score is greater than the true score, it is considered to be false news; otherwise, it is considered to be true news.

[0042] The object of the present invention is achieved in this way.

[0043] The present invention proposes a multimodal fake news detection method based on inter-modal conflict. All word-region pairs between two modalities, i.e., text and image, are divided into matching parts and conflicting parts by similarity scores. The matching parts are aggregated by cosine similarity, and the conflicting parts are bitwise added to obtain feature representations of the fragment pairs. The representations of the matching parts are then fused. m and the conflicting part c , and classify according to these representations to obtain classification results For real news, the focus is on clues provided by the matching portion, while for fake news, the focus is on clues provided by the conflicting portion, which improves the accuracy of the multimodal fake news detection method. Furthermore, during training, the present invention calculates a mean squared error loss function based on a weighted softmax function of the matching and conflicting portions, and uses this as part of the loss function, further improving the accuracy of the modal fake news detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of a specific implementation of the multimodal fake news detection method based on inter-modal conflict of the present invention;

[0045] Figure 2 This is a schematic diagram of the principles of a specific implementation of the multimodal fake news detection method based on inter-modal conflict of the present invention. DETAILED DESCRIPTION

[0046] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0047] Figure 1 This is a flowchart of a specific implementation of the multimodal fake news detection method based on inter-modal conflict of the present invention.

[0048] In this embodiment, if Figure 1 As shown, the multimodal fake news detection method based on inter-modal conflict of the present invention includes the following steps:

[0049] Step S1: Feature extraction

[0050] In the present invention, a feature encoder is used to extract features from multimodal news. Figure 2 As shown in Figure 1, the feature encoder includes a text encoder and an image encoder. Multimodal news contains content in multiple different modalities, where text and images are denoted as W and R respectively.

[0051] For text W, we first use the text encoder to extract the text embedding representation Where N is the number of words in the text W, Represents the features extracted from the i-th word in the text W, which is dimension d W Column vector of , Indicates d W A real number matrix with N rows and N columns, where one column is the feature of a word, is then embedded into the text representation E using a 1D convolutional network with kernel sizes of 1, 2, and 3 and a fully connected layer. W Map to a subspace of dimension d to get word features Among them, t i Features The mapped feature is a d-dimensional column vector.

[0052] For an image R, we first use an image encoder to extract the image region embedding representation Where M represents the number of image regions, Represents the features extracted from the jth image region in image R, with dimension d R Column vector of , Indicates d W A real number matrix with M rows and M columns, where one column represents the feature of an image region. Then, a fully connected layer is used to embed the image region and map it to a subspace of dimension d to obtain the regional feature. Among them, v j Features The mapped feature is a d-dimensional column vector.

[0053] In this embodiment, the text encoder is a pre-trained BERT encoder, and the image encoder is a SwinTransformer encoder. Figure 2 As shown, the text is extracted by the text encoder and mapped to obtain word features t1, t2, ..., t N , the image is extracted by the image encoder and mapped to obtain the regional features v1, v2,…, v M , are both d-dimensional column vectors.

[0054] Step S2: Segmentation

[0055] The present invention divides all segment pairs (word-region pairs) into matching parts and conflicting parts according to the similarity scores between different modal segments, such as Figure 2 As shown, the segmentation module uses cosine similarity to evaluate the relevance of segments of different modalities.

[0056] First, for word features T = {t1, t2, ..., t N} and regional features V={v1,v2,…,v M}, the formula for calculating the cosine similarity of word-region pairs, i.e., fragment pairs, is:

[0057]

[0058] Where superscript T represents transposition, i∈[1,N], j∈[1,M], s ij ∈[-1,1].

[0059] Then, set a hyperparameter λ∈[0,1) as the threshold and transform the similarity matrix S=(s ij ) N×M Split into two parts:

[0060]

[0061] Among them, S m Indicates the matching part, S c Indicates the conflicting part.

[0062] Step S3: Fusion

[0063] like Figure 2 As shown, the present invention adopts a fusion module to fuse the matching part and the conflicting part including the cosine similarity according to the corresponding scores, thereby obtaining the representation z of the matching part. m and the conflicting part c Specifically, the following steps are included:

[0064] For the matching part S m , if there is a cosine similarity s of the i-th row in the similarity matrix S ij , then refer to the attention mechanism to aggregate them according to cosine similarity:

[0065]

[0066] The column number of the cosine similarity sij of the i-th row

[0067] in, represents the word feature representation that contains complementary information of image modalities, and σ represents the softmax function.

[0068] For the conflicting part S c The cosine similarity s ij , the feature representation of the fragment pair is obtained by bitwise addition:

[0069]

[0070] Two multi-layer perceptron networks are used to calculate word feature representations respectively Consistency score And the feature representation c ij Conflict score

[0071]

[0072]

[0073] Among them, MLP represents the multi-layer perceptron network and sigmoid represents the activation function.

[0074] The matching part and the conflicting part including the cosine similarity are fused according to the corresponding scores to obtain the representation z of the matching part m and the conflicting part c :

[0075]

[0076] Step S4: Classification

[0077] First, for the matching part and the conflict part, the following classifiers are used to learn their weights respectively:

[0078]

[0079] Among them, w m For matching partial weights, w c is the weight of the conflict part, W z and b z is the trainable parameter of the classifier, W z is a row vector of dimension d, b z is the bias, Indicates the weight w of the splicing or matching part m The conflicting weight w cAs two elements form a two-dimensional row vector, σ represents the softmax function, and its value w mc It is also a two-dimensional row vector, that is, a 1×2 matrix.

[0080] Then, the multimodal representation z of the multimodal news is obtained by adding the weights, and the classification result is obtained

[0081]

[0082]

[0083] in, Representation z of the spliced ​​or matched part m and the conflicting part c As two columns form a d×2 matrix, the obtained multimodal representation z is a 1×d matrix, are the trainable parameters of the classifier, d f ×d matrix, d f ×1 matrix, ReLU is the rectified linear unit, 2×d f The matrix, is a 2×1 matrix, classification results It is a two-dimensional row vector, i.e. a 1×2 matrix, where the first and second element values ​​represent the true and false scores of the multimodal news respectively.

[0084] Step S5: Training

[0085] For classification results Calculate the cross entropy loss function L1 with the multimodal news label y, and at the same time for the two-dimensional row vector w mx , calculate its difference with the region label y p The mean square error loss function L2 is used to obtain the loss function L = L1 + L2 of the training classifier. The classifier is trained based on the loss function L and the trainable parameters of the entire model are updated. The label y of the real multimodal news is [1,0], and the corresponding regional label y p is [1,0]; the label y of false multimodal news is [0,1], and the corresponding regional label y p is [0,1].

[0086] Step S6: Detection

[0087] For multimodal news, process it according to the method from steps S1 to S3 to obtain the classification results If the false score is greater than the true score, it is considered to be false news; otherwise, it is considered to be true news.

[0088] Examples

[0089] The effect of the method of the present invention (ICL-net) was tested on two large-scale benchmark datasets, Twitter and Weibo. From the experimental results, it can be seen that the method proposed by the present invention is better than the state-of-the-art method.

[0090]

[0091] Table 1

[0092] It can be concluded from Table 1 that the accuracy of the model proposed in the present invention on the Twitter and Weibo datasets are 0.923 and 0.901 respectively, which are 2.6% and 1.6% higher than the highest level method HMCAN, thus achieving the purpose of the present invention.

[0093] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A multimodal fake news detection method based on inter-modal conflict, characterized by: The following steps are involved: (1) Feature extraction A feature encoder is used to extract features from multimodal news. The feature encoder includes a text encoder and an image encoder. Multimodal news contains content in multiple different modalities, where text and images are denoted as W and R respectively. For text W, we first use the text encoder to extract the text embedding representation Where N is the number of words in the text W, Represents the features extracted from the i-th word in the text W, which is dimension d W Column vector of , Indicates d W A real number matrix with N rows and N columns, where one column is the feature of a word, is then embedded into the text representation E using a 1D convolutional network with kernel sizes of 1, 2, and 3 and a fully connected layer. W Map to a subspace of dimension d to get word features Among them, t i Characterized by The mapped feature is a d-dimensional column vector; For an image R, we first use an image encoder to extract the image region embedding representation Where M represents the number of image regions, Represents the features extracted from the jth image region in image R, with dimension d R Column vector of , Indicates d W A real number matrix with M rows and M columns, where one column represents the feature of an image region. Then, a fully connected layer is used to embed the image region and map it to a subspace of dimension d to obtain the regional feature. Among them, v j Characterized by The mapped feature is a d-dimensional column vector; (2) Segmentation First, for word features T = {t1, t2, ..., t N } and regional features V={v1,v2,…,v M }, the formula for calculating the cosine similarity of word-region pairs, i.e., fragment pairs, is: Where superscript T represents transposition, i∈[1,N], j∈[1,M], s ij ∈[-1,1]; Then, set a hyperparameter λ∈[0,1) as the threshold and transform the similarity matrix S=(s ij ) N×M Split into two parts: Among them, S m Indicates the matching part, S c Indicates the conflicting part; (3) Integration For the matching part S m , if there is a cosine similarity s of the i-th row in the similarity matrix S ij , then refer to the attention mechanism to aggregate them according to cosine similarity: in, represents the word feature representation that contains complementary information of image modalities, and σ represents the softmax function; For the conflicting part S c The cosine similarity s ij , the feature representation of the fragment pair is obtained by bitwise addition: c ij =t i ⊕v j Two multi-layer perceptron networks are used to calculate word feature representations respectively Consistency score And the feature representation c ij Conflict score Among them, MLP represents the multi-layer perceptron network, sigmoid represents the activation function; The matching part and the conflicting part including the cosine similarity are fused according to the corresponding scores to obtain the representation z of the matching part m and the conflicting part c : (4) Classification First, for the matching part and the conflict part, the following classifiers are used to learn their weights respectively: Among them, w m For matching partial weights, w c is the weight of the conflict part, W z and b z is the trainable parameter of the classifier, W z is a row vector of dimension d, b z is the bias, Indicates the weight w of the splicing or matching part m The conflicting weight w c As two elements form a two-dimensional row vector, σ represents the softmax function, and its value w mc Is a two-dimensional row vector, i.e. a 1×2 matrix; Then, the multimodal representation z of the multimodal news is obtained by adding the weights, and the classification result is obtained in, Representation z of the spliced ​​or matched part m and the conflicting part c As two columns form a d×2 matrix, the obtained multimodal representation z is a 1×d matrix, are the trainable parameters of the classifier, d f ×d matrix, d f ×1 matrix, ReLU is the rectified linear unit, 2×d f The matrix, is a 2×1 matrix, classification results It is a two-dimensional row vector, i.e. a 1×2 matrix, where the first and second elements represent the true and false scores of the multimodal news respectively; (5) Training For classification results Calculate the cross entropy loss function L1 with the multimodal news label y, and at the same time for the two-dimensional row vector w mc , calculate its difference with the region label y p The mean square error loss function L2 is used to obtain the loss function L = L1 + L2 of the training classifier. The classifier is trained based on the loss function L and the trainable parameters of the entire model are updated. The label y of the real multimodal news is [1,0], and the corresponding regional label y p is [1,0], the label y of false multimodal news is [0,1], and the corresponding regional label y p is [0,1]; (6) Detection For multimodal news, process it according to steps (1) to (4) to obtain the classification results If the false score is greater than the true score, it is considered to be false news; otherwise, it is considered to be true news.

Images