Multimodal fake news detection method and device based on semantic truthfulness features

Abstract

The present invention discloses a multimodal false news detection method and device based on semantic authenticity features. The method comprises the following steps: obtaining text modal data and corresponding image modal data of news in a target task dataset; performing feature extraction on the modal data to obtain unimodal features; clustering the unimodal features to obtain semantic pseudo-labels; extracting semantic features from the unimodal features and then learning them to obtain learned semantic features; extracting true and false features from the unimodal features; obtaining semantic-based true and false features based on the semantic features and the true and false features; fusing the semantic-based true and false features of the text modal and image modalities to obtain multimodal features of the news; training a false news detection network; and dividing all news into multiple clusters based on the semantic pseudo-labels. Each cluster uses a corresponding classifier to classify the news within the cluster as true or false. The present invention has the advantages of simple implementation, low implementation cost, high detection efficiency, and high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of news detection technology, and in particular to a multimodal false news detection method and device based on semantic authenticity features. Background Art

[0002] Fake news detection involves intelligently detecting fake news through natural language processing and image processing techniques. Existing techniques for fake news detection typically extract authenticity features directly from semantic features. This involves first extracting the news' text and image features, then extracting features related to the news' authenticity directly from these features. By distinguishing these authenticity features, the authenticity of the news can be determined. However, news on different topics actually possess different semantic features, and different semantic features possess different authenticity features. This means that the semantic features of news and their corresponding authenticity features are inconsistent. Therefore, the traditional method of directly extracting authenticity features from semantic features is not only complex and difficult to implement, but also has a low accuracy rate for identifying fake news.

[0003] To address the issue of news on different topics having different semantic features, some researchers have proposed extracting similar semantic features to learn authenticity features. Specifically, for news with different semantic features, the different semantic features between the news items are removed, and only the similar semantic features of all the news items are extracted. Authenticity features are then learned from these similar semantic features to eliminate the influence of the semantic features of different news items. However, this approach loses the unique semantic features of each news item when extracting similar semantic features from all the news items, and fails to fully utilize the differences in the different semantic features. As a result, the subsequent extraction and detection of authenticity features for news items remains inaccurate. Summary of the Invention

[0004] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a multimodal fake news detection method and device based on semantic authenticity features, which has a simple implementation method, low implementation cost, high detection efficiency and accuracy.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A multimodal fake news detection method based on semantic authenticity features, comprising the following steps:

[0007] Step 1: Obtain the text modality data and corresponding image modality data of the news in the target task dataset;

[0008] Step 2: performing feature extraction on the acquired text modal data and the corresponding image modal data of the news to obtain text unimodal features and image unimodal features of the news;

[0009] Step 3: Clustering the text unimodal features and image unimodal features of all news in the target dataset to obtain text semantic pseudo labels and image semantic pseudo labels of all news;

[0010] Step 4: Semantic features of the text unimodal features and the image unimodal features are extracted respectively, and the extracted semantic features are learned based on the text semantic pseudo-labels and the image semantic pseudo-labels of the news to obtain learned text modal semantic features and image modal semantic features of the news;

[0011] Step 5: extracting the true and false features of the text unimodal features and the image unimodal features, and obtaining the text modal true and false features and the image modal true and false features of the news respectively;

[0012] Step 6: Obtaining semantic-based truth and falsehood features of the news text modality and image modality, respectively, based on the text modality semantic features and image modality semantic features of the news and the text modality truth and falsehood features and image modality truth and falsehood features of the news;

[0013] Step 7: Fusing the semantic-based true / false features in the news text modality and the semantic-based true / false features in the image modality to obtain multimodal features of the news;

[0014] Step 8: Construct a loss function for network training based on the multimodal features of news, and select training data from the target task dataset to train the fake news detection network. After training, obtain the multimodal features of all trained news.

[0015] Step 9: Divide the multimodal features of all news into multiple clusters based on semantic pseudo-labels. Each cluster uses the corresponding classifier to classify the news in the cluster as true or false.

[0016] Furthermore, in step 4, the extracted semantic features are learned by using triple learning. When using triple learning, a loss function of triple learning is constructed based on the text semantic pseudo-labels and image semantic pseudo-labels of the news, and each triple is grouped according to the distance d between the target sample and the positive and negative samples. The triple data is selected from the grouping results for network training. After training, the learned text modal semantic features and image modal semantic features of the news are obtained.

[0017] Furthermore, the step of learning the extracted semantic features using triple learning includes:

[0018] Step 401: Select a news item from the target dataset as a target sample and sample a positive sample and a negative sample to construct a triplet. The positive sample and the target sample have the same semantic pseudo-label, while the negative sample and the target sample have different semantic pseudo-labels.

[0019] Step 402. Calculate the distance d between the target sample and the positive and negative samples in each triplet based on the distance d1 between the target sample and the positive sample and the distance d2 between the target sample and the negative sample;

[0020] Step 403. Divide each triple in the target data set into a simple triple, a semi-hard triple, and a most difficult triple according to the distance d between the target sample and the positive and negative samples, wherein the distance d between the target sample and the positive and negative samples corresponding to the simple triple, the semi-hard triple, and the most difficult triple decreases in sequence;

[0021] Step 404. Use the semi-difficult triples and the most difficult triples to train a semantic feature extraction network, wherein the semi-difficult triples are first used for multiple pre-training, and then the most difficult triples are used to perform network training, and after the training is completed, the semantic features of the learned news are obtained.

[0022] Furthermore, the specific steps of step 6 include:

[0023] Step 601: Based on the semantic pseudo-labels of the news text modality and image modality, the semantic features of the news text modality and image modality are weighted using an attention mechanism to obtain weighted semantic features of the text modality and image modality;

[0024] Step 602. Based on the true and false features of the text modality and image modality of the news obtained in step 5 and the weighted semantic features of the text modality and image modality, a semantic-based true and false feature extraction network is used to obtain the semantic-based true and false features corresponding to the text modality and image modality of the news.

[0025] 9. Furthermore, in step 601, the news s is calculated according to the following formula i The weighted semantic feature H i :

[0026]

[0027] Among them, N oi Yes and News i The news content set with the same semantic pseudo-label is a parameter to measure the importance of different data in the set, h j For News j The semantic features of are calculated as follows:

[0028]

[0029] Among them, h k For News k The semantic features of Score(h i ,h j ) is used to calculate news i and s j The correlation of the semantic features of is as follows:

[0030] Score(h i , h j )=v1tanh(W1[(h i ;h j )])

[0031] Among them, v1 and W1 are the neural network parameters that need to be learned.

[0032] Furthermore, in step 5, GRU is used to calculate the semantic-based truth and falsehood features, and / or in step 8, GRU is used to fuse the semantic-based truth and falsehood features in the text modality and image modality to obtain the multimodal features of the news. The use of GRU to calculate the semantic-based truth and falsehood features includes taking the truth and falsehood features of the text modality and image modality of the news as the input data of the GRU network, and using the text modality and image modality semantic features after weighting the semantic features of the text modality and image modality of the news using the attention mechanism as the hidden layer data of the GRU network, and obtaining the semantic-based truth and falsehood features corresponding to the text modality and image modality of the news from the output data of the GRU network; the use of GRU to fuse the semantic-based truth and falsehood features in the text modality and image modality to obtain the multimodal features of the news includes: taking the semantic-based truth and falsehood features in the text modality of the news as the input data of the GRU network, and using the semantic-based truth and falsehood features in the image modality of the news as the hidden layer data of the GRU network, and obtaining the multimodal features of the news from the output data of the GRU network.

[0033] Furthermore, step 8 includes: for the multimodal features of the news, according to the text semantic pseudo-labels of the news obtained in step S3, respectively constructing a semantic-based truth loss function and a semantic-based intra-cluster distance loss function, the semantic-based truth loss function is used to distinguish true and false news under different semantics, the semantic-based intra-cluster distance loss function is used to strengthen the semantic features of news under the same semantics, and constructing a fully connected layer neural network for predicting the true and false labels of news, using cross entropy loss for network training to obtain the fake news detection network.

[0034] Furthermore, the construction of the semantic-based truth loss function includes:

[0035] Step 801: Divide all news into multiple clusters based on semantic pseudo-labels. Each cluster contains news with the same semantics, and news with different semantics are in different clusters.

[0036] Step 802: Sample triplets within each cluster and classify the sampled triplets into simple triplets, semi-hard triplets, and most difficult triplets based on the distance d between the target sample and the positive and negative samples, wherein the distance d between the target sample and the positive and negative samples corresponding to the simple triplets, semi-hard triplets, and most difficult triplets decreases in order;

[0037] Step 803: Use the semi-hard triples and the most difficult triples sampled from all clusters as training data to calculate a semantic-based true and false triple learning loss function.

[0038] Furthermore, all news are divided into multiple clusters according to semantic pseudo-labels, each cluster contains news with the same semantics, and news with different semantics are in different clusters; the distance between any two data points in each cluster is calculated and the average intra-cluster distance is calculated, and the intra-cluster average distances of all clusters are added together to obtain the semantic-based intra-cluster distance loss function.

[0039] A multimodal fake news detection device based on semantic authenticity features, comprising:

[0040] The acquisition module is used to obtain the text modality data and corresponding image modality data of the news in the target task dataset;

[0041] A unimodal feature extraction module is used to extract features from the acquired text modal data and the corresponding image modal data of the news to obtain text unimodal features and image unimodal features of the news;

[0042] A clustering module, configured to cluster the text unimodal features and image unimodal features of all news in the target dataset, respectively, to obtain text semantic pseudo labels and image semantic pseudo labels of all news;

[0043] a semantic feature extraction module for extracting semantic features of the text unimodal features and the image unimodal features, respectively, and learning the extracted semantic features based on the text semantic pseudo-labels and image semantic pseudo-labels of the news to obtain learned text modal semantic features and image modal semantic features of the news;

[0044] A true / false feature extraction module is used to extract true / false features of the text unimodal features and the image unimodal features, and obtain true / false features of the text modality and true / false features of the news respectively;

[0045] A semantic and truth / fake feature fusion module is used to obtain semantic-based truth / fake features of the news text modality and image modality respectively based on the semantic features of the news text modality and image modality as well as the truth / fake features of the news text modality and image modality;

[0046] A multimodal feature fusion module is used to fuse the semantic-based true / false features in the text modality of the news and the semantic-based true / false features in the image modality to obtain the multimodal features of the news;

[0047] The detection network training module is used to construct a loss function for network training based on the multimodal features of news, and select training data from the target task dataset to train the fake news detection network. After training, the multimodal features of all trained news are obtained;

[0048] The true / false classification module is used to divide the multimodal features of all news into multiple clusters based on semantic pseudo-labels. Each cluster uses a corresponding classifier to classify the news within the cluster as true or false.

[0049] Or the multimodal fake news detection device based on semantic authenticity features includes a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0050] Compared with the prior art, the advantages of the present invention are: based on the use of multimodal features, the present invention first clusters the unimodal features of news to cluster news with similar semantic features into a cluster, then extracts the truth and falsehood features of the news data in each cluster respectively, and then combines the truth and falsehood features with the semantic features of the news to obtain semantic-based truth and falsehood features, so that different truth and falsehood recognition can be performed based on the different semantics of the news. After the false detection network is trained in combination with the semantic-based truth and falsehood features, false news recognition is performed separately in each cluster. Not only can the different semantic features of the news be fully utilized, but also the different truth and falsehood features of news with different semantics can be used to greatly improve the extraction efficiency of the truth and falsehood features of the news, thereby significantly improving the recognition accuracy of false news. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 4 is a flowchart illustrating an implementation of a multimodal fake news detection method based on semantic authenticity features in this embodiment.

[0052] Figure 2 2 is a schematic diagram of the model training and testing process for implementing multimodal fake news detection in this embodiment.

[0053] Figure 3 Schematic diagram of the principle of using triples to learn semantic features in this embodiment. DETAILED DESCRIPTION

[0054] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.

[0055] Considering the inconsistency between the semantic features and the truth and falsehood features of news, the present invention, based on the utilization of multimodal features, first clusters the unimodal features of news to cluster news with similar semantic features into one cluster, while news with different semantic features belong to different clusters. Then, the truth and falsehood features of the news data in each cluster are extracted respectively, and the truth and falsehood features are combined with the semantic features of the news to obtain a semantic-based truth and falsehood feature. The semantic-based truth and falsehood feature can characterize the truth and falsehood features of different semantics, making it identifiable for news with different semantics, and different truth and falsehood recognition can be performed based on the different semantics of the news. Therefore, the false detection network is trained in combination with the semantic-based truth and falsehood feature, and false news recognition is then performed separately in each cluster. This not only fully utilizes the different semantic features of the news, but also greatly improves the extraction efficiency of the news truth and falsehood features based on the different truth and falsehood features of the extracted news with different semantics, and solves the problem of inconsistency between the semantic features and the truth and falsehood features of the news. Compared with the traditional method of directly extracting the news truth and falsehood features from the semantic features to realize false news detection, the false news recognition accuracy can be significantly improved.

[0056] like Figure 1 As shown, the detailed steps of the multimodal fake news detection method based on semantic authenticity features in this embodiment include:

[0057] Step 1: Obtain the text modality data and corresponding image modality data of the news in the target task dataset.

[0058] This embodiment, specifically tailored to the characteristics of the fake news detection task, acquires text modal data and corresponding image modal data for news items in the target task dataset. A target news sample typically contains only one piece of text data, but corresponds to multiple images. This embodiment randomly selects one image from the target news sample as the image modal data for that target news sample, corresponding to one piece of text data. For all news items in the target task dataset, the corresponding piece of text modal data and one piece of image modal data are determined.

[0059] Step 2: Perform feature extraction on the acquired news text modal data and the corresponding image modal data to obtain the news text unimodal features and image unimodal features.

[0060] This embodiment specifically uses the natural language preprocessing model and the image preprocessing model to extract the text unimodal features and image unimodal features of the news in the target task dataset, wherein the news text data {T1, T2, ..., T n} Input natural language preprocessing model g T , get the preliminary text unimodal features of the news {s1,s2,…,s n}; The news picture data {I1, I2, ..., I n} Input image preprocessing model g I , get the preliminary image unimodal features of news {v1,v2,…,v n The above natural language preprocessing model and image preprocessing model can be selected according to actual needs.

[0061] Step 3: Cluster the text unimodal features and image unimodal features of all news in the target dataset separately to obtain the text semantic pseudo labels and image semantic pseudo labels of all news.

[0062] This embodiment specifically utilizes an unsupervised clustering algorithm to cluster the text unimodal features and image unimodal features of all news in the target dataset, so that news with similar semantic features are clustered into one cluster, while news with different semantic features belong to different clusters. Specifically, the unsupervised clustering algorithm can be a KMeans algorithm, etc. The specific algorithm can be selected based on actual needs.

[0063] Specifically, the news text unimodal features {s1, s2, ..., s n} Input unsupervised clustering algorithm to obtain the text semantic pseudo label of news {O T 1,O T 2,…,O T n}, where O T i The value range is 1-K T , K T Represents the number of categories for text clustering. i The text modal data set of news with the same pseudo label is denoted as N T oi , where oi represents s i Pseudo labels. The news image unimodal features {v1,v2,…,v n} Input unsupervised clustering algorithm to obtain the semantic pseudo labels of news images {O I 1,O I 2,…,O I n} where O I iThe value range is 1-K I , K I Represents the number of categories for image clustering. i The image modality data set of news with the same pseudo label is denoted as N I oi .

[0064] Step 4: Extract the semantic features of text unimodal features and image unimodal features respectively, and learn the extracted semantic features based on the text semantic pseudo-labels and image semantic pseudo-labels of the news to obtain the learned text modal semantic features and image modal semantic features of the news.

[0065] In this embodiment, the semantic feature extraction network f1 of the text modality and the image modality is constructed respectively. T and f1 I , the text unimodal features {s1,s2,…,s n} and image unimodal features {v1,v2,…,v n} Input their respective semantic feature extraction networks to obtain the news text modal semantic features {h T 1,h T 2,…,h T n} and the image modal semantic features of news {h I 1,h I 2,…,h I n In a specific application embodiment, the BERT pre-training model can be used to extract the semantic features of the text modality in multimodal news, and the VGG-19 pre-training model can be used to extract the semantic features of the image modality of multimodal news.

[0066] This embodiment adopts a triple learning method to learn semantic features. Specifically, a triple learning loss function is constructed based on the text semantic pseudo-labels and image semantic pseudo-labels of the news obtained in step 3, and each triple is grouped according to the distance d between the target sample and the positive and negative samples. Triple data is selected from the grouping results for network training, and the network is trained by sampling different triple data. After training, the learned text modal semantic features and image modal semantic features of the news are obtained.

[0067] In triplet learning, data representation is learned through the distance relationship between the target sample and the positive sample and the negative sample in the triplet. Specifically, the distance d1 between the target sample and the positive sample is calculated, and the distance d2 between the target sample and the negative sample is calculated at the same time. The data representation is learned by controlling d2-d1> the preset threshold margin, so that the distance between the positive sample and the target sample in all triplets is less than the distance margin between the negative sample and the target sample.

[0068] In this embodiment, the specific steps of using triple learning to learn the extracted semantic features include:

[0069] Step 401: Select a news item from the target dataset as a target sample and sample a positive sample and a negative sample to construct a triplet. The positive sample and the target sample have the same semantic pseudo-label, while the negative sample and the target sample have different semantic pseudo-labels.

[0070] Step 402. Calculate the distance d between the target sample and the positive and negative samples in each triplet based on the distance d1 between the target sample and the positive sample and the distance d2 between the target sample and the negative sample, i.e., d = d2 - d1;

[0071] Step 403. Divide each triple in the target data set into simple triples, semi-hard triples, and most difficult triples according to the distance d between the target sample and the positive and negative samples, wherein the distance d between the target sample and the positive and negative samples corresponding to the simple triples, semi-hard triples, and most difficult triples decreases in sequence;

[0072] Step 404. Use the semi-difficult triples and the most difficult triples to train the semantic feature extraction network, wherein the semi-difficult triples are first used for multiple pre-training, and then the most difficult triples are used to perform the main network training. After the training is completed, the semantic features of the learned news are obtained.

[0073] In the above step S403, if Figure 3 As shown, the specific condition d(h I a,h I n)-d(h I a,h I p)>The triplet of the preset distance difference margin is defined as a simple triplet, which satisfies 0 <d(h I a,h I n)-d(h I a,h I The triplet with p)<the preset distance difference margin is defined as a semi-hard triplet, and the triplet that satisfies d(h I a,h I n)-d(h I a,h IThe triplet with p) < 0 is defined as the most difficult triplet.

[0074] Take the image modality processing as an example, Figure 3 As shown, first select a news item from the target dataset as the target sample, denoted as h I a, then sample a positive sample from the target dataset, denoted as h I p, the positive sample and the target sample have the same image semantic pseudo label, that is, Oa = Op; at the same time, a negative sample is sampled from the target dataset, denoted as h I n, the negative sample and the target sample have different image semantic pseudo labels, that is, Oa is not equal to On. The above three data are combined into a triplet in the form of: (target sample, positive sample, negative sample).

[0075] Then, the distance between the target sample and the positive and negative samples in each triple is calculated. In this embodiment, the Euclidean distance is used to measure the distance between samples. The formula is as follows:

[0076] d(h I a,h I p)=||h I ah I p||2 2 (1)

[0077] d(h I a,h I n)=||h I ah I n||2 2 (2)

[0078] Then sample all the triplets from the target dataset, calculate the distance between the target sample and the positive and negative samples, and then divide the triplets in the target dataset into simple triplets, semi-hard triplets and the most difficult triplets according to the distance d between the target sample and the positive and negative samples. Remove the simple triplets from all the sampled triplets, and retain the semi-hard triplets and the most difficult triplets.

[0079] Construct the triplet loss function of the semantic feature extraction network, and then use the retained semi-hard triples and the most difficult triples to train the network, and finally obtain the trained news image modality semantic features {h I 1,h I 2,…,h I n}.

[0080] For the text modality of news, the same processing principle as above is adopted, that is, the trained news text modality semantic features {h I 1,h I 2,…,h In}.

[0081] Step 5: Extract the true and false features of the text unimodal features and the image unimodal features to obtain the text modal true and false features and image modal true and false features of the news respectively.

[0082] Specifically, construct the news text mode and image mode true and false feature extraction network f2 respectively T and f2 I , the text unimodal features {s1,s2,…,s n} and image unimodal features {v1,v2,…,v n} Input the respective true and false feature extraction networks, and obtain the true and false features of the news text modality {r T 1,r T 2,…,r T n}, and the news image modality true and false features {r I 1,r I 2,…,r I n}.

[0083] Step 6: According to the news text modality semantic features, image modality semantic features, text modality truth and falsehood features, image modality truth and falsehood features of the news, semantic-based truth and falsehood features of the news text modality and image modality are obtained respectively.

[0084] Step 601. Based on the semantic pseudo-labels of the text modality and image modality of the news, the semantic features of the text modality and image modality of the news are weighted using the attention mechanism to obtain weighted text modality and image modality semantic features.

[0085] The attention mechanism can be used to calculate the correlation between the input query data and keyword data, assign different levels of importance to the input data, and obtain output data under different attention. Taking the text mode as an example, this embodiment specifically calculates the news s according to the following formula (3): i The weighted semantic features of the text model H T i :

[0086]

[0087] Among them, N T oi Yes and News i A collection of news text content with the same semantic pseudo-label, a ij A parameter that measures the importance of different data in a set. For News jThe text modal semantic features are calculated as follows:

[0088]

[0089] Among them, h T k For News k The text modal semantic features of Used to calculate news i and s j The correlation of the text modal semantic features is as follows:

[0090]

[0091] Among them, v T 1 and W T 1 are the neural network parameters that need to be learned in the attention network used in the text modality. The image modality can obtain the news s by following the same principle as above formulas (3) to (5). i The weighted semantic features of the text model H I i 、

[0092] Step 602. Based on the true and false features of the text modality and image modality of the news obtained in step 5 and the weighted semantic features of the text modality and image modality, a semantic-based true and false feature extraction network is used to obtain the semantic-based true and false features corresponding to the text modality and image modality of the news.

[0093] This embodiment specifically uses GRU to calculate the semantic-based true-false features, that is, the true-false features of the text modality and image modality of the news are used as the input data of the GRU network, and the semantic features of the text modality and image modality of the news are weighted using the attention mechanism as the hidden layer data of the GRU (Gated Recurrent Neural Unit) network. The semantic-based true-false features corresponding to the text modality and image modality of the news are obtained from the output data of the GRU network. By using an update gate and a reset gate of the GRU to control the contribution of the input data and hidden layer data to the output data, it is possible to selectively extract the features of the input data and hidden layer data, thereby utilizing the characteristics of the GRU to efficiently realize the generation of semantic-based true-false features.

[0094] Taking the text modality as an example, the detailed process of calculating the semantic-based true and false features is as follows:

[0095] First, based on the semantic pseudo-labels of the news text modality {O T 1,O T 2,…,O T n}, and the obtained news text modal semantic features {h T 1,h T 2,…,h T n}, use the attention mechanism to obtain the weighted text modal semantic features {H T 1,H T 2,…,H T n};

[0096] Then based on the obtained news text modality true and false features {r T 1,r T 2,…,r T n}, construct a semantic-based true-false feature extraction network for text modality, combine the obtained weighted semantic features, and obtain the semantic-based true-false feature {c T 1,c T 2,…,c T n}. This step is implemented using the GRU network, that is, {r T 1,r T 2,…,r T n} as the input data of GRU, {H T 1,H T 2,…,H T n} as the hidden layer data of GRU, and then get the output data of GRU {c T 1,c T 2,…,c T n} is the semantic-based truth and falsehood feature of the news text modality.

[0097] According to the same principle as above, the image modality data of news is processed, and based on the obtained image modality semantic features of news {h I 1,h I 2,…,h I n}, and the obtained image modality true and false features {r I 1,r I 2,…,r I n} and semantic pseudo-labels of image modality {O I 1,O I 2,…,O I n}, construct a semantic-based true-false feature extraction network for image modality, and input the above two into the GRU network, that is, the semantic-based true-false feature {cI 1,c I 2,…,c I n}.

[0098] Step 7: Fuse the semantic-based true / false features in the text modality of the news and the semantic-based true / false features in the image modality to obtain the multimodal features of the news.

[0099] Specifically, the semantic-based true and false features of the news text modality obtained by fusion {c T 1,c T 2,…,c T n} and semantic-based authenticity features in image modality {c I 1,c I 2,…,c I n}, get the multimodal features of news {m1,m2,…,m n This step can be implemented by GRU, that is, the semantic-based true and false features in the text modality of the news {c T 1,c T 2,…,c T n} is used as the input data of GRU, and the semantic-based true and false features in the image modality of the news {c I 1,c I 2,…,c I n} is used as the hidden layer data of GRU, and the output result is the multimodal features of the news {m1,m2,…,m n By using GRU to fuse the semantic features and authenticity features of news, the resulting news multimodal features have the foundation of news semantic features and can selectively extract the semantic features and authenticity features of news to better identify fake news.

[0100] Step 8: Construct a loss function for network training based on the multimodal features of news, and select training data from the target task dataset to train the fake news detection network. After the training is completed, the multimodal features of all trained news are obtained.

[0101] Specifically, by constructing a loss function to implement the network training in steps 5-7 above, and then training the network with a portion of the target dataset, the trained network can be used to detect fake news. The loss function construction specifically includes: constructing a semantically based truth / fake loss function and a semantically based intra-cluster distance loss function based on the multimodal features of the news and the textual semantic pseudo-labels of the news obtained in step 3. The semantically based truth / fake loss function is used to distinguish true from false news with different semantics, while the semantically based intra-cluster distance loss function is used to strengthen the semantic features of news with the same semantics. Furthermore, a fully connected neural network is constructed to predict the true / fake labels of news. The network is trained using cross-entropy loss to obtain a fake news detection network, thereby obtaining the trained multimodal features {m1, m2, …, mn} of all news.

[0102] The above-mentioned semantic-based true / false loss function specifically includes:

[0103] Step 801: Divide all news into K categories according to semantic pseudo labels. T clusters, each cluster contains news with the same semantics, and news with different semantics are in different clusters;

[0104] Step 802: Sample triplets within each cluster and classify the sampled triplets into simple triplets, semi-hard triplets, and most difficult triplets based on the distance d between the target sample and the positive and negative samples, wherein the distance d between the target sample and the positive and negative samples corresponding to the simple triplets, semi-hard triplets, and most difficult triplets decreases in order;

[0105] Step 803: Use the semi-hard triples and the most difficult triples sampled from all clusters as training data to calculate a semantic-based true and false triple learning loss function.

[0106] Specifically, by dividing all news into K T Clusters are formed, each cluster contains news with the same semantics, and news with different semantics are in different clusters; the distance between any two data points in each cluster is calculated and the average distance within the cluster is calculated. For example, the average of these distances is used as the average distance within the cluster, and the average distances within all clusters are added together to obtain the semantic-based cluster distance loss function.

[0107] Step 9: Divide the multimodal features of all news into multiple clusters based on semantic pseudo-labels. Each cluster uses the corresponding classifier to classify the news in the cluster as true or false.

[0108] Considering that the fake news detection model trained directly in step 8 still cannot fully utilize semantic features, after training the fake news detection network to obtain multimodal features, this embodiment uses a semantic-based approach to predict the truth of news. Specifically, all news multimodal features {m1, m2, ..., mn} are first divided into K T Clusters are formed, each cluster contains news with the same semantics, and news with different semantics are in different clusters. Then, a corresponding classifier (such as SVM or MLP, etc.) is designed and implemented for each cluster. Classifiers (such as SVM or MLP, etc.) are used in each cluster for classification training and testing, so as to classify all news in the cluster as true or false. That is, different clusters have different classifiers, and finally semantic-based prediction of news truth or falsehood is achieved. Semantic features can be fully utilized to achieve news truth or falsehood prediction, effectively improving the accuracy of truth or falsehood prediction.

[0109] This method extracts the authenticity characteristics of news within each news cluster based on their distinct semantic features. This solves the problem of inconsistency between semantic and authenticity characteristics of news, enabling the model to learn semantically based authenticity characteristics. Furthermore, when performing falsehood detection on all news, it also uses different semantic features, resulting in higher false news detection accuracy.

[0110] The multimodal false news detection device based on semantic authenticity features of this embodiment includes:

[0111] The acquisition module is used to obtain the text modality data and corresponding image modality data of the news in the target task dataset;

[0112] A unimodal feature extraction module is used to extract features from the acquired text modal data and the corresponding image modal data of the news to obtain text unimodal features and image unimodal features of the news;

[0113] A clustering module, configured to cluster the text unimodal features and image unimodal features of all news in the target dataset, respectively, to obtain text semantic pseudo labels and image semantic pseudo labels of all news;

[0114] a semantic feature extraction module for extracting semantic features of the text unimodal features and the image unimodal features, respectively, and learning the extracted semantic features based on the text semantic pseudo-labels and image semantic pseudo-labels of the news to obtain learned text modal semantic features and image modal semantic features of the news;

[0115] The true and false feature extraction module is used to extract the true and false features of the text unimodal features and the image unimodal features, and obtain the true and false features of the text modality and the image modality of the news respectively;

[0116] A semantic and truth / fake feature fusion module is used to obtain semantic-based truth / fake features of the news text modality and image modality respectively based on the semantic features of the news text modality and image modality as well as the truth / fake features of the news text modality and image modality;

[0117] A multimodal feature fusion module is used to fuse the semantic-based true / false features in the text modality of the news and the semantic-based true / false features in the image modality to obtain the multimodal features of the news;

[0118] The detection network training module is used to construct a loss function for network training based on the multimodal features of news, and select training data from the target task dataset to train the fake news detection network. After training, the multimodal features of all trained news are obtained;

[0119] The true-false classification module is used to divide the multimodal features of all news into multiple clusters based on semantic pseudo-labels. Each cluster uses the corresponding classifier to classify the news within the cluster as true or false.

[0120] The multimodal false news detection apparatus based on semantic authenticity features in this embodiment corresponds one-to-one to the multimodal false news detection method based on semantic authenticity features described above, and will not be described in detail here.

[0121] In another embodiment, the multimodal fake news detection device based on semantic authenticity features of the present invention can also include: a processor and a memory, the memory is used to store a computer program, and it is characterized in that the processor is used to execute the computer program to perform the multimodal fake news detection method based on semantic authenticity features as described above.

[0122] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed above with reference to the preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent variations, and modifications to the above embodiment that do not depart from the technical solution of the present invention and are based on the technical essence of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A multimodal fake news detection method based on semantic authenticity features, characterized by the following steps: include: Step 1: Obtain the text modality data and corresponding image modality data of the news in the target task dataset; Step 2: performing feature extraction on the acquired text modal data and the corresponding image modal data of the news to obtain text unimodal features and image unimodal features of the news; Step 3: Clustering the text unimodal features and image unimodal features of all news in the target dataset to obtain text semantic pseudo labels and image semantic pseudo labels of all news; Step 4: Semantic features of the text unimodal features and the image unimodal features are extracted respectively, and the extracted semantic features are learned based on the text semantic pseudo-labels and the image semantic pseudo-labels of the news to obtain learned text modal semantic features and image modal semantic features of the news; Step 5: extracting the true and false features of the text unimodal features and the image unimodal features, and obtaining the text modal true and false features and the image modal true and false features of the news respectively; Step 6: Obtaining semantic-based truth and falsehood features of the news text modality and image modality, respectively, based on the text modality semantic features and image modality semantic features of the news and the text modality truth and falsehood features and image modality truth and falsehood features of the news; Step 7: Fusing the semantic-based true / false features in the news text modality and the semantic-based true / false features in the image modality to obtain multimodal features of the news; Step 8: Construct a loss function for network training based on the multimodal features of news, and select training data from the target task dataset to train the fake news detection network. After training, obtain the multimodal features of all trained news. Step 9: Divide the multimodal features of all news into multiple clusters based on semantic pseudo-labels. Each cluster uses a corresponding classifier to classify the news within the cluster as true or false. The step 8 includes: constructing a semantic-based truth loss function and a semantic-based intra-cluster distance loss function for the multimodal features of the news according to the text semantic pseudo-labels of the news obtained in step S3, respectively. The semantic-based truth loss function is used to distinguish true and false news under different semantics, and the semantic-based intra-cluster distance loss function is used to strengthen the semantic features of news under the same semantics; and constructing a fully connected layer neural network for predicting the true and false labels of news, using cross entropy loss for network training to obtain the fake news detection network.

2. The multimodal fake news detection method based on semantic authenticity features according to claim 1 is characterized in that: In step 4, the extracted semantic features are learned by using triple learning. When using triple learning, a loss function of triple learning is constructed based on the text semantic pseudo-labels and image semantic pseudo-labels of the news, and each triple is grouped according to the distance d between the target sample and the positive and negative samples. Triple data is selected from the grouping results for network training. After training, the learned text modal semantic features and image modal semantic features of the news are obtained.

3. The multimodal false news detection method based on semantic authenticity features according to claim 2 is characterized in that: The step of using triple learning to learn the extracted semantic features includes: Step 401: Select a news item from the target dataset as a target sample and sample a positive sample and a negative sample to construct a triplet. The positive sample and the target sample have the same semantic pseudo-label, while the negative sample and the target sample have different semantic pseudo-labels. Step 402. Calculate the distance d between the target sample and the positive and negative samples in each triplet based on the distance d1 between the target sample and the positive sample and the distance d2 between the target sample and the negative sample; Step 403. Divide each triple in the target data set into a simple triple, a semi-hard triple, and a most difficult triple according to the distance d between the target sample and the positive and negative samples, wherein the distance d between the target sample and the positive and negative samples corresponding to the simple triple, the semi-hard triple, and the most difficult triple decreases in sequence; Step 404. Use the semi-difficult triples and the most difficult triples to train a semantic feature extraction network, wherein the semi-difficult triples are first used for multiple pre-training, and then the most difficult triples are used to perform network training, and after the training is completed, the semantic features of the learned news are obtained.

4. The multimodal fake news detection method based on semantic authenticity features according to claim 1 is characterized in that: The specific steps of step 6 include: Step 601: Based on the semantic pseudo-labels of the news text modality and image modality, the semantic features of the news text modality and image modality are weighted using an attention mechanism to obtain weighted semantic features of the text modality and image modality; Step 602. Based on the true and false features of the text modality and image modality of the news obtained in step 5 and the weighted semantic features of the text modality and image modality, a semantic-based true and false feature extraction network is used to obtain the semantic-based true and false features corresponding to the text modality and image modality of the news.

5. The multimodal false news detection method based on semantic authenticity features according to claim 4 is characterized in that: In step 601, the news is calculated according to the following formula s i The weighted semantic features H i : in, N oi Yes and News s i A collection of news content with the same semantic pseudo-labels, A parameter that measures the importance of different data in a set. h j For News s j The semantic features of are calculated as follows: in, h k For News s k Semantic features of Score( h i , h j ) for computing news s i and s j The correlation of the semantic features of is as follows: in, v 1 and W 1 are the neural network parameters that need to be learned.

6. The multimodal fake news detection method based on semantic authenticity features according to claim 1 is characterized in that: In step 6, the semantic-based true / false features are calculated using GRU, and / or in step 7, the semantic-based true / false features in the text modality and the image modality are fused using GRU to obtain the multimodal features of the news. The using GRU to calculate the semantic-based true / false features includes using the true / false features of the text modality and the image modality of the news as input data of the GRU network, using the semantic features of the text modality and the image modality of the news after weighting the semantic features using the attention mechanism as the hidden layer data of the GRU network, and obtaining the semantic-based true / false features corresponding to the text modality and the image modality of the news from the output data of the GRU network; The method of using GRU to fuse the semantic-based true or false features in the text modality and image modality to obtain the multimodal features of the news includes: using the semantic-based true or false features in the text modality of the news as the input data of the GRU network, using the semantic-based true or false features in the image modality of the news as the hidden layer data of the GRU network, and obtaining the multimodal features of the news from the output data of the GRU network.

7. The multimodal false news detection method based on semantic authenticity features according to claim 1 is characterized in that: The construction of the semantic-based true / false loss function includes: Step 801: Divide all news into multiple clusters based on semantic pseudo-labels. Each cluster contains news with the same semantics, and news with different semantics are in different clusters. Step 802: Sample triplets within each cluster and classify the sampled triplets into simple triplets, semi-hard triplets, and most difficult triplets based on the distance d between the target sample and the positive and negative samples, wherein the distance d between the target sample and the positive and negative samples corresponding to the simple triplets, semi-hard triplets, and most difficult triplets decreases in order; Step 803: Use the semi-hard triples and the most difficult triples sampled from all clusters as training data to calculate a semantic-based true and false triple learning loss function.

8. The multimodal fake news detection method based on semantic authenticity features according to claim 1 is characterized in that: By dividing all news into multiple clusters based on semantic pseudo-labels, each cluster contains news with the same semantics, and news with different semantics are in different clusters; The distance between any two data points in each cluster is calculated, and the average distance within the cluster is calculated. The average distances within all clusters are added together to obtain the semantic-based intra-cluster distance loss function.

9. A multimodal fake news detection device based on semantic authenticity features, characterized by: include: The acquisition module is used to obtain the text modality data and corresponding image modality data of the news in the target task dataset; A unimodal feature extraction module is used to extract features from the acquired text modal data and the corresponding image modal data of the news to obtain text unimodal features and image unimodal features of the news; A clustering module, configured to cluster the text unimodal features and image unimodal features of all news in the target dataset, respectively, to obtain text semantic pseudo labels and image semantic pseudo labels of all news; a semantic feature extraction module for extracting semantic features of the text unimodal features and the image unimodal features, respectively, and learning the extracted semantic features based on the text semantic pseudo-labels and image semantic pseudo-labels of the news to obtain learned text modal semantic features and image modal semantic features of the news; A true / false feature extraction module is used to extract true / false features of the text unimodal features and the image unimodal features, and obtain true / false features of the text modality and true / false features of the news respectively; A semantic and truth / fake feature fusion module is used to obtain semantic-based truth / fake features of the news text modality and image modality respectively based on the semantic features of the news text modality and image modality as well as the truth / fake features of the news text modality and image modality; A multimodal feature fusion module is used to fuse the semantic-based true / false features in the text modality of the news and the semantic-based true / false features in the image modality to obtain the multimodal features of the news; The detection network training module is used to construct a loss function for network training based on the multimodal features of news, and select training data from the target task dataset to train the fake news detection network. After training, the multimodal features of all trained news are obtained; The true / false classification module is used to divide the multimodal features of all news into multiple clusters based on semantic pseudo-labels. Each cluster uses a corresponding classifier to classify the news within the cluster as true or false. The detection network training module includes: constructing a semantic-based truth / fake loss function and a semantic-based intra-cluster distance loss function based on the multimodal features of the news and the text semantic pseudo-labels of the news obtained by the clustering module, respectively. The semantic-based truth / fake loss function is used to distinguish true and false news with different semantics, and the semantic-based intra-cluster distance loss function is used to strengthen the semantic features of news with the same semantics. A fully connected neural network is constructed to predict the true and false labels of the news, and the network is trained using cross-entropy loss to obtain the fake news detection network. Or the multimodal fake news detection device based on semantic authenticity features includes a processor and a memory, the memory is used to store a computer program, and it is characterized in that the processor is used to execute the computer program to perform the method according to any one of claims 1 to 8.

Images