False news identification method based on image-text multi-scale features and background knowledge

Abstract

The invention provides a false news identification method based on image-text multi-scale features and background knowledge, and the method carries out the full and effective extraction of news contents from the three aspects of news text features, news image features and news background knowledge features. And comparing the extracted text features with the image features through a comparison network to obtain a comparison result of the multi-scale image-text features, comparing the background knowledge with the news text to obtain a comparison result of the news and the background knowledge, and finally performing cross-modal fusion on the two groups of comparison results and the news text features. And a fusion result is sent to the authenticity identifier, so that the authenticity of the news is identified. According to the method, the accuracy and robustness of false news detection can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of online false news identification, and in particular to a false news identification method based on multi-scale features of images and texts and background knowledge. Background Art

[0002] Current fake news detection technologies can be divided into two categories: single-modal technology and multi-modal technology. Among them, single-modal technology usually only feeds the extracted text information or image information into the authenticity discriminator, ignoring the modal correlation between text and image. Existing multi-modal technology usually directly splices the encoded text and image and other effective information, and then feeds the spliced ​​multi-modal information into the authenticity discriminator to achieve the authenticity identification of news. However, these methods not only lack effective mining of multi-modal information, but also lack full utilization of rich background information. Fake news itself carries rich and multi-modal information, and there is a strong correlation between modal information. By organically combining multi-modal information and related background knowledge, the accuracy and robustness of fake news detection can be significantly improved. Summary of the invention

[0003] The present invention proposes a false news identification method based on multi-scale features of images and texts and background knowledge. The method fully and effectively extracts news content from three aspects: news text features, news image features, and news background knowledge features. The extracted text features are compared with the image features to obtain the image-text comparison results. The background knowledge is compared with the news text to obtain the comparison results of news and external knowledge. Finally, the two groups of comparison results, news text features, and image features are fused, and the fusion results are sent to the authenticity identifier, thereby realizing the authenticity identification of news. The method can significantly improve the accuracy and robustness of false news detection. The method includes six processes: news text feature extraction, news image feature extraction, external knowledge extraction, cross-modal multi-feature comparison, cross-modal multi-feature fusion, and authenticity identification.

[0004] The news text feature extraction process first uses the named entity recognition technology based on the SpaCy framework to identify important information such as people, targets, events, time and place from the news text, then uses the pre-trained text sentiment recognition model based on the SKEP framework to identify the sentiment of the news text, and then uses the text topic dictionary to classify the text topic. Finally, the text style analysis technology is used to analyze the writing style of the news text from the dimensions of part of speech, sentence length and text symbols.

[0005] The news image feature extraction process adopts a method of coarse-grained and fine-grained mixed annotation of the data set, annotates the character emotions in a fine-grained manner, and annotates the characters, targets, time and space in a coarse-grained manner, thereby realizing multi-feature extraction of the image. The features that can be extracted include characters, targets, emotions, time and space.

[0006] The news background knowledge extraction process first adopts a dictionary-based external knowledge construction method to build an external knowledge base based on Wikipedia entries, and then retrieves important background knowledge related to the news from the external knowledge base, including character knowledge, target knowledge, event knowledge, and common sense knowledge.

[0007] The cross-modal multi-feature comparison process first uses the pre-trained BERT model to encode the news text, the extracted multi-scale text features, the multi-scale image features, and the background knowledge. Then the encoded multi-scale text features and the multi-scale image features are sent to the comparison network for feature semantic comparison, and the encoded news text and background knowledge are sent to the comparison network for semantic comparison of text content and background knowledge, thereby obtaining the comparison results of the multi-scale features of the text and the image and the comparison results of the text and the background knowledge.

[0008] The cross-modal multi-feature fusion process uses tensor splicing to splice the comparison results of text and image multi-scale features, the comparison results of text and background knowledge, and the encoded text tensor, and then sends the spliced ​​fusion results to the authenticity discriminator.

[0009] The authenticity identification process constructs a news authenticity discriminator based on a multi-layer perceptron (MLP), which includes an input layer, a hidden layer, an output layer, and a Sigmoid function. The cross-modal multi-feature fusion result is then nonlinearly processed, and the Sigmoid function outputs the normalized news authenticity confidence, and the authenticity identification of the news is completed based on the authenticity confidence.

[0010] The technical problem solved by the present invention is a false news identification method based on multi-scale features of images and texts and background knowledge. An efficient and robust false news identification method is provided, which provides rich cross-modal multi-scale feature information for the authenticity identification model by fully extracting multi-scale features of images and texts and related background knowledge, thereby improving the accuracy and robustness of false news identification and achieving a breakthrough in the technology of false news identification on the Internet. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of the fake news identification method based on multi-scale features of images and texts and background knowledge of the present invention.

[0012] Figure 2 This is a comparison network flow chart of the present invention.

[0013] Figure 3 This is a flow chart of the authenticity discriminator based on the multi-layer perceptron MLP in the present invention. DETAILED DESCRIPTION

[0014] The present invention will be further described below in conjunction with the accompanying drawings.

[0015] A fake news identification method based on multi-scale features of images and text and background knowledge includes six processes: news text feature extraction, news image feature extraction, external knowledge extraction, cross-modal multi-feature comparison, cross-modal multi-feature fusion, and authenticity identification. The overall process is shown in the attached figure. Figure 1 As shown in the figure. In the multi-scale text feature extraction part, the feature extraction of people, targets, events, time and place is first converted into named entity recognition, and then the named entity recognition technology based on the SpaCy framework is used to accurately identify important information such as people, targets, events, time and place from the news text. Then, the pre-trained text emotion recognition model based on the SKEP framework is used to identify the emotion of the news text. A subject word dictionary based on text entity information is constructed, and the text theme is determined by querying the subject classification of people, targets and events in the subject dictionary. Finally, the part of speech, sentence length and text symbols of the text are multi-dimensionally counted, and the multi-dimensional statistical results are used as the writing style of the news text, thereby realizing the analysis of the text style. In the multi-scale text feature extraction part, the multi-target recognition framework based on the YOLO model is used to extract the multi-scale information of the image. It should be noted that in order to achieve accurate emotion recognition, in the process of labeling the training data set, the training data is labeled by a coarse-fine granularity mixed labeling method, and the expression of the characters is fine-grained, and the characters, targets, and time and space information are coarse-grained. In the background knowledge query process, we first built a background knowledge base dictionary based on Wikidata, and then used the dictionary key-value query method to search for background knowledge related to people, targets, and events in the background knowledge base. After extracting text features, image features, and background knowledge, we used the pre-trained BERT model to encode the text, background knowledge, and all features, and performed a convolution operation on the text encoding result to obtain the text feature tensor. Finally, the encoded external background knowledge tensor, text feature tensor, image feature tensor, and text feature tensor were sent to the comparison network.

[0016] The comparison network is shown in the attached Figure 2As shown, the comparison network has two feature input interfaces, namely feature input interface 1 and feature input interface 2. When performing multi-scale feature comparison of text and images, feature input interface 1 receives the encoded text feature tensor, and feature input interface 2 receives the encoded image feature tensor. When comparing text with external background knowledge, feature input interface 1 receives the encoded text tensor, and feature input interface 2 receives the encoded external background knowledge tensor. After receiving the two input features, the comparison network performs Hadamard product operations and point difference operations on the input features, respectively, as shown below:

[0017] f 1 (x,y)=x⊙y (1)

[0018]

[0019] Among them, x is the tensor of input interface 1, y is the tensor of input interface 2, represents the point difference operation, and ⊙ represents the Hadamard product operation. Then, the above results are concatenated together by tensor concatenation and connected with a fully connected layer. The process is shown as follows:

[0020]

[0021] Among them, cpr represents the output result of the comparison network, fc represents the fully connected operation, Represents the tensor concatenation operation. After obtaining the comparison feature results, the tensor concatenation operation is used again to concatenate the comparison results of the multi-scale features of the image and text, the comparison results of the text and background knowledge, and the text features together to obtain cross-modal multi-scale features. The concatenation process is shown as follows:

[0022]

[0023] Among them, cpr tv Represents the comparison results of multi-scale features of images and texts, cpr tk represents the comparison result between the text and the background knowledge, and t represents the text feature tensor. Then the above results are sent to the discriminator for authenticity identification.

[0024] The discriminator is as follows Figure 3As shown, it consists of an input layer, a hidden layer, an output layer, and a Sigmoid activation function. The input layer transmits the input features to the hidden layer in a fully connected manner. In the process of transmission between hidden layers, a Relu activation layer and a node drop layer are added to ensure that the discriminator has the ability to learn nonlinear features, and can learn more useful features from cross-modal multi-scale features. The addition of the node drop layer enables the model to avoid overfitting. The features learned by the hidden layer are sent to the Sigmoid function through the output layer, thereby outputting the true or false confidence of the news identification, and finally giving the authenticity of the news according to the true or false confidence. In the discriminator, news with a confidence less than 0.5 is identified as fake news, and news with a confidence greater than 0.5 is identified as true news.

Claims

1. A false news identification method based on multi-scale features of images and text and background knowledge, characterized in that: The method fully and effectively extracts news content from three aspects: news text features, news image features, and news background knowledge features, and compares the extracted text features with the image features to obtain image-text comparison results, and compares the background knowledge with the news text to obtain the comparison results of news and external knowledge. Finally, the two groups of comparison results, news text features, and image features are fused, and the fusion results are sent to the authenticity discriminator to achieve authenticity identification of news. The method can achieve high accuracy and high robustness of false news detection results, and specifically includes the following steps: Step 1: Extract multi-scale text features from news texts through named entity recognition, topic dictionary construction and query, text writing style statistics, and text sentiment recognition; Step 2: Use the coarse-grained and fine-grained annotated image training dataset to train the multi-target recognition framework, and use the trained multi-target recognition model to extract multi-scale image features from news images; Step 3: Use Wikipedia entry data to build an external background knowledge dictionary based on Wikipedia data, and obtain background knowledge related to news content through dictionary key-value query; Step 4: Construct a multi-scale feature comparison network, and use the comparison network to compare the encoded background knowledge with the text content, and compare the image multi-scale features with the text multi-scale features, so as to obtain the comparison results between the text and the background knowledge and the comparison results between the image and the text multi-scale features; Step 5: Use multi-feature tensor splicing to splice text features, the comparison results of text and background knowledge, and the comparison results of image and text multi-scale features to obtain cross-modal multi-scale fusion features; Step 6: Construct an MLP-based authenticity discriminator and send the multi-scale fusion features into the discriminator for authenticity identification.

2. According to claim 1, a method for identifying fake news based on multi-scale features of images and text and background knowledge, characterized in that: The text multi-scale features extracted in step 1 include characters, targets, events, time, place, emotions, and text style, and the image multi-scale features include characters, targets, time, place, and character emotions.

3. The method for identifying fake news based on multi-scale features of images and text and background knowledge according to claim 1, characterized in that: The dataset annotation method in step 2 is a mixed annotation of coarse and fine granularity, and the character emotions are fine-grained annotations.

4. The method for identifying fake news based on multi-scale features of images and text and background knowledge according to claim 1, characterized in that: The background knowledge base in step 3 is constructed using a dictionary-based key-value construction method to ensure the efficiency of background knowledge query.

5. The method for identifying fake news based on multi-scale features of images and text and background knowledge according to claim 1, characterized in that: The comparison network structure in step 4 includes parallel Hadamard product operations and point difference operations, and the two operation results are concatenated and sent to the fully connected layer.

Images