A False News Detection Method Based on Adaptive Fusion of Multimodal Features

Abstract

The present invention provides a false news detection method based on multi-modal feature adaptive fusion, including: preprocessing the text information and image information in the news to be detected; extracting different discriminant features from the preprocessed text and image data, including extracting text sentiment features from the preprocessed news text by using senta-bilstm, extracting text semantic features from the news text by using the BERT pre-trained model and the masked attention network, and adaptively fusing the above-mentioned feature representations by adding learnable weight parameters to obtain the final discriminant feature vector; inputting it into a fully connected layer with softmax for prediction. The present invention can effectively introduce text sentiment information, accurately and comprehensively capture the semantic connection between text and image information, and reduce the noise introduced by vector splicing, thereby improving the false news detection effect.

Description

Technical Field

[0001] The present invention belongs to the fields of deep learning and information processing, and in particular relates to a method for detecting fake news based on adaptive fusion of multi-modal features. Background Art

[0002] With the rapid development of social media such as Sina Weibo and Twitter, it provides opportunities for the spread of fake news in a short time, and it also provides convenience for manipulators of public events to fabricate and distort facts. This makes fake news increasingly difficult to control. Different from the linear and one-way communication mode of traditional media, the communication characteristics and methods of social media make the spread of news more rapid, and the adverse effects brought by it are wider and the social harm is greater. In order to effectively eliminate the bad social impact caused by fake news, it is imperative to realize the automatic detection of fake news on social media. With the development of multimedia technology, the form of news has gradually changed from simple text content to multimedia content with pictures or videos, thus attracting more readers. Therefore, most of the existing research work focuses on the research of multi-modal information.

[0003] In terms of multi-modal fake news detection, early research mainly enhanced the extraction of features and the fusion of multi-modal information through simple splicing. Some other research used auxiliary tasks, such as feature reconstruction and event discrimination, to improve the reliability of fake news detection and the generalization ability of the model. Although people have made great efforts in multi-modal fake news detection, and previous methods have also achieved good performance in the fake news detection task, there are still the following problems:

[0004] (1) Deep emotional features of text are not fully utilized. For the processing of news text information, when extracting text semantic features, most methods only consider the semantic features of the text, and often ignore the deep hidden features of the text, such as emotional features, etc., which is very helpful for identifying highly inflammatory fake news.

[0005] (2) Image semantic features cannot be accurately obtained. For the processing of news image information, when extracting image features, since most of the fake news pictures in social media are in the form of coexistence of pictures and texts, that is, there are a large number of chat screenshots, video screenshots or news report screenshots, etc., and most of the image feature extraction models currently used are trained on standard natural images on a large scale. The deep semantic features extracted in this way are easily interfered by the text in the image, resulting in difficult to accurately express the image and unable to achieve the ideal effect.

[0006] (3) When fusing multi-modal features, only by means of vector splicing, blindly fusing image and text features often introduces noise, resulting in difficult improvement in the detection effect of multi-modal fake news. Summary of the Invention

[0007] In view of this, the present invention aims to propose a false news detection method based on multi-modal feature adaptive fusion, so as to effectively introduce text sentiment information, comprehensively and accurately capture the semantic connection between text and image information, and reduce the noise introduced by vector splicing, thereby improving the false news detection effect.

[0008] To achieve the above object, the technical solution of the present invention is realized as follows: A false news detection method based on multi-modal feature adaptive fusion, comprising the following steps:

[0009] S1: Preprocess the news data to be detected to obtain effective news text and images;

[0010] S2: Based on the extracted text and images, obtain multi-modal feature representations including sentiment, text, images, etc.;

[0011] S3: Based on the obtained multi-modal features, perform adaptive weighted feature splicing by assigning different learnable parameters to obtain the multi-modal feature representation of the final news;

[0012] S4: Based on the obtained multi-modal feature representation of the news, perform news true / false prediction through a classifier.

[0013] Further, in S1, the preprocessing of the news data to be detected to obtain effective news text and images includes:

[0014] Preprocess the text in the news, and obtain a text sequence through steps such as text standardization, text cleaning, traditional Chinese conversion, and spelling correction;

[0015] Preprocess the pictures in the news, and retain clear and processable pictures and unify the picture sizes through steps such as image deduplication and filtering invalid format images to obtain the final news images.

[0016] Further, in S2, the obtaining of multi-modal feature representations including sentiment, text, images, etc. based on the extracted text and images includes:

[0017] Use the senta-bilstm model to calculate the sentiment polarity of the news text, obtain the sentiment polarity score corresponding to the text, and construct sentiment features to finally obtain the sentiment polarity feature representation;

[0018] Use the BERT pre-trained model and masked attention network to extract text features from the news text to obtain the text semantic feature representation of the news;

[0019] The VIT-GPT2 model and the CnOCR tool are used to extract and splice the image description information and the text information in the image respectively, and are sent to the feature metric network together with the semantic features of the news text for calculation, and finally the semantic-level difference metric features between the text and the image are obtained.

[0020] Further, in S2, the feature metric network uses a combination operation of element-wise subtraction and multiplication to preliminarily match the features of the two texts. At the same time, in order to reduce the number of parameters, improve the model calculation efficiency and generalization ability, the present invention uses conv1D to replace the fully connected layer to transform the above-mentioned matching features. These operations together constitute the feature vector comparison function, and finally the result is sent to the max pooling layer to obtain the graphic-text semantic difference metric feature representation.

[0021] Further, in S3, based on the obtained multi-modal features, by assigning different learnable parameters, adaptive weighted feature splicing is performed to obtain the multi-modal feature representation of the final news, including:

[0022] Add different learnable parameters before the obtained news sentiment polarity feature representation, news text feature representation, and news graphic-text difference metric feature representation respectively;

[0023] Splice the different feature representations with added learnable parameters to obtain the final news multi-modal feature representation.

[0024] Further, in S4, based on the obtained news multi-modal feature representation, through a text classifier, news true or false prediction is performed, including:

[0025] Based on the obtained news multi-modal feature representation, it is sent to a fully connected layer classifier with softmax for news true or false prediction.

[0026] Compared with the prior art, the false news detection method based on multi-modal feature adaptive fusion of the present invention has the following advantages:

[0027] 1) By introducing the deep sentiment polarity features of news texts, the news text feature information is enriched, and the accuracy of false news detection is improved;

[0028] 2) When extracting news image features, the text description information of the image and the text information in the image are fully considered, and by performing differential calculation with the text semantic features of the news, more reasonable graphic-text difference comparison features are obtained, avoiding the problems of large noise and large result deviation in simple image feature extraction;

[0029] 3) Introduce learnable parameters during multi-modal feature fusion, perform adaptive weighted splicing on different feature representations, effectively reduce the noise problem introduced by simply splicing text and image features, and improve the effect of fake news detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0031] Figure 1 It is a schematic flowchart of a fake news detection model based on multi-modal feature adaptive fusion according to an embodiment of the present invention;

[0032] Figure 2 It is a schematic structural diagram of a fake news detection model based on multi-modal feature adaptive fusion according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0034] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more.

[0035] In the description of the present invention, it should be noted that, unless otherwise clearly defined and limited, the terms "mounted", "connected", "coupled" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood through specific situations.

[0036] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0037] As Figure 1 and Figure 2 shown, the model structure of a fake news detection method based on multi-modal feature adaptive fusion of the present invention includes four parts: the first part realizes preprocessing of the news data to be detected, the second part is to obtain multi-modal feature representations including sentiment, text, images, etc. The third part is to fuse various features by means of adaptive weighted splicing. The fourth part is to send the obtained news multi-modal feature representations into a classifier for news true or false prediction. Specifically, it includes the following steps:

[0038] S1: Preprocess the news data to be detected to obtain effective news text and images.

[0039] First, preprocess the text in the news, mainly filter the redundant information in the text. After text standardization, text cleaning, Chinese and English word segmentation, word segmentation cleaning, word standardization, and spelling correction, the final word sequence S is obtained, and its formula is expressed as follows:

[0040] S = {word1, word2,..., word n}

[0041] where word i represents the i-th word in the word sequence.

[0042] Then, preprocess the pictures in the news. Since there are a small number of long screenshot pictures in the dataset, while removing these data, retain the pictures in JPG or JPEG format, and finally retain clear and processable pictures, and unify the picture size to 512×512 to obtain the final news image I.

[0043] S2: Based on the extracted text and images, obtain multi-modal feature representations including sentiment, text, images, etc.

[0044] 1) Use the senta-bilstm model to calculate the sentiment polarity of the news text and extract the sentiment polarity features of the text. Specifically, first fine-tune the model on the dataset of the fake news classification task, that is, convert the original text into the input format of the model through the Reader function provided by the model, and then set parameters such as the number of traversal epochs epoch = 1 and the batch size batch_size = 32 for Finetune, and periodically evaluate the model effect through the built-in evaluation function. After the fine-tuning is completed, use the model to predict the sentiment classification of the news text, and the result returns the prediction probabilities of the positive and negative emotions of the text. Finally, by constructing the text sentiment features, a 2×1-dimensional text sentiment polarity feature representation is obtained, and its formula is expressed as follows:

[0045] e s = senta_bilstm(S)

[0046] where e s is the sentiment polarity feature of the news to be measured.

[0047] 2) Use the BERT pre-trained model and masked attention network to extract text features from the news text to obtain the text semantic feature representation of the news.

[0048] Specifically, first encode the input text sentence S = {word1, word2,..., word n}, where word i represents the i-th word in the tokenized sentence. After adding the [CLS] and [SEP] tokens, obtain the token list T = [[CLS], token1,..., token n , [SEP]] where token i is the token of the i-th word, and its formula is as follows:

[0049] T = BertTokenizer(S)

[0050] where T is the token list of words, S is the word sequence after text preprocessing, and BertTokenizer(·) is the encoding function of BERT.

[0051] Then send the token list into the BERT model to obtain the word embedding vector W = [w cls , w1,..., w n , w sep , where wi is the embedding representation of the i-th word, and its formula is as follows:

[0052] W = BERT(T)

[0053] where W is the word embedding vector, T is the token list of the word sequence, and BERT(·) is the core calculation function of the BERT model.

[0054] Finally, send the embedding vectors of all words into the masked attention network. By calculating the weight of each word, its vector representation is weighted to highlight the influence of more important words on the sentence. Obtain the semantic feature representation of the news text sentence, and its formula is as follows:

[0055] e t = Mask_Attention(W)

[0056] where, e tThe semantic feature representation of the text sentence is a vector with a dimension of 768×1. W is the word embedding vector of the word, and Mask_Attention(·) is the core calculation function of the masked attention network.

[0057] 3) The VIT-GPT2 model and the CnOCR tool are used to extract and splice the image description information and the text information in the image respectively. The formula is as follows:

[0058] D i2t = image_to_text(I)

[0059] D ocr = Cn_OCR(I)

[0060] D = D i2t + D ocr

[0061] Among them, D i2t is the text of the extracted image description, D ocr is the text in the image, D is the spliced text of the image description text and the text in the image, image_to_text(·) is the core calculation function of the VIT-GPT2 model, and Cn_OCR(·) is the core calculation function of CnOCR.

[0062] Subsequently, the spliced text is input into the BERT model to obtain the text description feature representation of the image. The formula is as follows:

[0063] T d = BertTokenizer(D)

[0064] W a = BERT(T d )

[0065] e d = Mask_Attention(W d )

[0066] Among them, D is the spliced text of the image description text and the text in the image, T d is the token list of the words in the spliced text, W d [[ID=�5]]is the word embedding vector of the word, e d represents the semantic feature representation of the spliced text sentence. BertTokenizer(·) is the encoding function of the BERT model, BERT(·) is the core calculation function of the BERT model, and Mask_Attention(·) is the core calculation function of the masked attention network.

[0067] Finally, the semantic feature vectors of the obtained spliced text and the news text semantic feature vectors are fed into the feature metric network for calculation, and finally the semantic difference metric feature between the text and the image is obtained. Its formula is as follows:

[0068] f cmp (e d ,e t )=conv1D[e d -e t ,e d ⊙e t

[0069] e c =MaxPooling(f cmp (e d ,e t ))

[0070] Among them, e d represents the semantic feature vector of the spliced text sentence, e t represents the semantic feature vector of the text sentence, e c is the difference comparison feature vector between the semantic feature vector of the spliced text and the news text semantic feature vector, conv1D[·] is a one-dimensional convolution operation, ⊙ is the Hadamard product, f cmp (·) is the feature metric network vector comparison function, and MaxPooling(·) is the max pooling calculation.

[0071] S3: Based on the obtained multi-modal features, through assigning different learnable parameters, perform adaptive weighted feature splicing to obtain the multi-modal feature representation of the final news.

[0072] Specifically, different learnable parameters α, β, and γ are added before the obtained sentiment polarity feature of the news text, the news text semantic feature, and the news text-image difference comparison feature respectively. Then, the different feature representations added with learnable parameters are spliced to obtain the final multi-modal feature representation e m of the news. Its formula is as follows:

[0073] e m =αe s +βe t +γe c

[0074] Among them, e s is the sentiment polarity feature vector of the news text, e t is the news text semantic feature vector, e c is the news text-image semantic difference comparison feature vector, and e m is the multi-modal feature representation of the news.

[0075] ​S4: Based on the obtained multi-modal feature representations of news, use a classifier to predict the authenticity of the news.

[0076] To use the multi-modal feature e of the news m for fake news detection, use a fully connected layer with a Softmax activation function to output e m representing the probabilities of real news and fake news, specifically as follows:

[0077]

[0078] where is the predicted probability, W′ is the parameter of the fully connected layer, and b is the bias term.

[0079] According to the pre-set threshold thr (usually set to 0.5), obtain the final prediction result:

[0080]

[0081] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A false news detection method based on adaptive fusion of multi-modal features, characterized in that It includes the following steps: S1: Preprocess the news data to be detected to obtain valid news texts and images; S2: Based on the extracted texts and images, extract sentiment and text features, and at the same time construct text-image semantic difference measurement features through a feature measurement network; S3: Based on the obtained multi-modal features, perform adaptive weighted feature splicing by assigning different learnable parameters to obtain the multi-modal feature representation of the final news; S4: Based on the obtained multi-modal feature representation of the news, perform news authenticity prediction through a classifier; Use a combination of element-wise subtraction and multiplication operations to preliminarily match the features of the two texts. At the same time, to reduce the number of parameters, use conv1D instead of a fully connected layer to transform the matched features. These operations together form a feature vector comparison function, and finally send the result into a max pooling layer to obtain the text-image semantic difference measurement feature representation; Based on the obtained multi-modal features, perform adaptive weighted feature splicing by assigning different learnable parameters to obtain the multi-modal feature representation of the final news, including: adding different learnable parameters before the obtained news sentiment polarity feature representation, news text feature representation, and news text-image difference measurement feature representation respectively; splicing the different feature representations with added learnable parameters to obtain the final multi-modal feature representation of the news.

2. The false news detection method based on multi-modal feature adaptive fusion according to claim 1, characterized in that: Preprocess the news data to be detected to obtain valid news texts and images, including: Preprocess the text in the news. After steps of text normalization, text cleaning, traditional Chinese conversion, and spelling correction, obtain a text sequence; Preprocess the pictures in the news. After steps of image deduplication and filtering invalid format images, retain clear and processable pictures and unify the picture sizes to obtain the final news images.

3. The method for detecting fake news based on multi-modal feature adaptive fusion according to claim 1, wherein: Based on the extracted texts and images, obtain multi-modal feature representations including sentiment, text, and image, including: Use the senta-bilstm model to calculate the sentiment polarity of the news text, obtain the sentiment polarity score corresponding to the text, and construct sentiment features, and finally obtain the sentiment polarity feature representation; Use the BERT (Bidirectional Encoder Representations from Transformer) pre-trained model and masked attention network to extract text features from the news text to obtain the text semantic feature representation of the news; Use the VIT-GPT2 model and the CnOCR tool to extract and splice the image description information and the text information in the image respectively, and send them together with the news text semantic features into the feature measurement network for calculation, and finally obtain the semantic-level difference measurement features between the text and the image.

4. The false news detection method based on multi-modal feature adaptive fusion according to claim 1, characterized in that: Based on the obtained multi-modal feature representation of the news, perform news authenticity prediction through a text classifier, including: Based on the obtained multi-modal feature representation of the news, send it into a fully connected layer classifier with softmax for news authenticity prediction.

Images