False news detection system and method based on multi-modal fusion

Abstract

The invention discloses a false news detection system and method based on multi-modal fusion, and relates to the field of natural language processing and computer vision, and the method comprises the following steps: 1, collecting news samples, constructing a news sample data set which comprises news texts and news images, and carrying out the preprocessing; 2, sentence vectors in the news text and graph vectors in the news image are extracted, and the sentence vectors and the graph vectors are initial feature vectors representing the news text and the news image; step 3, using the initial feature vector for training learning of the feature updating and fusion module to obtain features capable of reflecting weights between modals, and then obtaining final feature representation through splicing; and step 4, inputting the final feature representation into a full-connection network for calculation, establishing a relation between the final feature representation and the news label, and further judging whether the news is true or false.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and computer vision, and in particular to a fake news detection system and method based on multimodal fusion. Background Art

[0002] In recent years, the number of Internet users has shown a rapid growth trend, and the Internet penetration rate has also been steadily increasing. With the continuous development and popularization of Internet technology, people's ways of obtaining information have become more diverse, and the role of the Internet in information dissemination has become increasingly important. Although the Internet has brought great convenience to information acquisition, the problems it brings cannot be ignored. In the era of information explosion, the influx of a large amount of false information has posed a serious threat to public cognition.

[0003] As academia and industry pay more attention to the research of online false information, the automatic detection technology of false information has gradually improved. Traditional methods rely on expert knowledge to extract manual features from text content, and then use traditional machine learning algorithms to train false information classifiers, such as Naïve Bayes (NB), Decision Tree (DT) and Support Vector Machine (SVM). Although these methods based on hand-crafted features are simple, they lack comprehensiveness and flexibility. A large number of studies have shown that designing artificial features is crucial for various natural language processing tasks. In the era of big data, learning highly distinguishable feature representations has become a new bottleneck for artificial intelligence applications. In the past decade, deep neural networks, because they contain multiple layers of nonlinear transformations, can automatically learn more accurate and effective features from raw data, and perform efficient optimization through layer-by-layer gradient descent.

[0004] With the development of Internet-related technologies, the form of false information has changed greatly and is no longer limited to a single text form. When users generate and spread false information, they often accompany it with multiple forms of information such as pictures and videos, which makes it more difficult to distinguish the authenticity of the information. Current technology and algorithms cannot effectively mine information that truly reflects the content of fake news, and it is difficult to effectively combine text and image information to distinguish true from false news.

[0005] Therefore, technicians in this field are committed to developing a new fake news detection system and method to solve the above-mentioned defects in the prior art. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to dig out information that can truly reflect the content of fake news, and effectively combine the information of text and image to distinguish the true from the false of news.

[0007] To achieve the above-mentioned purpose, the present invention provides a fake news detection system based on multimodal fusion, including a data reading and preprocessing module, a text feature extraction module, an image feature extraction module, a feature update and fusion module, and a detection module; The data reading and preprocessing module collects news samples, performs preprocessing, constructs a news sample data set, and divides the news sample data set into a training set, a validation set, and a test set according to a preset ratio; the news samples include news text and news images; The text feature extraction module is connected to the data reading and preprocessing module, directly inputs the news text into the pre-trained BERT model, and uses the obtained sentence vector as the feature representation of the news text; The image feature extraction module is connected to the data reading and preprocessing module, directly inputs the news image into the pre-trained VGG-19 model, and uses the obtained image vector as the feature representation of the news image; The feature update and fusion module is connected to the text feature extraction module and the image feature extraction module, and includes a plurality of weight matrices for projecting the sentence vector and the image vector into a query vector, a key vector and a value vector, calculating the attention weight and updating the sentence vector and the image vector, and then fusing the sentence vector and the image vector through vector concatenation to obtain a final feature representation that can highlight the common features of the text and the image; The detection module is connected to the feature updating and fusion module, and includes a plurality of fully connected layers. The parameters in the feature updating and fusion module are updated through training, and the connection between the final feature representation and the news label is established, so as to judge the truth or falsity of the news.

[0008] Furthermore, in the data reading and preprocessing module, a one-to-one matching is performed on the naming and storage location of the news text and the news image, and the news text contains a label of whether the news is true or false.

[0009] Furthermore, in the data reading and preprocessing module, invalid pictures existing in the news image are regarded as abnormalities, and blank images are generated; for each image in the news image, it is preprocessed into an RGB color image with three channels, and the size of each image is ensured to be consistent.

[0010] The present invention also provides a fake news detection method based on multimodal fusion, comprising the following steps: Step 1: Collect news samples, build a news sample dataset, including news text and news images, and perform preprocessing; Step 2: extracting sentence vectors from the news text and image vectors from the news image, wherein the sentence vectors and image vectors are initial feature vectors representing the news text and the news image; Step 3: Using the initial feature vector for training and learning the feature update and fusion module to obtain features that can reflect the weights between modalities, and then obtaining the final feature representation by splicing; Step 4: Input the final feature representation into a fully connected network for calculation, establish a connection between the final feature representation and the news label, and then judge the truth or falsity of the news.

[0011] Furthermore, the step 1 includes the following sub-steps: Step 1.1, collecting the news samples, constructing the news sample data set, including the news text and the news image, and dividing the news sample data set into a training set, a validation set and a test set in a ratio of 3:1:1; Step 1.2, preprocessing the news text, including word segmentation, stop word removal, and stem extraction; scaling or cropping the news image to unify the size; Step 1.3: Randomize the training set, the validation set, and the test set.

[0012] Furthermore, the step 2 includes the following sub-steps: Step 2.1, using the pre-trained BERT model to extract the sentence vector in the news text; Step 2.2, using the pre-trained VGG-19 model to extract the graph vector in the news image; Wherein, the step 2.1 comprises: Step 2.1.1, segment the news text, decompose the sentence into words or subwords, and use a vocabulary to map the words or subwords into corresponding word embedding vectors; Step 2.1.2, based on the word embedding vector, the BERT model formed by stacking multiple Transformer encoders introduces position encoding to represent the position information of the word or subword in the sentence; Step 2.1.3, extracting the sentence vector in the news text through the BERT model processing.

[0013] Furthermore, the step 2.2 includes the following sub-steps: Step 2.2.1, load weight parameters from the VGG-19 model; Step 2.2.2: Input the data of the news image into the VGG-19 model and perform forward propagation calculation; stop the forward propagation calculation before the last convolution layer of the VGG-19 model and output the image vector.

[0014] Furthermore, the step 3 includes the following sub-steps: Step 3.1: Record the sentence vector as a text feature T , and the text features T Perform feature projection to obtain the query vector , key vector Sum value vector , the projection process can be expressed as: in, , and is the corresponding projection matrix; Step 3.2: Record the image vector as image feature I , and the image features I Perform feature projection to obtain the query vector , key vector Sum value vector , the projection process can be expressed as: in, , and is the corresponding projection matrix.

[0015] Furthermore, the step 3 also includes the following sub-steps: Step 3.3: Use mutual attention mechanism to calculate the text features T and the image features I The attention weights between: in, is the dimension of the key vector, represents the attention weight of the text on the image, Represents the attention weight of the image to the text; Use the attention weights to perform weighted summation on the features to obtain updated text features and image features : Step 3.4: Use splicing to combine the updated text features and image features To perform the fusion: in, F It is the fused feature representation, that is, the final feature representation.

[0016] Furthermore, step 4 includes the following sub-steps: Step 4.1: F Input into the fully connected network including multiple hidden layers and an output layer for calculation. l There are hidden layers, and the calculation process can be expressed as: in, Indicates l The output of the hidden layer is and Respectively represent l The weights and biases of the hidden layers; Indicates l -1 hidden layer or input features F ; Step 4.2: Take the output of the last hidden layer as input and perform binary classification. The calculation process of the output layer can be expressed as: in, is the predicted output of the model, and are the weights and biases of the output layer, is the bias of the penultimate hidden layer; ultimately, the probability of the news being true or false can be obtained.

[0017] The fake news detection system and method based on multimodal fusion provided by the present invention has at least the following technical effects: 1. The technical solution provided by the present invention provides multimodal information fusion, which improves the accuracy and robustness of fake news detection by combining text and image information, and is suitable for various news detection tasks involving text and images.

[0018] 2. The technical solution provided by the present invention can achieve efficient feature updating. The strategy of transfer learning can not only speed up the training speed but also fully extract the features of the image. The mutual attention mechanism is used to make the feature updating process more efficient and can make full use of information from different modalities.

[0019] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a fake news detection module according to a preferred embodiment of the present invention; Figure 2 A schematic diagram of a news content feature extraction network structure of a preferred embodiment of the present invention; Figure 3 A schematic diagram of the structure of a modal feature module for updating a mutual attention mechanism according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0022] By fusing the features of multiple modal data to identify false information, it will become a more efficient detection method. Multimodal data comes from different perspectives or data domains of the same research object, where each data domain or perspective is called a modality. Compared with single-modal information analysis, multimodal information analysis extracts and fuses features of multiple modal data based on learning single-modal features. Multimodal information is applied to the classification and prediction of false information, mainly through the combination of text, image, voice and video modal information for identification and classification. Multimodal Fusion provides more effective information for false information classification by establishing a model that can analyze and process data in different modalities.

[0023] The current technology and algorithm cannot well mine the information that can truly reflect the content of fake news, and it is difficult to effectively combine the information of text and image to distinguish the true and false news. In order to overcome the shortcomings of the prior art, the embodiment of the present invention provides a fake news detection system and method that combines text and image features. By extracting the features of news text and image respectively, and using the cross-attention mechanism to update and fuse the features, the accuracy of fake news detection is improved. By combining the bidirectional encoder representation method BERT, the deep convolutional neural network (CNN) architecture VGG-19 and Transformer, a fake news detection network that can learn and effectively combine the features of different modalities of news is constructed, realizing a more accurate and effective fake news detection function. The embodiment of the present invention can be trained through a large number of news samples and corresponding image data, and the feature extraction module can be applied to the fake news detection task based on transfer learning. Finally, the updated features are fused and the fake news detection is performed through the classifier.

[0024] Specifically, the fake news detection method provided by the embodiment of the present invention includes the following steps: first, collect news information from popular social media (such as X, Weibo) in today's society, construct a training set, a validation set and a test set based on news with both text and image information, and construct a label reflecting the truth or falsity of the news. Then, perform news feature extraction, convert the news text into a feature vector through the pre-trained text feature extraction model BERT, and use VGG-19 to extract image features based on transfer learning after the news picture is pre-processed by cropping, rotating, etc. Through the above process, feature vectors that respectively reflect text and image information can be obtained. The mutual attention mechanism is used to calculate the attention weights between text features and image features, and the weights are used to update the features, and then the splicing method is used for fusion. Finally, a feature that can characterize the news as a whole is obtained, and the feature is input into a fully connected neural network classifier to detect fake news. After completing the training of the model, the fake news detection method provided by the embodiment of the present invention can be applied to actual fake news detection scenarios.

[0025] Example 1 like Figure 1 As shown, an embodiment of the present invention provides a fake news detection system based on multimodal fusion, including a data reading and preprocessing module, a text feature extraction module, an image feature extraction module, a feature update and fusion module, and a detection module; The data reading and preprocessing module collects news samples, performs preprocessing, constructs a news sample data set, and divides the news sample data set into a training set, a validation set, and a test set according to a preset ratio; the news samples include news text and news images; A one-to-one match is performed between the naming and storage location of news text and news images, and the news text contains a label of whether the news is true or false.

[0026] In the process of data preprocessing, there may be invalid images in the image data, that is, images that cannot be used for feature extraction during the model training process. Invalid images in news images are regarded as exceptions, and the exception handling method is used to automatically generate blank images; since the content of invalid images is extremely low, this method can ensure both the robustness of the present invention and the accuracy of fake news detection. For each image in the news image, it is preprocessed into an RGB color image with three channels, and the size of each image is guaranteed to be consistent.

[0027] In order to obtain more accurate modal feature representation, the embodiment of the present invention uses the pre-trained BERT model and VGG-19 model to represent the features of news text and images respectively. Since these two models have been pre-trained on a large number of samples and have strong feature extraction capabilities, the introduction of transfer learning strategies can effectively reduce training costs and ensure accuracy.

[0028] In particular, the text feature extraction module connects the data reading and preprocessing modules, directly inputs the news text into the pre-trained BERT model, and uses the obtained sentence vector as the feature representation of the news text; In particular, the image feature extraction module connects the data reading and preprocessing modules, directly inputs the news image into the pre-trained VGG-19 model, and uses the obtained graph vector as the feature representation of the news image.

[0029] The feature updating and fusion module connects the text feature extraction module and the image feature extraction module, and includes multiple weight matrices for projecting sentence vectors and graph vectors into query vectors (Query, Q), key vectors (Key, K) and value vectors (Value, V), calculating attention weights and updating sentence vectors and graph vectors, and then fusing the sentence vectors and graph vectors through vector splicing to obtain the final feature representation that can highlight the common features of text and image, so as to better characterize news samples; the feature updating and fusion module based on the mutual attention mechanism is the core of the present invention.

[0030] The detection module is connected to the feature update and fusion module, and includes multiple full connection layers. Its main function is to update the parameters in the feature update and fusion module through training, establish the connection between the final feature representation and the news label, and judge the authenticity of the news when applying the solution provided by the embodiment of the present invention. The final detection module is also the key.

[0031] In particular, the news samples are collected from social media platforms.

[0032] Example 2 like Figure 2 As shown, an embodiment of the present invention provides a fake news detection method based on multimodal fusion, comprising the following steps: Step 1: Collect news samples, build a news sample dataset, including news text and news images, and perform preprocessing; Step 2: extracting sentence vectors from news texts and graph vectors from news images, wherein the sentence vectors and graph vectors are initial feature vectors representing news texts and news images; Step 3: Use the initial feature vector for feature update and training of the fusion module to obtain features that can reflect the weights between modalities, and then obtain the final feature representation by splicing; Step 4: Input the final feature representation into the fully connected network for calculation, calculate the weight matrix in the network, and establish the connection between the final feature representation and the news label; in the test model stage, the test module can calculate the probability of the category to which the news sample belongs, and then judge the truth or falsity of the news.

[0033] Example 3 Based on Example 2, step 1 includes the following sub-steps: Step 1.1, collect news samples, build a news sample dataset, including news text and news images, and divide the news sample dataset into training set, validation set and test set in a ratio of 3:1:1; the training set is used to train the network, the validation set is used to adjust parameters and prevent overfitting during training, and the test set is used to test the performance of the model.

[0034] Step 1.2: Perform a series of preprocessing on the news text, including word segmentation, stop word removal, stem extraction, etc., so that it can be used by the subsequent feature extraction module; the news image is scaled or cropped to unify the size, for example: 224x224 pixels.

[0035] Step 1.3: Randomize the training set, validation set, and test set to ensure the randomness of the samples, avoid the model's dependence on the order of the data, and ensure that the categories in each data set are relatively evenly distributed to avoid category imbalance problems.

[0036] Example 4 Based on Example 2 or 3, step 2 includes the following sub-steps: Step 2.1: Use the pre-trained BERT model to extract sentence vectors from news texts; Step 2.2: Use the pre-trained VGG-19 model to extract the graph vectors in the news images; Among them, step 2.1 includes: Step 2.1.1: Tokenize the news text, decompose the sentence into words or subwords, and use the vocabulary to map the words or subwords into corresponding word embedding vectors. Step 2.1.2: Based on the word embedding vector, the BERT model, which is composed of multiple stacked Transformer encoders, introduces position encoding to represent the position information of a word or subword in a sentence; position encoding is usually a vector of fixed dimension, which is used to indicate the position of a word in a sentence. Each Transformer encoder in the BERT model contains multiple self-attention layers and feedforward neural network layers. Step 2.1.3: After being processed by the BERT model, the sentence vectors in the news text are extracted and have rich semantic information.

[0037] Step 2.2 includes the following sub-steps: Step 2.2.1. Load the weight parameters from the VGG-19 model; these parameters are trained on a large-scale image dataset and have good image feature extraction capabilities; Step 2.2.2: Input the news image data into the VGG-19 model for forward propagation calculation. During the forward propagation process, the image data will pass through multiple convolutional layers and pooling layers to finally obtain a high-dimensional feature representation. Stop the forward propagation calculation before the last convolutional layer of the VGG-19 model and output the graph vector. These feature representations contain the visual information of the image at different levels of abstraction.

[0038] Example 5 Based on Example 2, 3 or 4, step 3 includes the following sub-steps: Figure 3 As shown: Step 3.1: Record sentence vectors as text features T , and the text features TPerform feature projection to obtain the query vector , key vector Sum value vector , the projection process can be expressed as: in, , and is the corresponding projection matrix; Step 3.2: Record the graph vector as image feature I , and the image features I Perform feature projection to obtain the query vector , key vector Sum value vector , the projection process can be expressed as: in, , and is the corresponding projection matrix.

[0039] Step 3.3: Use mutual attention mechanism to calculate text features T and image features I The attention weights between: in, is the dimension of the key vector, represents the attention weight of the text on the image, Represents the attention weight of the image to the text; Use the attention weights to perform weighted summation on the features to obtain updated text features and image features : Step 3.4: Use splicing to update the text features and image features To perform the fusion: in, F It is the fused feature representation, that is, the final feature representation.

[0040] Example 6 Based on Embodiment 2, 3, 4 or 5, step 4 includes the following sub-steps: Step 4.1: F Input into a fully connected network consisting of multiple hidden layers and an output layer for calculation. l There are hidden layers, and the calculation process can be expressed as: in, Indicates l The output of the hidden layer is and Respectively represent l The weights and biases of the hidden layers; Indicates l -1 hidden layer or input features F ; Step 4.2: Take the output of the last hidden layer as input and perform binary classification. The calculation process of the output layer can be expressed as: in, is the predicted output of the model, and are the weights and biases of the output layer, is the bias of the penultimate hidden layer; ultimately, the probability of the news being true or false can be obtained.

[0041] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A fake news detection system based on multimodal fusion, characterized in that: The system includes a data reading and preprocessing module, a text feature extraction module, an image feature extraction module, a feature updating and fusion module, and a detection module; The data reading and preprocessing module collects news samples, performs preprocessing, constructs a news sample data set, and divides the news sample data set into a training set, a validation set, and a test set according to a preset ratio; the news samples include news text and news images; The text feature extraction module is connected to the data reading and preprocessing module, directly inputs the news text into the pre-trained BERT model, and uses the obtained sentence vector as the feature representation of the news text; The image feature extraction module is connected to the data reading and preprocessing module, directly inputs the news image into the pre-trained VGG-19 model, and uses the obtained image vector as the feature representation of the news image; The feature update and fusion module is connected to the text feature extraction module and the image feature extraction module, and includes a plurality of weight matrices for projecting the sentence vector and the image vector into a query vector, a key vector and a value vector, calculating the attention weight and updating the sentence vector and the image vector, and then fusing the sentence vector and the image vector through vector concatenation to obtain a final feature representation that can highlight the common features of the text and the image; The detection module is connected to the feature updating and fusion module, and includes a plurality of fully connected layers. The parameters in the feature updating and fusion module are updated through training, and the connection between the final feature representation and the news label is established, so as to judge the truth or falsity of the news.

2. The fake news detection system based on multimodal fusion as claimed in claim 1, characterized in that: In the data reading and preprocessing module, a one-to-one matching is performed on the naming and storage location of the news text and the news image, and the news text contains a label of whether the news is true or false.

3. The fake news detection system based on multimodal fusion as claimed in claim 2, characterized in that: In the data reading and preprocessing module, invalid pictures in the news image are regarded as abnormalities, and blank images are generated; for each image in the news image, it is preprocessed into an RGB color image with three channels, and the size of each image is ensured to be consistent.

4. A fake news detection method based on multimodal fusion, characterized in that: The method comprises the following steps: Step 1: Collect news samples, build a news sample dataset, including news text and news images, and perform preprocessing; Step 2: extracting sentence vectors from the news text and image vectors from the news image, wherein the sentence vectors and image vectors are initial feature vectors representing the news text and the news image; Step 3: Using the initial feature vector for training and learning the feature update and fusion module to obtain features that can reflect the weights between modalities, and then obtaining the final feature representation by splicing; Step 4: Input the final feature representation into a fully connected network for calculation, establish a connection between the final feature representation and the news label, and then judge the truth or falsity of the news.

5. The fake news detection method based on multimodal fusion as claimed in claim 4, characterized in that: The step 1 comprises the following sub-steps: Step 1.1, collecting the news samples, constructing the news sample data set, including the news text and the news image, and dividing the news sample data set into a training set, a validation set and a test set in a ratio of 3:1:1; Step 1.2, preprocessing the news text, including word segmentation, stop word removal, and stem extraction; The news image is scaled or cropped to unify the size; Step 1.3: Randomize the training set, the validation set, and the test set.

6. The fake news detection method based on multimodal fusion as claimed in claim 4, characterized in that: The step 2 includes the following sub-steps: Step 2.1, using the pre-trained BERT model to extract the sentence vector in the news text; Step 2.2, using the pre-trained VGG-19 model to extract the graph vector in the news image; Wherein, the step 2.1 comprises: Step 2.1.1, segment the news text, decompose the sentence into words or subwords, and use a vocabulary to map the words or subwords into corresponding word embedding vectors; Step 2.1.2, based on the word embedding vector, the BERT model formed by stacking multiple Transformer encoders introduces position encoding to represent the position information of the word or subword in the sentence; Step 2.1.3, extracting the sentence vector in the news text through the BERT model processing.

7. The fake news detection method based on multimodal fusion as claimed in claim 6, characterized in that: The step 2.2 includes the following sub-steps: Step 2.2.1, load weight parameters from the VGG-19 model; Step 2.2.2: Input the data of the news image into the VGG-19 model and perform forward propagation calculation; stop the forward propagation calculation before the last convolution layer of the VGG-19 model and output the image vector.

8. The fake news detection method based on multimodal fusion as claimed in claim 4, characterized in that: The step 3 includes the following sub-steps: Step 3.1: Record the sentence vector as a text feature T , and the text features T Perform feature projection to obtain the query vector , key vector Sum value vector , the projection process can be expressed as: in, , and is the corresponding projection matrix; Step 3.2: Record the image vector as image feature I , and the image features I Perform feature projection to obtain the query vector , key vector Sum value vector , the projection process can be expressed as: in, , and is the corresponding projection matrix.

9. The fake news detection method based on multimodal fusion as claimed in claim 8, characterized in that: The step 3 also includes the following sub-steps: Step 3.3: Use mutual attention mechanism to calculate the text features T and the image features I The attention weights between: in, is the dimension of the key vector, represents the attention weight of the text on the image, Represents the attention weight of the image to the text; Use the attention weights to perform weighted summation on the features to obtain updated text features and image features : Step 3.4: Use splicing to combine the updated text features and image features To perform the fusion: in, F It is the fused feature representation, that is, the final feature representation.

10. The fake news detection method based on multimodal fusion as claimed in claim 9, characterized in that: The step 4 includes the following sub-steps: Step 4.1: F Input into the fully connected network including multiple hidden layers and an output layer for calculation. l There are hidden layers, and the calculation process can be expressed as: in, Indicates l The output of the hidden layer is and Respectively represent l The weights and biases of the hidden layers; Indicates l -1 hidden layer or input features F ; Step 4.2: Take the output of the last hidden layer as input and perform binary classification. The calculation process of the output layer can be expressed as: in, is the predicted output of the model, and are the weights and biases of the output layer, is the bias of the penultimate hidden layer; ultimately, the probability of the news being true or false can be obtained.

Images