False news detection method based on granularity enhancement and multi-modal high-order fusion

Abstract

The invention relates to a false news detection method based on granularity enhancement and multi-modal high-order fusion, and the method comprises the following steps: obtaining to-be-detected news data, and inputting the news data into a trained granularity-enhanced multi-modal high-order fusion network for detection; wherein the granularity enhancement multi-modal high-order fusion network comprises an encoding module, a granularity enhancement module, a multi-modal high-order fusion module and a classification module which are connected in sequence, and the encoding module comprises a text encoder constructed based on BERT and an image encoder constructed based on CLIP, which are respectively used for performing fine-granularity multi-layer encoding on text information and image information; the granularity enhancement module is used for grouping multilayer coding results to obtain intermediate layer representation; the multi-modal high-order fusion module is constructed based on a bilinear attention network and is used for obtaining high-order fusion features; and the classification module is used for predicting the probability of authenticity of the to-be-detected news data based on the high-order fusion features. Compared with the prior art, the accuracy of the false news detection result can be further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a fake news detection method based on granularity enhancement and multimodal high-order fusion. Background Art

[0002] With the rapid development of the Internet and the popularity of social media, the speed and scope of information dissemination have reached unprecedented heights. However, the accompanying false information has also spread on the Internet at an alarming speed and with a wide range of influence. Timely identification of these false information has become an important task in maintaining the information environment and social order. Therefore, false information detection methods have been widely used in many fields, including but not limited to social media monitoring, news authenticity verification, public opinion analysis, public health information management, and election security.

[0003] Traditional fake news detection focuses on the detection of text content. With the continuous development of multimedia technology, rumormongers are increasingly using multimodal content (such as pictures) to attract public attention. Therefore, research on multimodal fake news detection methods is receiving more and more attention. The key factors affecting the performance of multimodal fake news detection methods are mainly the representation and fusion of multimodal information. Conventional multimodal information representation mainly uses recurrent neural networks to extract text features and convolutional neural networks to extract image features. However, the above methods ignore intermediate hidden states and fine-grained details, which may lead to information loss and failure to capture detailed features. In the fusion of multimodal information, conventional fusion techniques mostly assign weights to information from various modalities through linear fusion. This method mainly relies on first-order feature interactions and fails to perform deep operations, resulting in insufficient information interaction between modalities. With the development of attention mechanism, many studies have begun to use variants of attention mechanism for multimodal information fusion and achieved satisfactory results. For example, Chinese patent application 202111152971.4 proposed a fine-grained multimodal fake news detection method, which uses multimodal collaborative attention Transformer to model the interaction between text features and visual entity features and visual features, thereby capturing the interaction between images and text at different semantic levels. However, the above methods mainly focus on the similarity or consistent semantics between text and visual content, ignoring inconsistent semantics. One of the main characteristics of fake news is the mismatch of information within or between modalities. Therefore, it is necessary to redesign a fake news detection method to further improve the accuracy of fake news detection results. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a fake news detection method based on granularity enhancement and multimodal high-order fusion, reduce the context loss of multimodal semantics, solve the inconsistency problem in the inter-modal fusion process, and further improve the accuracy of fake news detection results.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] The present invention provides a fake news detection method based on granularity enhancement and multimodal high-order fusion, comprising the following steps:

[0007] Obtain news data to be detected, including text information and image information, and input it into the trained granularity-enhanced multimodal high-order fusion network for detection;

[0008] Among them, the granularity-enhanced multimodal high-order fusion network includes a coding module, a granularity enhancement module, a multimodal high-order fusion module and a classification module connected in sequence. The coding module includes a text encoder constructed based on BERT and an image encoder constructed based on CLIP, which are respectively used to perform fine-grained multi-layer encoding of text information and image information; the granularity enhancement module is used to group and sum the results of multi-layer encoding, extract and capture the intermediate layer representation; the multimodal high-order fusion module is constructed based on a bilinear attention network, and is used to obtain high-order fusion features according to the intermediate layer representation; the classification module is used to predict the probability of the authenticity of the news data to be detected based on the high-order fusion features, and the output is the detection result.

[0009] Furthermore, the text encoder and the image encoder each include multiple intermediate layers and an output layer. In the granularity enhancement module, the following operation is performed on each encoder: after all layers are grouped, the outputs of each layer in each group are added together to obtain the intermediate layer representation.

[0010] Furthermore, in the multimodal high-order fusion module, the intermediate layer representation is averaged into a vector, and the query Q and a set of secret keys are obtained based on the attention mechanism. and a set of values Through the bilinear structure, query Q, a set of secret keys and a set of values Perform FIME(·) and FIMD(·) operations to obtain the high-order fusion feature H.

[0011] Furthermore, the definitions of FIMD(·) and FIMD(·) are as follows: For two sets of input vectors and Each element in a set of input vectors is multiplied by all elements in another set of input vectors to generate a feature interaction matrix. The FIMD(·) operation represents extracting the diagonal elements in the feature interaction matrix. The FIMD(·) operation represents calculating the eigenvalues ​​of the feature interaction matrix and extracting the eigenvalue vector of the feature interaction matrix.

[0012] Furthermore, the process of obtaining the high-order fusion feature H is as follows:

[0013] Pair of keys Perform FIME(·) operation on the query Q to obtain a set of vectors Pair Value Perform FIME(·) operation on query Q to obtain enhanced attention distribution

[0014] Pair Vector Perform spatial information aggregation and spatial information retention operations respectively to obtain the attention distribution α and a set of vectors

[0015] To enhance the distribution of attention and Perform FIMD(·) operation to obtain the new attention distribution β;

[0016] Perform FIMD(·) operation on the new attention distribution β and the attention distribution α to obtain the high-order fusion feature H.

[0017] Furthermore, the specific process of the spatial information aggregation operation is: vector Average pooling is performed, followed by compression and aggregation through a linear layer.

[0018] Furthermore, the specific process of the spatial information preservation operation is: embedding the vector Projection is performed, followed by softmax and layer normalization.

[0019] Furthermore, the formula for obtaining the high-order fusion feature H is as follows:

[0020] H=FIMD(β,α)

[0021]

[0022] g i =f layernorm (f softmax (W g E i ))

[0023] E i =σ(W e Ai )

[0024]

[0025] Among them, W a , W g , W i k , and is the weight parameter, f layernorm is the layer normalization operation, f softmax is the softmax operation, σ is the ReLu activation function, and N is the number of vector elements.

[0026] Furthermore, a classification loss function L is constructed based on the cross entropy loss, and the granularity enhanced multimodal high-order fusion network is trained.

[0027] Furthermore, the expression of the classification loss function L is as follows:

[0028]

[0029] Among them, y m and are the actual value and predicted value of the mth sample respectively, and M is the number of samples.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The present invention provides a fake news detection method based on granularity enhancement and multimodal high-order fusion. A granularity enhancement multimodal high-order fusion network is designed to detect news data to be detected. The granularity enhancement multimodal high-order fusion network includes an encoding module, a granularity enhancement module, a multimodal high-order fusion module and a classification module connected in sequence. The encoding module includes a text encoder constructed based on BERT and an image encoder constructed based on CLIP, which are respectively used to perform fine-grained multi-layer encoding on text information and image information to minimize the loss of original semantic details. Among them, BERT has a strong transfer learning ability and can capture rich semantic features. CLIP can process the image into blocks to capture the fine-grained features of the image; the granularity enhancement module is used to group and sum the results of multi-layer encoding, extract and capture the intermediate layer representation, which can reduce the complexity of the model on the one hand, and make full use of the intermediate hidden state and fine-grained details on the other hand, avoid information loss, and better capture detailed features; the multimodal high-order fusion module is constructed based on a bilinear attention network, and is used to obtain high-order fusion features according to the intermediate layer representation. The above design can further improve the accuracy of fake news detection results.

[0032] 2. The present invention defines FIME(·) operation and FIMD(·) operation in the multimodal high-order fusion module, and performs high-order operations on the feature interaction matrix of multiple input fusions, so as to extract more representative comprehensive features. Among them, the FIMD(·) operation is to extract the diagonal elements in the feature interaction matrix, which can not only retain the high-dimensional information of the fused features, but also ensure that the important core features of each modality are not lost. The FIME(·) operation is to calculate the eigenvalues ​​of the feature interaction matrix and extract the eigenvalue vector of the feature interaction matrix. The eigenvalue represents the global property of the matrix and provides a high-dimensional global description of cross-modal information. By capturing important consistent features between modalities, such as similar patterns and potential correlations, the eigenvalue vector can generate a more consistent global representation, thereby effectively reducing the semantic inconsistency between modalities and enhancing the fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the structure of the granularity-enhanced multimodal high-order fusion network.

[0034] Among them, FC represents the fully connected layer;

[0035] Figure 2 It is a structural diagram of the multimodal high-order fusion module.

[0036] Among them, K is the key, Q is the query, V is the value, ELU represents the activation function, FIME and FIMD are the operation methods defined in the present invention, and H is the high-order fusion feature. DETAILED DESCRIPTION

[0037] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0038] Example:

[0039] This embodiment provides a fake news detection method based on granularity enhancement and multi-modal high-order fusion, which runs on a GPU and includes the following steps:

[0040] Obtain the news data to be detected, including text information and image information, and input it into the trained granularity-enhanced multimodal high-order fusion network for detection.

[0041] The structure of the granular enhanced multimodal high-order fusion network is as follows Figure 1 As shown, it includes an encoding module, a granularity enhancement module, a multimodal high-order fusion module and a classification module which are connected in sequence.

[0042] The encoding module includes a text encoder and an image encoder built based on a pre-trained model, which are used to perform fine-grained multi-layer encoding of text information and image information, respectively. Specifically, the text encoder uses BERT (Bidirectional Encoder Representations from Transformers) to capture the text features of the input data, and the image encoder uses CLIP with a visual Transformer (ViT-B / 32) to divide the output image into 49 image blocks and convert each image block into a vector representation to capture the fine-grained features of the image.

[0043] The granularity enhancement module is used to group and sum the results of multi-layer encoding, extract and capture the intermediate layer representation. The text encoder and image encoder each have 11 intermediate layers and 1 output layer. In order to extract richer semantics, all intermediate layers and output layers are used. The 12 layers are grouped into 4 groups to reduce the complexity of the model, and then the contents of each group of three layers are added. By extracting and capturing the semantics of the intermediate layers, the input text and image can be represented by four groups of vectors respectively.

[0044] Each set of vectors is input into the multimodal high-order fusion module to achieve multimodal fusion and capture high-order information. The high-order fusion module is built based on the bilinear attention network, such as Figure 2 As shown in Figure 2, when a modality is used as a query, it is averaged into a unified vector. At this time, the query Q and a set of secret keys can be obtained based on the attention mechanism. and a set of values Through the bilinear structure, query Q, a set of secret keys and a set of values By performing FIME(·) and FIMD(·) operations, the high-order fusion feature H can be obtained.

[0045] The definitions of FIMD(·) and FIMD(·) operations are as follows: For two sets of input vectors and Each element in one set of input vectors is multiplied by all elements in another set of input vectors to generate a feature interaction matrix. The FIMD (·) operation represents the extraction of diagonal elements in the feature interaction matrix. The FIME (·) operation represents the calculation of the eigenvalues ​​of the feature interaction matrix and the extraction of the eigenvalue vector of the feature interaction matrix. For an n×n matrix, n eigenvalues ​​are generated to form an n-dimensional eigenvalue vector. By performing high-order operations on the feature interaction matrix for different modal representations, more representative comprehensive features can be extracted, the context loss of the semantics of the two modalities can be reduced, and the inconsistency problem in the process of inter-modal fusion can be solved. Different modalities (such as images, texts, etc.) often have their own unique feature structures and semantics. Through the FIMD (·) operation, not only can the high-dimensional information of the fused features be retained, but also the important core features of each modality can be ensured not to be lost. Usually, in the process of cross-modal fusion, the problem of inconsistency is encountered, that is, the features at the same position of different modalities often express different contents, which makes it impossible to directly and meaningfully fuse the information of the features of the two modalities at the same position. The eigenvalue extraction of the feature interaction matrix through the FIME (·) operation can effectively solve this problem. The eigenvalue represents the global properties of the matrix and provides a high-dimensional global description of cross-modal information. By capturing important consistent features between modalities, such as similar patterns and potential correlations, the eigenvalue vector can generate a more consistent global representation, thereby effectively reducing the semantic inconsistency between modalities and enhancing the fusion effect.

[0046] The process of obtaining the high-order fusion feature H is as follows:

[0047] Pair of keys Perform FIME(·) operation on the query Q to obtain a set of vectors Pair Value Perform FIME(·) operation on query Q to obtain enhanced attention distribution

[0048]

[0049] Pair Vector Perform spatial information aggregation and spatial information retention operations respectively to obtain the attention distribution α and a set of vectors The spatial information aggregation operation is as follows: Perform average pooling, and then compress and aggregate through a linear layer:

[0050]

[0051] The spatial information preservation operation is: embedding the vector Projection is performed, followed by softmax and layer normalization:

[0052] E i =σ(W e A i )

[0053] g i =f layernorm (f softmax (W g E i ))

[0054] To enhance the distribution of attention and Perform FIMD(·) operation to obtain the new attention distribution β:

[0055]

[0056] Perform FIMD(·) operation on the new attention distribution β and the attention distribution α to obtain the high-order fusion feature H:

[0057] H=FIMD(β,α)

[0058] In the above formula, W a , W g , and is the weight parameter, f layernorm is the layer normalization operation, f softmax is the softmax operation, σ is the ReLu activation function, and N is the number of vector elements.

[0059] Finally, the high-order fusion feature H is input into the classification module to predict the probability of the authenticity of the news data to be detected, and the output is the detection result. The classification module includes splicing, fully connected layer and layer normalization.

[0060] Based on the cross entropy loss, a classification loss function L is constructed to train the granularity enhanced multimodal high-order fusion network. The expression of the classification loss function L is as follows:

[0061]

[0062] Among them, y m and are the actual value and predicted value of the mth sample respectively, and M is the number of samples.

[0063] To verify the effectiveness of the above method, this embodiment tests the above method on two public test data sets, selecting accuracy, F1 score, prediction rate and recall rate as indicators. This embodiment uses the public fake news detection data sets Pheme and Weibo. The data sets are collected from the foreign social media platform Twitter and the domestic social media platform Weibo. The data set annotation format is csv format. In the end, there are 2018 Pheme data and 9528 Weibo data, each of which contains a pair of image-text pairs. At the same time, this embodiment also uses the same data set to test some mainstream methods currently used in fake news detection tasks. The experimental data of the Pheme data set are shown in Table 1, and the experimental data of the Weibo data set are shown in Table 2. In Tables 1 and 2, method 1 is a unimodal data set using text data by the CNN method, method 2 is a multimodal data set using image-text pairs by the MRML method, method 3 is a multimodal data set using image-text pairs by the HMCAN method, method 4 is a multimodal data set using image-text pairs by the MFAN method, and method 5 is the method of the present invention.

[0064] Table 1 Experimental data of Pheme dataset

[0065]

[0066]

[0067] Table 2 Experimental data of Weibo dataset

[0068] Accuracy F1 score Prediction rate Recall Method 1 69.0 68.7 65.7 64.3 Method 2 90.7 87.2 88.0 86.3 Method 3 88.1 85.4 89.1 85.1 Method 4 88.7 86.2 87.0 85.6 Method 5 93.6 90.0 90.2 87.0

[0069] From the comparison between method 1 and method 5 (the method of the present invention), it can be seen that compared with the single-modality method, the method of the present invention supplements the additional information of other modalities, which greatly improves the detection performance. From the comparison between method 2 and method 5, it can be seen that the method of the present invention extracts multi-level information of the original content and performs fine-grained processing on the image representation, so that the detection performance is improved. From the comparison between method 3 and method 4 and method 5, it can be seen that the method of the present invention retains the high-dimensional features of modal interaction for multiple calculations, and fuses the modalities multiple times through a bilinear form, so that the detection performance is improved.

[0070] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0071] The above description of the embodiments is to facilitate the understanding and use of the invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the present invention should be within the scope of protection of the present invention.

Claims

1. A fake news detection method based on granularity enhancement and multimodal high-order fusion, characterized in that: The following steps are involved: Obtain news data to be detected, including text information and image information, and input it into the trained granularity-enhanced multimodal high-order fusion network for detection; Among them, the granularity-enhanced multimodal high-order fusion network includes a coding module, a granularity enhancement module, a multimodal high-order fusion module and a classification module connected in sequence. The coding module includes a text encoder constructed based on BERT and an image encoder constructed based on CLIP, which are respectively used to perform fine-grained multi-layer encoding of text information and image information; the granularity enhancement module is used to group and sum the results of multi-layer encoding, extract and capture the intermediate layer representation; the multimodal high-order fusion module is constructed based on a bilinear attention network, and is used to obtain high-order fusion features according to the intermediate layer representation; the classification module is used to predict the probability of the authenticity of the news data to be detected based on the high-order fusion features, and the output is the detection result.

2. According to claim 1, a fake news detection method based on granularity enhancement and multimodal high-order fusion is characterized in that: The text encoder and the image encoder each include a plurality of intermediate layers and an output layer. In the granularity enhancement module, the following operation is performed on each encoder: after all layers are grouped, the outputs of each layer in each group are added together to obtain the intermediate layer representation.

3. According to claim 1, a fake news detection method based on granularity enhancement and multimodal high-order fusion is characterized in that: In the multimodal high-order fusion module, the intermediate layer representation is averaged into a vector, and the query Q and a set of secret keys are obtained based on the attention mechanism. and a set of values Through the bilinear structure, query Q, a set of secret keys and a set of values Perform FIME(·) and FIMD(·) operations to obtain the high-order fusion feature H.

4. According to claim 3, a fake news detection method based on granularity enhancement and multimodal high-order fusion is characterized in that: The definitions of the FIME(·) operation and the FIMD(·) operation are as follows: For two sets of input vectors and Each element in one set of input vectors is multiplied by all elements in another set of input vectors to generate a feature interaction matrix. The FIMD(·) operation represents extracting the diagonal elements in the feature interaction matrix. The FIMF(·) operation represents calculating the eigenvalues ​​of the feature interaction matrix and extracting the eigenvalue vector of the feature interaction matrix.

5. The fake news detection method based on granularity enhancement and multimodal high-order fusion according to claim 4 is characterized in that: The process of obtaining the high-order fusion feature H is as follows: Pair of keys Perform FIME(·) operation on query Q to obtain a set of vectors Pair Value Perform FIME(·) operation on query Q to obtain enhanced attention distribution Pair Vector Perform spatial information aggregation and spatial information retention operations respectively to obtain the attention distribution α and a set of vectors To enhance the distribution of attention and Perform FIMD(·) operation to obtain the new attention distribution β; Perform FIMD(·) operation on the new attention distribution β and the attention distribution α to obtain the high-order fusion feature H.

6. The fake news detection method based on granularity enhancement and multimodal high-order fusion according to claim 5, characterized in that: The specific process of the spatial information aggregation operation is: Average pooling is performed, followed by compression and aggregation through a linear layer.

7. The fake news detection method based on granularity enhancement and multimodal high-order fusion according to claim 5, characterized in that: The specific process of the spatial information preservation operation is: embedding the vector A= Projection is performed, followed by softmax and layer normalization.

8. The fake news detection method based on granularity enhancement and multimodal high-order fusion according to claim 5, characterized in that: The formula for obtaining the high-order fusion feature H is as follows: H=FIMD(β,α) g i =f layernorm (f softmax ((W g E i )) E i =σ(W e A i ) Among them, W a , W g , W i k , W i v and is the weight parameter, f iayernorm is the layer normalization operation, f softmax is the softmax operation, σ is the ReLu activation function, and N is the number of vector elements.

9. The fake news detection method based on granularity enhancement and multimodal high-order fusion according to claim 1, characterized in that: A classification loss function L is constructed based on the cross entropy loss, and the granularity enhanced multimodal high-order fusion network is trained.

10. The fake news detection method based on granularity enhancement and multimodal high-order fusion according to claim 9, characterized in that: The expression of the classification loss function L is as follows: Among them, y m and are the actual value and predicted value of the mth sample respectively, and M is the number of samples.

Images