A fake news identification method based on multimodal anomaly detection mechanism

Abstract

This invention discloses a fake news identification method based on a multimodal anomaly detection mechanism, comprising the following steps: S1, obtaining information about images and text in news from news websites; S2, mining relevant information about the images and text based on a specific algorithm; and S3, detecting whether the image and text information in the input news are consistent, thereby distinguishing true from false news. This invention proposes a novel fake news identification method based on a multimodal anomaly detection mechanism for the first time, addressing the blind spots of single-modal fake news identification methods by exploring and utilizing the specific relationships between regional features in images and word features in text.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a fake news identification method based on a multimodal anomaly detection mechanism. Background Art

[0002] Currently, fake news can have a devastating impact on individuals, countries, and societies. There is a significant demand for AI-based fake news identification. Fake news identification methods based on text analysis identify semantic inconsistencies within the text context to determine if the news is fake. Fake news identification methods based on image manipulation detection can identify whether images in news have been modified, thereby determining whether the news is fake.

[0003] Existing fake news identification methods, whether based on text analysis or image fraud detection, only utilize unimodal data such as text and images. For news containing both text and images, these methods fail to fully consider the multimodal nature of news and the connections between these data, resulting in poor fake news identification accuracy. Among fake news, there is a type of news where the image is unmodified and the contextual semantics of the text are consistent, but the mismatch between the image and text results in mismatches. Neither text analysis nor image fraud detection methods can identify this type of fake news, a blind spot for unimodal fake news identification methods. Summary of the Invention

[0004] In response to fake news caused by mismatch between images and text, the present invention provides a fake news identification method based on a multimodal anomaly detection mechanism to solve the blind spot problem of existing fake news identification methods.

[0005] The present invention provides the following technical solutions:

[0006] A fake news identification method based on a multimodal anomaly detection mechanism, characterized in that it comprises the following steps:

[0007] S1, obtains information about pictures and texts in news from news websites;

[0008] S2, mines relevant information of images and texts based on specific algorithms;

[0009] S3, detects whether the image and text information in the input news are consistent and determines whether the news is true or false.

[0010] Furthermore, the S2 step of mining the relevant information of the image according to the specific algorithm includes: obtaining the image feature I={I1,I2,…,I m}, where Faster-Rcnn is used to extract the features of the region in the image.m is the feature of a single region, and m is the number of regions.

[0011] Furthermore, the relevant information of the text is mined according to a specific algorithm in S2, including: obtaining the text feature C={C1, C2, ..., C n}, where BERT is used to extract the features of a single word in the text, C n is the feature of a single region, and n is the number of words.

[0012] Furthermore, the S2 step of mining the relevant information of the image and text according to a specific algorithm includes: mapping the image and text features into the same semantic space through a fully connected layer, and the formula is as follows:

[0013] I′=W img I;

[0014] C′=W cap C.

[0015] Where C is the text feature, I is the image feature; W img is the parameter that needs to be learned in the fully connected layer for image embedding, W caP is the parameter that needs to be learned in the fully connected layer for text embedding; C′ is the text feature containing word features, C={C1,C2,...,C n}; I′ is the image feature containing regional features, I′={I′1, I′2, ..., I′ m}.

[0016] Furthermore, mining the relevant information of the image and text according to the specific algorithm in S2 includes:

[0017] In the conduction rules of graph convolutional neural networks, it is necessary to input the adjacency matrix and the node feature matrix; for the adjacency matrix, the expanded correlation matrix is used instead; in the same semantic space, for an image, there is an image feature I′={I′1,I′2,…,I′ m}, for a text, there are text features C′={C′1, C′2, ..., C′ n};

[0018] In order to obtain the correlation between images and texts, the Attontion mechanism is introduced to obtain the correlation matrix between different regions in the image and different words in the text. The correlation matrix A is calculated using the following formula:

[0019]

[0020] where I′ iis the image feature containing regional features, C′ j is a text feature containing word features, T is the transposition symbol in linear algebra; i is the label of the image area in the image, j is the label of the word in the text; A ij Represents the correlation matrix between different regions in the image and different words in the text;

[0021] The expanded matrix is used as the adjacency matrix of the graph, so that the subsequent graph convolutional neural network can grasp the key points when mining node information in the graph and mine important related information between image regions and text words;

[0022] For node features, the image and text features in the same semantic space are spliced together as the node features as input; the features of each node in the graph can be obtained by splicing the image and text features using the following formula:

[0023] X = concat{I′, C′};

[0024] The concat function is used to concatenate two features in the first dimension; C is the image feature containing the regional feature, C′={C′1,C′2,...,C′ n}; I′ is the text feature containing word features, I′={I′1,I′2,...,I′ m};

[0025] In order to mine the correlation information between nodes in the graph, two layers of graph convolution layers are used to obtain the information. The contribution of each node in the graph to the graph can be obtained through the two-layer graph convolutional neural network, which is calculated by the following formula:

[0026]

[0027]

[0028] Where f(X, A) is the conduction function of the graph convolutional neural network, W (0) is the parameter that needs to be learned in the first layer of graph convolution layer, Z (0) is the output of the first graph convolutional layer, W (1) are the parameters that need to be learned in the second layer of graph convolution; X is the total node features after the image features and text features are spliced together, and A is the expansion matrix obtained from the correlation matrix. is the normalized adjacency matrix.

[0029] Furthermore, the step of detecting whether the image and text information in the input news are consistent in S3 includes:

[0030] After two layers of graph convolution, a fully connected layer is added to predict whether the input image and text are consistent. This is achieved using the following formula:

[0031] Y=WZ (1)

[0032] Where Y is the prediction result, W is the parameter to be learned for the final fully connected layer, and Z (1) is the output after two convolutional layers.

[0033] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0034] This paper proposes for the first time a new method for fake news identification and detection based on a multimodal anomaly detection mechanism, which solves the blind spots of single-modal fake news identification methods and aims to explore and utilize the specific relationship between regional features in images and word features in texts.

[0035] It can quickly and accurately detect whether the input image and text are consistent. Its main application is to detect the consistency of images and texts in fake news identification to assist in distinguishing true from false news. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0037] The embodiments of the present invention are described in detail below. The following embodiments are implemented based on the technical solutions of the present invention, and provide detailed implementation methods and specific operating procedures. However, the protection scope of the present invention is not limited to the following embodiments.

[0038] Example

[0039] See also Figure 1 The present invention provides a fake news identification method based on a multimodal anomaly detection mechanism, which includes the following steps:

[0040] First, we use pre-trained models, namely Faster-RCNN and BERT, to perform fine-grained feature extraction on image data and text data to obtain regional features in the image and word features in the text.

[0041] Since the pictures in the news include different areas and the text also includes different words, the present invention introduces the Attention mechanism by considering the multiple area information of the pictures and the word information of the text, expands the calculated correlation matrix, and splices the features of the pictures and text, and regards the areas in the pictures and the words in the text as nodes in the "graph".

[0042] Considering that graph convolutional neural networks have the characteristic of mining information between each node, a graph convolutional neural network is introduced, and the expanded matrix and spliced features are used as the input of the graph convolutional network to mine information about areas in images and words in texts.

[0043] After passing through two layers of graph convolution, we classify the input image-text pairs. The method proposed in this paper can be used as a possible method for identifying fake news on social media in the future, providing a possible approach for identifying fake news.

[0044] like Figure 1 As shown, the present invention specifically includes the following steps:

[0045] S1, obtains information about pictures and texts in news from news websites;

[0046] S2, mines relevant information of images and texts based on specific algorithms;

[0047] S3, detects whether the image and text information in the input news are consistent and determines whether the news is true or false.

[0048] Furthermore, the mining of relevant information of the image according to the specific algorithm in S2 includes: obtaining the image feature I={I1,I2,…,I m}, where Faster-Rcnn is used to extract the features of the region in the image. m is the feature of a single region, and m is the number of regions.

[0049] Furthermore, the mining of text-related information according to a specific algorithm in S2 includes: obtaining text features C={C1, C2, ..., C n}, where BERT is used to extract the features of a single word in the text, C n is the feature of a single region, and n is the number of words.

[0050] Furthermore, mining the relevant information of the image and text according to the specific algorithm in S2 includes: mapping the image and text features into the same semantic space through a fully connected layer, and the formula is as follows:

[0051] I′=W img I

[0052] C′=W cap C

[0053] Where C is the text feature, I is the image feature; W img is the parameter that needs to be learned in the fully connected layer for image embedding, W capis the parameter that needs to be learned in the fully connected layer for text embedding; C′ is the text feature containing word features, C′={C′1,C′2,…,C′ n}; I′ is the image feature containing regional features, I′={I′1, I′2, ..., I′ m}.

[0054] Furthermore, mining the relevant information of the image and text according to the specific algorithm in S2 includes:

[0055] In the conduction rule of the graph convolutional neural network, it is necessary to input the adjacency matrix and the node feature matrix; for the adjacency matrix, the expanded correlation matrix is used instead in the present invention; in the same semantic space, for an image, there is an image feature I′={I′1,I′2,...,I′ m}, for a text, there are text features C′={C′1, C′2, ..., C′ n};

[0056] In order to obtain the correlation between images and texts, the Attention mechanism is introduced to obtain the correlation matrix between different regions in the image and different words in the text. The correlation matrix A is calculated using the following formula:

[0057]

[0058] where I′ i is the image feature containing regional features, C′ j is a text feature containing word features, T is the transposition symbol in linear algebra; i is the label of the image region in the image, and j is the label of the word in the text;

[0059] The expanded matrix is used as the adjacency matrix of the graph, so that the subsequent graph convolutional neural network (GCN) can grasp the key points when mining node information in the graph and mine important related information between image regions and text words;

[0060] For node features, the image and text features in the same semantic space are spliced together as the node features as input; the features of each node in the graph can be obtained by splicing the image and text features using the following formula:

[0061] X = concat{I′, C′}

[0062] The concat function is used to concatenate two features in the first dimension; C′ is the image feature containing the regional feature, C′={C′1,C2,...,C′ n}; I′ is the text feature containing word features, I′={I′1,I′2,...,I′ m};

[0063] In order to mine the correlation information between nodes in the graph, the present invention uses two layers of graph convolution layers to obtain the information. The contribution of each node in the graph to the graph can be obtained through the two-layer graph convolutional neural network, which is calculated by the following formula:

[0064]

[0065]

[0066] Where f(X, A) is the conduction function of the graph convolutional neural network, W (0) is the parameter that needs to be learned in the first layer of graph convolution layer, Z (0) is the output of the first graph convolutional layer, W (1) are the parameters that need to be learned by the second graph convolution layer; X is the total node features after the image features and text features are concatenated; A is the expanded matrix obtained from the correlation matrix, which is the normalized adjacency matrix.

[0067] Furthermore, the step of detecting whether the image and text information in the input news are consistent in S3 includes:

[0068] After two layers of graph convolution, a fully connected layer is added to predict whether the input image and text are consistent. This is achieved using the following formula:

[0069] Y=WZ (1)

[0070] Where Y is the prediction result, W is the parameter to be learned for the final fully connected layer, and Z (1) is the output after two convolutional layers.

[0071] The above-mentioned embodiment of the present invention proposes for the first time a new method for fake news identification and detection based on a multimodal anomaly detection mechanism, which solves the blind spots of single-modal fake news identification methods and aims to explore and utilize the specific relationship between regional features in images and word features in texts.

[0072] The above describes in detail the preferred embodiments of the present invention. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A fake news identification method based on a multimodal anomaly detection mechanism, characterized in that: The main steps include: S1, obtains information about pictures and texts in news from news websites; S2, mines relevant information of images and texts based on specific algorithms; S3, detects whether the image and text information in the input news are consistent and distinguishes true from false news; The mining of relevant information of images and texts according to a specific algorithm in S2 includes: In the conduction rules of graph convolutional neural networks, it is necessary to input the adjacency matrix and the node feature matrix; for the adjacency matrix, the expanded correlation matrix is used instead; in the same semantic space, for an image, there is an image feature containing regional features I'={I'1,I'2,...,I' m }, for a text with a text feature containing word features C'={C'1,C'2,...,C' n In order to obtain the correlation between the image and the text, the Attention mechanism is introduced to obtain the correlation matrix between different regions in the image and different words in the text. The correlation matrix A is calculated using the following formula: where I′ i is the image feature containing regional features, C′ j is a text feature containing word features, T is the transposition symbol in linear algebra; i is the label of the image area in the image, j is the label of the word in the text; A ij Represents the correlation matrix between different regions in the image and different words in the text; The expanded matrix is used as the adjacency matrix of the graph, so that the subsequent graph convolutional neural network can focus on the node information in the graph and mine important related information between the image area and the text words; For node features, the image and text features in the same semantic space are spliced together as the node features as input; the image and text features are spliced together using the following formula to obtain the features of each node in the graph: X = concat{I',C'}; The concat function is used to concatenate two features in the first dimension; I′ is the image feature containing the regional feature, C′={C′1,C′2,...,C′ n }; C′ is the text feature containing word features, I′={I′1,I′2,...I′ m }; In order to mine the correlation information between nodes in the graph, two layers of graph convolution layers are used to obtain the information. The contribution of each node in the graph to the graph can be obtained through the two-layer graph convolutional neural network, which is calculated by the following formula: Where f(X,A) is the conduction function of the graph convolutional neural network, W (0) is the parameter that needs to be learned in the first layer of graph convolution layer, Z (0) is the output of the first graph convolutional layer, W (1) are the parameters that need to be learned in the second layer of graph convolution; X is the total node features after the image features and text features are spliced together, and A is the expansion matrix obtained from the correlation matrix. is the normalized adjacency matrix, Z (1) is the output after two convolutional layers.

2. A fake news identification method based on a multimodal anomaly detection mechanism according to claim 1, characterized in that: The mining of relevant information of the image according to a specific algorithm in S2 includes: The image is passed through Faster-Rcnn to obtain the image features I={I1,I2,…,I m }, where Faster-Rcnn is used to extract the features of the region in the image. m is the feature of a single region, and m is the number of regions.

3. The fake news identification method based on a multimodal anomaly detection mechanism according to claim 1 is characterized in that: The S2 step of mining the relevant information of the text according to the specific algorithm includes: obtaining the text feature C={C1, C2, ..., C n }, where BERT is used to extract the features of a single word in the text, C n is the feature of a single region, and n is the number of words.

4. The fake news identification method based on a multimodal anomaly detection mechanism according to claim 1 is characterized in that: The S2 step mines the relevant information of the image and text according to a specific algorithm, including mapping the image and text features into the same semantic space through a fully connected layer. The formula is as follows: I'=W img I; C'=W cap C; Where C is the text feature, I is the image feature; W img is the parameter that needs to be learned in the fully connected layer for image embedding, W cap is the parameter that needs to be learned in the fully connected layer for text embedding; C′ is the text feature containing word features, C′={C′1,C′2,…,C′ n }; I′ is the image feature containing regional features, I′={I′1, I′2, ..., I′ m }.

5. The fake news identification method based on a multimodal anomaly detection mechanism according to claim 1 is characterized in that: The step S3 of detecting whether the image and text information in the input news are consistent includes: After two layers of graph convolution, a fully connected layer is added to predict whether the input image and text are consistent. This is achieved using the following formula: Y=WZ (1) Where Y is the prediction result, W is the parameter to be learned for the final fully connected layer, and Z (1) is the output after two convolutional layers.

Images