Method for detecting false information based on multi-modal fusion mechanism of common attention

Abstract

A method for detecting false information based on a common-attention multi-modal fusion mechanism comprises the following steps: S1, constructing a BERT model for extracting text features; s2, constructing an R-CNN model for extracting visual features; s3, constructing a co-attention converter layer Co-TRM model fusing the text and the visual features; s4, inputting the fused feature vectors into a domain classifier, mapping multi-modal features of different information to the same feature space, classifying texts into different events, and deleting special features of the events; and S5, inputting the fused feature vectors into a false information detector, and judging whether the information is true or false by using potential multi-modal features. According to the method, a common attention mechanism is utilized, a random pixel sampling mechanism is used for enhancing the robustness of visual performance, improving a multi-modal model structure and respectively enhancing a text feature extraction method and a visual feature extraction method, so that information exchange among different modals is improved, and the accuracy of false information detection is improved.

Description

technical field [0001] The invention belongs to the technical field of natural language processing, and in particular relates to a method for detecting false information based on a multimodal fusion mechanism based on co-attention. Background technique [0002] In recent years, with the rapid development of computer technology, the electronic age has arrived, which prompts people to actively or passively accept a large amount of information every day. Most of the information we accept is the combination of pictures and texts. Vivid and striking. However, some people are motivated by interests and will publish some false information. These false information often contain tampered or forged pictures, combined with words, to interfere with everyone's vision. The dissemination of serious false information can affect a person's reputation and future at light, and affect the election of a country's leader at worst. Most of the time, it is difficult for people to judge whether a ...

AI Technical Summary

Problems solved by technology

[0004] Disadvantage 1: The text and image features are extracted separately, and the text and image features are simply fused. Before the multi-modal feature fusion, the information in the text does not take into account the influence of the information on the image and the information in the image on the text. What is the effect, the information between the text and the image is not well fused

[0005] Disadvantage 2: Attention is generally placed on key image areas, and the impact of images that have nothing to do with text on false information classification is not considered

Furthermore, feature representation capabilities are limited to a given category of such task-specific models, while broader semantic visual information, such as scene and emotion, are lost in object detection models

Images