Visual question and answer fusion enhancement method based on multi-modal fusion

Abstract

The invention discloses a visual question and answer fusion enhancement method based on multi-modal fusion. The method comprises the following steps of 1, constructing a time sequence model by utilizing a GRU structure, obtaining feature representation learning of a problem, and utilizing output which is extracted from Faster R-CNN and is based on an attention model from bottom to top as the feature representation; 2, performing multi-modal reasoning based on an attention model Transformer, and introducing the attention model for performing multi-modal fusion on a picture-problem-answer tripleset, and establishing an inference relation; and 3, providing different reasoning processes and result outputs for different implicit relationships, and performing label distribution regression learning according to the result outputs to determine answers. According to the method, answers are obtained based on specific pictures and questions and directly applied to applications serving the blind,the blind or visually impaired people can be helped to better perceive the surrounding environment, the method is also applied to a picture retrieval system, and the accuracy and diversity of pictureretrieval are improved.

Description

technical field [0001] The invention belongs to the technical fields of computer vision and natural language. In particular, the invention relates to a multimodal fusion-based visual question-answer fusion enhancement method. Background technique [0002] Visual Question Answer (VQA for short) is a task that combines the fields of computer vision and computer natural language. What it needs to solve is to ask a specific question for a specific picture and reason its answer. VQA has many potential application scenarios, the most direct ones are those that help blind and visually impaired users. It can understand the surrounding environment for blind or visually impaired users. Through interactive programs, it can perceive the Internet and real life scenes. ; Another obvious application is to integrate VQA into an image retrieval system, which can affect image retrieval through natural language, which has a huge impact on social or business. [0003] VQAtask mainly solves the...

AI Technical Summary

Problems solved by technology

[0011] Disadvantages: These methods almost only consider the relationship between the question and the image, but ignore the hidden relationship between the image-question-answer triplet. We can imagine that when you know the specific answer, you also There may be the ability to speculate on the question, so the role of the answer may play a very important role in the reasoning process, but these methods ignore this important information. Of course, there are also works that take this relationship into account and try to represent it by image features. , question feature representation, and answer feature representation for simple splicing, or to fuse images and questions and then map to answer features, but these two methods are difficult to fully express the relationship between triplets

[0012] Combining the above technologies, it is not difficult to see that the current visual question answering is mainly based on the fusion of image features and question features, but the answer features do not play their due role. Sometimes the answer features can even greatly improve the task accuracy, and the image- The complex relationship between questions and answers cannot be fully expressed, and has the following disadvantages:

[0013] 1. Without effective use of the answer information, the huge role of the answer information cannot be brought into play;

[0014] 2. When multi-modal fusion of question features and image features, it is impossible to use the attention mechanism concisely and effectively to obtain the most worthy of attention.

Images