Neural network visual dialogue model and method based on KR product fusion multi-modal information

Abstract

The invention discloses a neural network visual dialogue model and method based on KR product fusion multi-modal information. The model comprises a modal feature extraction module, a different modal information fusion module and a candidate answer prediction module. The modal feature extraction module extracts features of question texts and features of historical information through an LSTM network, extracts entity features of pictures by using a Faster R-CNN network, and extracts visual features related to questions by using an attention mechanism; the different modal information fusion module captures feature information in different modals by using a later fusion method, captures associated information among the different modals through a feature fusion method based on a KR product, and fuses the information in the modals and the information among the modals; and the candidate answer prediction module performs answer prediction by using a fusion vector fusing intra-modal information and inter-modal information, so that related answers can be found out more accurately. According to the method, the current situation that in a traditional visual dialogue model, associated information among different modals is insufficiently captured through later fusion is overcome.

Description

technical field [0001] The present invention relates to the technical field of visual dialogue and multimodal fusion, specifically, to a model and a method for judging the real answer from candidate answers for a certain picture, historical dialogue information and corresponding questions. Background technique [0002] Visual dialogue is a challenging task at the intersection of language and vision, which needs to consider the historical information of multiple rounds of dialogue and related information in images to find the best candidate answer to the current question. Visual dialogue appears in many application scenarios, such as helping blind people understand their surroundings, interactive search, indoor navigation, etc. In the visual dialogue task, in order to capture the information related to the answer, the model needs to understand the question, capture the visual information and historical information related to the question, and capture the potential association...

AI Technical Summary

Problems solved by technology

[0003] In order to achieve a better visual dialogue model, the current main challenge is: the visual dialogue task needs to model the image content and dialogue history, and obtain useful information for answer prediction from the question vector, visual vector and historical vector

Images