Visual question and answer method based on multi-modal decomposition model

Abstract

A visual question and answer method based on a multi-modal decomposition model is provided. The method is characterized in that the image is trained in an ImageNet dataset and image features are extracted, the question is marked as texts into feature vectors; a collaborative attention model is introduced into the basic network architecture, image and question related features are learned, and fine-grained correlation among multi-modal features are characterized; and multi-modal features enter a multi-modal decomposition bilinear pool (MFB) or a multi-modal decomposition high order cell (MFH) module image to generate a fusion image question feature z, and the z is put into the classifier to predict the best matching answer. According to the method provided by the present invention, the collaborative attention model is used to predict the correlation between each spatial grid in the image and the question, and the best matching answer can be accurately predicted in a facilitated manner;and in combination with the image attention mechanism, the model can effectively understand which image region is important to the question, so that the performance of the model and the accuracy of the question and answer can be significantly improved.

Description

technical field [0001] The invention relates to the field of visual question answering, in particular to a method for visual question answering based on a multimodal decomposition model. Background technique [0002] With the continuous development of machine vision, the automatic understanding of semantic representation in images by machines has been widely studied. Visual question answering is often used in image retrieval, intelligent transportation, visual education, artificial intelligence and other fields. Specifically, in the field of image retrieval, it can By understanding images and questions, decompose key information and obtain corresponding text descriptions. In the field of visual education, specific features are decomposed, and the correct answer is obtained through inference combined with the feature information in the text. A step forward in improving artificial intelligence. The current existing research only considers the visual features in the image, bu...

AI Technical Summary

Problems solved by technology

[0004] Aiming at the current low performance and low accuracy, the present invention uses MFB to fuse the problem and the visual features of the image, and uses MFH to obtain more relevant visual features, and cooperates with the attention model to predict the relationship between each spatial grid in the image and the problem. Correlation, which is conducive to accurately predicting the best matching answer, combined with this image attention mechanism, can make the model effectively understand which image region is important for the problem, and significantly improve the performance of the model

Images