Image title automatic generation method based on multi-modal attention

Abstract

The invention belongs to the cross technical field of the computer vision and natural language processing, and provides an image title automatic generation method based on multi-modal attention. The alignment problem of vision feature and language feature and the sentence feature ignoring problem in the word prediction process in the traditional method based on neural network are solved, the convergence speed and the quality of the image title are improved. The method comprises the following steps: automatically performing feature extraction on an image region by utilizing a convolutional neural network; and then realizing sentence feature extraction by utilizing LSTM with vision attention; and finally producing a final image title by designing the LSTM with the multi-modal attention (vision attention and hidden variable attention). The experiment proves that the proposed method has excellent result on the MS COCO and like standard data set.

Description

technical field [0001] The invention belongs to the cross technical field of computer vision and natural language processing, and relates to an image caption automatic generation method based on multimodal attention. Background technique [0002] The essence of generating captions for images is to translate images into language. Designing an efficient algorithm for automatic generation of image captions can enable systems (humans or computers) that lack or have poor vision to have the ability to perceive the surrounding environment. In recent years, there has been a lot of novel work fusing advances in computer vision and natural language processing with promising results. According to the way of caption generation, these works can be divided into three categories: template matching-based methods, transfer-based methods and neural network-based methods. [0003] The method based on template matching first uses multiple classifiers to identify the objects, attributes, and a...

AI Technical Summary

Problems solved by technology

[0005] There are two problems in traditional neural network-based methods: 1. The image classification dataset used to train CNN contains a limited number of categories, and many semantic information (such as color and size) that often appear in image captions are not available. Included in a limited number of image categories, so there is no strict alignment between the visual information extracted by CNN trained on image classification datasets and the semantic information in image captions

Images