Image description generation method based on depth LSTM network

Abstract

The invention relates to an image description generation method based on a depth LSTM network, comprising the following steps: (1) extracting the CNN characteristics of an image in an image description dataset, and acquiring an embedded vector corresponding to the image and describing the words in a reference sentence; (2) building a double-layer LSTM network, and carrying out series modeling based on the double-layer LSTM network and a CNN network to generate a multimodal LSTM model; (3) training the multimodal LSTM model by means of joint training; (4) gradually increasing the number of layers of the LSTM network in the multimodal LSTM model, carrying out training each time one layer is added to the LSTM network, and finally, getting a gradual multi-objective optimization and multilayer probability fused image description model; and (5) fusing the probability scores output by the branches of the multilayer LSTM network in the gradual multi-objective optimization and multilayer probability fused image description model, and outputting the word corresponding to the maximum probability through common decision. Compared with the prior art, the method has such advantages as multiple layers, improved expression ability, effective updating, and high accuracy.

Description

technical field [0001] The invention relates to the field of image understanding, in particular to an image description generation method based on a deep LSTM network. Background technique [0002] Image caption generation is a very challenging task, and it has broad application prospects in the fields of early childhood education, visually impaired assistance, and human-computer interaction. It combines the two fields of natural language processing and computer vision to describe a natural image in the form of natural language, or translate the image into natural language. It first requires the system to be able to accurately understand the content in the image, such as identifying the scene in the image, various objects, object attributes, ongoing actions, and the relationship between objects, etc.; then according to grammatical rules and language structure, generate human understandable sentences. [0003] A variety of methods have been proposed to solve this problem, i...

AI Technical Summary

Problems solved by technology

However, these methods rely too much on the previous visual technology, the processing process is complicated, and the language model for generating sentences at the back end of the system is insufficiently optimized; when LSTM units are used to generate sentences, the model depth is relatively shallow (1-layer or 2-layer LSTM is often used), The level of multimodal information transformation is not enough, the semantic information of the generated sentences is not strong, and the overall performance is difficult to improve

Images