A Generative Approach from Structured Text to Image Descriptions

Abstract

The invention discloses a generation method of an image description from a structured text. The generation method comprises the steps of downloading pictures from the internet to form a picture training set; conducting morphological analysis on descriptions which correspond to the pictures in the picture training set to form the structured text; using an existing neural network model to extract convolution neural network characteristics of the pictures in the training set, and using <, picture characteristics and structured text < as inputs to form a multitasking recognition model; using the structured text extracted from the training set and a description which corresponds to the structured text as inputs of a recurrent neural network, and conducting training to obtain a parameter of a recurrent neural network model; inputting the convolution neural network characteristics of an image ready to be described, and obtaining a predicted structured text through the multitasking recognition model; inputting the predicted structured text, and obtaining the image description through the recurrent neural network model. Compared with the prior art, a better image description effect, accuracy and sentence variety can be generated through the method, and the generation method of the image description from the structured text can be effectively popularized in an application of image retrieval.

Description

technical field [0001] The invention relates to the technical field of computer vision content automatic understanding and multimedia retrieval, in particular to a generation method from structured text to image description. Background technique [0002] In the fields of computer vision and multimedia, it is a very important and challenging task to describe the semantic information of images by generating natural language. For example: when people see a picture, especially the objects in the picture have distinctive features or attributes, they will have a certain degree of understanding of the picture, and can use language to tell what happened in the picture. For example, using a sentence like "a yellow school bus" to describe the image, especially "yellow" and "school bus", can describe the attributes of the car in detail. However, in the face of a large number of images, it takes a lot of time, manpower and financial resources to manually describe the images one by one....

AI Technical Summary

Problems solved by technology

However, this method has certain limitations. For example, the single template form of the language leads to a relatively fixed sentence structure, and it takes a lot of time to train image features in order to realize the recognition and pre-processing of objects and actions in the image. Annotation of each object and action category in the image

Most importantly, this method ignores the inherent properties of objects, making the generated sentences lose a lot of information

Images