Attention mechanism-based image description generation method

Abstract

The invention provides an attention mechanism-based image description generation method. Firstly, more accurate image description is generated by using an attention mechanism according to significantobject information in an image and information of a relationship between objects, then, finer-grained image description is generated by using a double-layer language generation model, and finally, thewhole model is optimized by using reinforcement learning. The method has the advantages that image information can be enriched through fusion of the relation features and the object features, the double-layer language model can generate image description with finer granularity, and the problem of exposure deviation can be relieved by further optimizing the training model through reinforcement learning.

Description

technical field [0001] The invention belongs to the fields of computer vision and natural language processing, and relates to an image language description generation method, in particular to an attention mechanism-based image description generation method. Background technique [0002] In many situations in life, it is necessary to convert image content into text descriptions, such as automatically generating text summaries of images in social software when the network status is poor, and helping visually impaired people understand image content. Existing image description methods are mainly based on deep learning methods, which use convolutional neural networks as image processing models to extract image features, and input image features into recurrent neural networks as language generation models to generate image description languages. However, the model usually uses global or object-level image features. Using such features is difficult to focus on the salient target o...

AI Technical Summary

Problems solved by technology

However, the model usually uses global or object-level image features. Using such features is difficult to focus on the salient target objects in the image and will lose a lot of important information in the image. It is difficult to fully apply the important visual semantic relationship information in the image. into the model

And the existing model is mostly a one-step forward process. When the model generates the next word, it can only use the words that have been generated before, so if a wrong word is generated during the generation process, it will cause an error later. cumulative

On the other hand, the existing model maximizes the joint probability of the sequence ge

Images