Attention mechanism-based image description generation method

Abstract

The invention provides an attention mechanism-based image description generation method. Firstly, more accurate image description is generated by using an attention mechanism according to significantobject information in an image and information of a relationship between objects, then, finer-grained image description is generated by using a double-layer language generation model, and finally, thewhole model is optimized by using reinforcement learning. The method has the advantages that image information can be enriched through fusion of the relation features and the object features, the double-layer language model can generate image description with finer granularity, and the problem of exposure deviation can be relieved by further optimizing the training model through reinforcement learning.

Description

technical field [0001] The invention belongs to the fields of computer vision and natural language processing, and relates to an image language description generation method, in particular to an attention mechanism-based image description generation method. Background technique [0002] In many situations in life, it is necessary to convert image content into text descriptions, such as automatically generating text summaries of images in social software when the network status is poor, and helping visually impaired people understand image content. Existing image description methods are mainly based on deep learning methods, which use convolutional neural networks as image processing models to extract image features, and input image features into recurrent neural networks as language generation models to generate image description languages. However, the model usually uses global or object-level image features. Using such features is difficult to focus on the salient target o...

AI Technical Summary

Problems solved by technology

However, the model usually uses global or object-level image features. Using such features is difficult to focus on the salient target objects in the image and will lose a lot of important information in the image. It is difficult to fully apply the important visual semantic relationship information in the image. into the model

And the existing model is mostly a one-step forward process. When the model generates the next word, it can only use the words that have been generated before, so if a wrong word is generated during the generation process, it will cause an error later. cumulative

On the other hand, the existing model maximizes the joint probability of the sequence generated by the model during training, so that the cross-entropy loss is minimized to train the model, and the joint probability of the generated reference words is maximized through back propagation, so that the model can learn What we get is the probability distribution of the words in the sentence, which is different from the automatic evaluation indicators usually used when judging the quality of the sentences generated by the image description model. These evaluation indicators are not differentiable and cannot be used directly as a loss function. This loss function is the same as The inconsistency of evaluation indicators prevents the model from being fully optimized

Images