Image description optimization method based on pointer network

Abstract

The invention provides an image description optimization method based on a pointer network, and the method comprises the steps: (1) extracting a visual feature vector of an input image through a convolutional neural network, and inputting the visual feature into a description statement generation module which consists of two layers of long-term and short-term memory networks; and (2) selecting a picture area concerned at each moment by utilizing an attention mechanism, and carrying out weighted fusion on image features based on the areas; and (3) selecting the most appropriate word from pre-detected picture attribute words by combining the selected regional features and the generation condition of the current sentence and utilizing a pointer network operation mechanism, filling the word generation position at the moment with the most appropriate word, and if the appropriate word cannot be found from alternative attribute words, determining that the model automatically judges which wordis generated at the moment based on the generation condition of the current sentence; and (4) repeating the above steps, sequentially generating each word in the sentence, and finally obtaining a description sentence closer to the picture content.

Description

technical field [0001] The invention relates to an image description optimization method based on a pointer network. Background technique [0002] Image Caption image description aims to use a machine to generate a fluent, fluent and appropriate description sentence for a picture. This field has attracted a large number of researchers in recent years. Because it involves both computer vision and natural language processing, there is a lot of room for optimization and improvement, and various approaches are also meaningful and representative. Among them, the improvement based on the attention mechanism has played an important role in improving the performance of Image Caption. Its starting point is to imitate the process of human observation and understanding of the content of a picture. The original Soft / Hard Attention algorithm calculates the attention weights assigned to different regions of the picture at each moment, realizing a dynamic shift of attention. Since then, ...

AI Technical Summary

Problems solved by technology

[0003] In addition to the Attention mechanism, another major improvement method for Image Caption is to use pre-trained attribute word detectors, scene classifiers, target detectors, etc. to extract visual elements in pictures in advance, so as to decouple the problem and let Caption generate The model only needs to focus on improving the quality of the sentence, and the task of capturing the visual information of the picture is handed over to the sub-modules. Although the target detection method has a better detection effect, it has the disadvantages of a large model structure and a large training cost. , and the detection of visual attribute words is often converted into a multi-label image recognition (Multi-label ImageClassification) task, which is relatively easy to implement, but the accuracy is average, and it is generally trained using Fully Convolutional Networks (FCN)

Images