Image description method based on space-time memory attention

Abstract

The invention discloses an image description method based on spatio-temporal memory attention. The method comprises the steps that (1) acquiring and pro-processing an MS COCO image description data set; (2) constructing an encoder model, pre-training the encoder model, and completing the encoding of MS COCO image data I to obtain an image feature V; (3) constructing a decoder, and decoding the image feature V; and (4) model training. According to the model built by the method, gate control and memory in the long-short-term memory network are adopted in an original attention model. Compared with a traditional attention model, a memory matrix is newly added to the space-time memory attention model and used for dynamically storing past attention features, continuous self-updating is conductedunder the control effect of an input gate, an output gate and a forgetting gate, and finally relevant attention features in time sequence space are output. Based on the STMA model, the method is moreaccurate in image attention position, and the image description result is more accurate.

Description

technical field [0001] The invention belongs to the interdisciplinary research field of computer vision and natural language processing. Specifically, the invention designs an image description method based on spatio-temporal memory attention. Background technique [0002] Image description, in short, is to describe the main content of the image in one sentence, which requires the machine to have the ability to understand the content of the image and the ability to express content similar to humans. Image description is a difficult problem connecting the two research fields of computer vision and natural language processing. It is not easy to determine the existence, attributes and relationship of objects in the image. Using appropriate sentences to describe this information makes this task more difficult. How to accurately describe the content of images with fluent sentences is the research goal in the field of image description. The research and development of image descr...

AI Technical Summary

Problems solved by technology

The model structure based on the attention mechanism is called the attention model. The introduction of this model allows the decoder to have the ability to focus on part of the coding features, that is, to focus on the local area in the original image, which has promoted the great progress of the image description method, but currently There are also problems in the introduced attention model

[0005] It is generally believed that language description is a temporal representation, but the attention model combined with the current image description method only realizes the extraction of image space features, ignoring the temporal nature of language expression, that is, the attention obtained by the attention model at each moment. The force features are independent of each other in time series, which is different from the process of human observation of things, which directly results in the inaccurate acquisition of attention features, which hinders the further improvement of the image description effect.

Images