Image description model training method and device

Abstract

The invention discloses a training method for an image description model. The method comprises the following steps: receiving a plurality of training images, and extracting an interested region feature vector, a category feature word vector and an image entity feature vector corresponding to each training image; creating an image description model, wherein the image description model comprises anencoding device comprising a plurality of layers of encoding modules, a decoding device comprising a plurality of layers of decoding modules, a self-attention mechanism feature fusion layer and a multi-dimensional convolution kernel feature extractor; the encoding module comprises a multi-dimensional convolution kernel feature extractor, two Self Attention feature extractors and a simple feedforward network; wherein the decoding module comprises a multi-dimensional convolution kernel feature extractor, a multi-head action feature extractor with a mask, two multi-head action feature extractorsand a simple feed-forward network, and the multi-head action feature extractor and the multi-head action feature extractors are connected with the simple feed-forward network. The encoding devices areconnected through a self-attention mechanism feature fusion layer; performing cross entropy loss and reinforcement learning training on the image description model based on the plurality of trainingimages; therefore, a training method is provided.

Description

technical field [0001] The invention relates to the technical field of image description, in particular to a training method and device for an image description model. Background technique [0002] The main purpose of Image Caption is to generate a natural language description for an image, and then through this natural language description, it can help applications understand the semantics expressed in the image visual scene. For example, image description can transform image retrieval into text retrieval, which can be used to classify images and improve image retrieval results. [0003] Early image description methods can be summarized as: extracting objects and attributes from images, and then filling the obtained objects and attributes into predefined sentence templates. With the popularity of deep learning, modern image description methods mainly adopt encoder-decoder architecture, in which Convolutional Neural Network (CNN) is usually used as an encoder for feature ex...

AI Technical Summary

Problems solved by technology

[0004] In the state-of-the-art, encoder-decoder image captioning models usually generate image captions based on global features extracted from the image, even though attention mechanisms are combined with encoder-decoder architectures to extract regions of interest from global features. feature to focus on the region of interest in the image, a large amount of detailed information in the visual scene of the image is still lost in the generation process

Images