Image description method

Abstract

The invention discloses an image description method, which is characterized in that a bilinear encoder and a multi-mode decoder are adopted to improve image description with fine-grained region objectfeatures. In the encoder, bilinear pooling is used for encoding fine-grained region image features, a simple encoder of a transformer is used for encoding region-of-interest features of an image, andall the encoded features are fused with a gate structure to serve as overall encoding features of the image. In the decoder, multi-modal features are extracted from the fine-grained region image features and category features and fused with the overall encoding features, and semantic information is decoded to generate a description. Compared with the prior art, the image description method provides a new solution for image description and application of the image description, and is simple, convenient and high in efficiency.

Description

technical field [0001] The invention relates to the field of computer vision, in particular to a method for enriching image description by fusing multi-level Transformer models with fine-grained features. Background technique [0002] Image Caption generates natural language descriptions for images, and uses the generated descriptions to help applications understand the semantics expressed in visual scenes of images. For example, image description can transform image retrieval into text retrieval, which can be used to classify images and improve image retrieval results. People usually only need a quick glance to describe the details of the visual scene of an image, but automatically adding a description to an image is a comprehensive and difficult computer vision task that needs to convert the complex information contained in the image into a natural language description. Compared with ordinary computer vision tasks, image captioning requires not only recognizing objects fr...

AI Technical Summary

Problems solved by technology

Even though the attention mechanism is combined with the encoder-decoder architecture to extract ROI features from global features to focus on image ROIs, a large amount of detailed information in the image's visual scene is still lost during generation

Thus, the encoder-decoder model with attention mechanism faces the following two challenges: 1) When complex objects and attributes are contained in the image, the regional features extracted from the global image feature map cannot well represent the objects Semantics

2) The inherent sequential nature of RNN makes it difficult to perform parallel optimization calculations, resulting in high time costs for model training

Images