Image description method based on adaptive enhanced self-attention network

Abstract

The invention relates to the technical field of image description, and discloses an image description method based on an adaptive enhanced self-attention network, which can be used for jointly modeling a relationship on a geometric level and a semantic level, and when a clear geometric or semantic relationship exists between two objects in an image, the two objects in the image can be accurately described. According to the method, the visual relationship in the given image can be adaptively enhanced, and high-precision and credible image description generation is realized.

Description

technical field [0001] The invention relates to the technical field of image description, in particular to an image description method based on an adaptive enhanced self-attention network. Background technique [0002] Image description aims to automatically generate a sentence description for a given image, which can combine vision and language well, and is an important multimodal task. The generated image description should not only identify the objects of interest in the image, but also describe the relationship between the objects. For image captioning, a key challenge is how to accurately and efficiently model the relationship between the identified objects, which is crucial for improving the quality of generation. Recently, geometric information has been widely studied to enhance features at the region level, since geometric features, i.e. relative distances and relative sizes, contain explicit positional relationships between objects. [0003] Visual relations inclu...

AI Technical Summary

Problems solved by technology

[0003] Visual relationships include geometric positions and semantic interactions, indicating the interrelationships between targets in region-level representations, but previous work only uses geometric positions to enhance the representation of visual relationships, and only shallow position information cannot cover action-complex semantics relation

Therefore, the limitation of the state-of-the-art is that it is difficult for image captioning models to generate plausible captions with accurate semantic relationship predictions.

Images