Automatic image description method and system based on mixed attention mechanism

Abstract

The invention provides an automatic image description method and system based on a mixed attention mechanism. The method comprises the following steps: acquiring regional image features and position information of a target bounding box in a to-be-described image; inputting the regional image features into a machine attention module to obtain machine attention features; acquiring cognitive data when the human beings perform the image description task, and constructing a visual cognitive model when the human beings perform the image description task according to the cognitive data; and obtaining attention features according to the visual cognition model, and performing fusion according to the attention features to obtain final image description. According to the automatic image description method, the attention guided by a human cognitive mechanism is combined with the attention of a traditional machine, and better reference is provided for the attention weight in the description generation process, so that more accurate description is generated, the performance of the automatic image description method is improved, and a more excellent result is obtained.

Description

technical field [0001] The invention relates to the technical field of computer images, in particular to an automatic image description method and system based on a mixed attention mechanism. Background technique [0002] In the computer field, image description generation is a comprehensive problem that integrates computer vision and natural language processing. Although the image description task is very easy for humans, it is very difficult for machines to understand the content of images and describe them in natural language due to the heterogeneous nature of data from different modalities. Not only are machines required to generate fluent and human-understandable sentences, but sentences are also required to represent complete image content. [0003] Inspired by the application of attention mechanism in machine translation, some researchers have introduced attention mechanism in the traditional "encode-decode" framework, which significantly improves the performance of ...

AI Technical Summary

Problems solved by technology

[0004] However, although the attention mechanism can effectively improve the performance of automatic image description methods, the current methods still have problems such as insufficient attention, which leads to the description of objects that do not appear in the image when performing image description.

Images