Image description automatic generation method based on relation constraint self-attention

Abstract

The technical scheme adopted by the invention is an image description automatic generation method based on relation constraint self-attention, relates to the three fields of natural image processing, computer vision and natural language processing, and has the following characteristics: 1) a self-attention mechanism (RCSA) based on visual semantic relation constraint is designed; constraint self-attention better focuses on areas related to description generation, irrelevant areas are ignored, and therefore the accuracy of image description generation is improved. 2) the RCSA comprises two sub-modules, and the RCSA-E and the RCSA-D respectively act on the encoding stage and the decoding stage of the image description model; the RCSA-E uses a visual relationship to make the self-attention weight more sparse; the RCSA-D embeds prior semantic relationship information into input high-level context features to enhance semantic expression in a decoding stage. and 3) sufficient experiments are performed on the off-line and on-line evaluation methods, and experimental results show the effectiveness of the proposed methods.

Description

technical field [0001] The invention relates to three fields of natural image processing, computer vision and natural language processing, aiming at the automatic generation target of natural image description, and designs an image description automatic generation method based on relation constraint self-attention. Background technique [0002] Image captioning, which aims to automatically generate natural descriptions for images, is an interdisciplinary task that combines computer vision and natural language processing. It requires the model not only to understand the objects, scenes and their interactions in the image, but also to generate natural language sequences. The research and development of image description depends on the progress of computer vision and natural language processing technology, and it also helps to promote the development of computer vision, natural language processing and other related fields, and also helps to promote the realization of artificial...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the problem that the image description effect is difficult to further improve due to the redundant noise and lack of a priori of the self-attention mechanism in the image description task, the present invention discloses a self-attention model (Relation Constraint Self -Attention, RCSA), which can introduce prior relationship information into self-attention to limit the distribution of self-attention, thereby improving the relationship learning ability of self-attention

Images