Image description method of self-attention mechanism based on sample adaptive semantic guidance

Abstract

The invention discloses an image description method of a self-attention mechanism based on sample adaptive semantic guidance, and belongs to the technical field of artificial intelligence. To solve the defect that parameters of each sample are fixed in a traditional method adopting a self-attention mechanism, the method comprises the following steps: 1) extracting features corresponding to a plurality of candidate areas of a to-be-described image by a target detector; 2) extracting a plurality of semantic concepts for the to-be-described image by adopting a semantic concept detector; 3) performing feature enhancement on the features extracted in the step 1) and the step 2) through different self-attention networks; 4) generating a parameter of a self-attention network by using the semantic concept feature enhanced in the step 3) and a parameter generation network; 5) inputting the visual features enhanced in the step 3) into the generated self-attention network, and realizing better visual expression through the self-attention network generated semantically; and 6) inputting the visual features output in the step 5) into a decoder, generating a description statement of the image, and defining a loss function.

Description

technical field [0001] The invention relates to automatic image description in the field of artificial intelligence, in particular to an image description method based on a self-attention mechanism guided by sample self-adaptive semantics, which uses natural language to describe the objective content of images. Background technique [0002] Automatic image description (Image Captioning) is an ultimate machine intelligence task proposed by the artificial intelligence community in recent years. Its task is to describe the objective content of the image in natural language for a given image. With the development of computer vision technology, tasks such as target detection, recognition, and segmentation can no longer meet people's production needs, and there is an urgent need for how to automatically and objectively describe image content automatically. Different from tasks such as target detection and semantic segmentation, automatic image description requires an overall and o...

AI Technical Summary

Problems solved by technology

[0016] The purpose of the present invention is to provide a sample-based self-adaptive semantic guidance for the problem that the generalization of the traditional transformer-based image description method decreases when the parameters are fixed in the test phase, and the semantic noise in the current model using semantic information is too large. Image description method with self-attention mechanism

Images