An image description generation system and method based on a weighing attention mechanism

Abstract

The invention relates to the field of image understanding, discloses an image description generation system and method based on a weighing attention mechanism, and solves the problems that an existingimage description scheme lacks a polishing process, the training process and the testing process are inconsistent, and the generation description recognition degree is not high. The method comprisesthe following steps: a, processing a data set: extracting global features and local features of an image, constructing the data set, marking words in the data set, and generating corresponding word embedding vectors; B, training an image description generation model: generating rough image description by adopting a first layer of decoder based on a residual attention mechanism, and carrying out polishing on the generated image description by adopting a second layer of decoder based on the residual attention mechanism; And c, further training the model in combination with reinforcement learning: simulating a test process of the model in the training process, guiding the training of the model by generating a described CIDEr score, and adjusting the model in combination with reinforcement learning.

Description

technical field [0001] The invention relates to the field of image understanding, in particular to an image description generation system and method based on deliberation and attention mechanism. Background technique [0002] The task of image description is: given a picture, automatically generate the corresponding natural language description. The generated sentences are required to be fluent and can describe the objects and scenes in the pictures. This research direction can be applied in many ways. For example: helping blind people understand the content of pictures. [0003] Traditional image description models generally adopt an encoder-decoder framework combined with an attention mechanism. The framework has achieved great results. But it still has the following defects: [0004] First, the training and testing process of the traditional model is to generate a description as the final result through a decoder. This method lacks the polishing process, so the resu...

AI Technical Summary

Problems solved by technology

[0007] The technical problem to be solved by the present invention is to provide an image description generation system and method based on deliberation and attention mechanism, which solves the problems of lack of polishing process, inconsistent training and testing process, and low recognition degree of generated description existing in existing image description schemes. question

Images