An image subtitle generation method based on MLL and ASCA-FR

Abstract

The invention discloses an image subtitle generation method of feature reconstruction ASCA-FR of a joint attention mechanism based on multi-scale learning MLL and adjacent time nodes. The invention mainly solves the problems of inaccurate generated caption description and unsmooth expression caused by the fact that the output of an attention model at a certain moment only considers the feature setof an image and the word vector at the previous moment and only uses a cross entropy loss function to train a network in the prior art. The method comprises the following specific steps: (1) generating a natural image test set and a training set; (2) extracting feature vectors; (3) constructing ASCA-FR network; (4) training ASCA-FR network; (5) obtaining natural image subtitles; according to themethod, the MLL loss function pair is utilized to train the constructed ASCA-FR network, so that the generated subtitles are accurately described and smoothly expressed.

Description

technical field [0001] The invention belongs to the technical field of image processing, and further relates to a feature reconstruction ASCA-FR (Adjacent Step Co- Attention and Feature Reconstruction) image caption generation method. The present invention can extract and process semantic information in any natural image to generate an image caption corresponding to the natural image. Background technique [0002] For a natural image, humans can organize vivid and vivid language in the brain to describe the visual scene information in the image with just a quick glance. With the rapid development of artificial intelligence and deep learning technology, image caption generation, as an important research topic in the field of natural language processing, has attracted more and more attention. The task of image caption generation is to automatically generate captions for any natural image that are closely related to its semantic information. However, due to the complex and d...

AI Technical Summary

Problems solved by technology

The disadvantage of this method is that the visual attention model used in this method only considers the feature set of the image and the generated word information at the previous moment when outputting, and only uses the forward generation process from image to subtitle , so that the accuracy of the subtitle description is low, and it cannot reflect the content of the image well

The disadvantage of this method is that the loss function used to train the network is only based on the cross-entropy loss function of label subtitles, which makes the training process too thin, and the generated subtitles are not smooth and have many grammatical errors

Images