Image description generation method based on external triple and abstract relationship

Abstract

The invention discloses an image description generation method based on an external triple and an abstract relation. The method comprises the following steps: firstly, extracting a triple in an image description text, constructing an external relation library, and carrying out feature coding on the triple; and clustering the triples of which the text similarity is higher than a threshold value into one class. Meanwhile, the model performs target detection on the image to obtain a target visual feature set and a target category set; and querying a triple of which the target is similar to the target category in an external relation library according to the text similarity. The model predicts the target, the attribute and the relation of the image by using the visual features of the target to generate a scene graph; and fusing visual features and text features by using a convolutional neural network, and carrying out feature coding on targets, attributes and relationships. And finally, fusing target, attribute and relation coding features of the scene graph and coding features of the similarity relation and the abstract relation, and inputting the fused features into a double-layer LSTM sequence generation model to obtain final image description. According to the invention, the expression of the model generation description is richer.

Description

technical field [0001] The invention in this paper relates to an image description generation method, specifically an image description generation method based on external triples and abstract relationships, and belongs to the field of image description generation. Background technique [0002] Image caption generation is a comprehensive task combining computer vision and natural language processing, which is extremely challenging. Inspired by codecs, attention mechanisms, and reinforcement learning-based training objectives in the field of natural language processing, modern image description generation models have made amazing progress, and researchers are increasingly paying attention to the field of image description generation. It even surpasses humans in some evaluation metrics. [0003] The technology of image description generation methods continues to develop, but there is a problem that has never been solved but cannot be ignored, that is, the existing model is on...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to address the deficiencies in the prior art and provide an image description generation method based on external triples and abstract relationships to solve the problem that the descriptions that cannot be generated by traditional image description generation methods are too simple, and based on the original improve forecast accuracy

Images