Scene graph generation method and device

Abstract

The invention discloses a scene graph generation method and device. The generation method comprises the following steps of acquiring the image data and the text data; obtaining the feature vectors of the image data through a Faster R-CNN target detector, and fusing the feature vectors to obtain the visual feature vectors; obtaining the word vectors of the text data based on a pre-trained fastText model, and fusing the word vectors to obtain the semantic feature vectors; and matching the visual feature vectors and the semantic feature vectors to obtain a visual relationship prediction value, and performing symmetric learning on the visual relationship prediction value by using a cross entropy function and a reverse cross entropy function to obtain a final visual relationship prediction value, thereby generating a scene graph. According to the generation method, the problem of label noise of a data set subjected to manual annotation can be effectively reduced.

Description

technical field [0001] The present disclosure generally relates to the field of computer vision, and more specifically, relates to a method and device for generating a scene graph based on symmetric learning. Background technique [0002] The rapid development of the field of computer vision has made many breakthroughs in visual tasks such as image classification, semantic segmentation and visual relationship detection in a short period of time. Network (Region-based CNN, R-CNN) and fully convolutional network (Fully Convolutional Network, FCN), driven. [0003] On this basis, the research on image understanding has gradually developed from low-level feature extraction to high-level semantic learning. The next step is to infer the semantic relationship between multiple objects, thereby promoting the development of multimodal tasks, such as visual question answering, image description and visual commonsense reasoning. Among them, the emergence of Scene Graph Generation (SGG...

AI Technical Summary

Problems solved by technology

However, in practical engineering, the cost of collecting and annotating large-scale datasets is high, and manual annotation is error-prone, and even high-quality datasets may contain noisy labels

At the same time, the visual visual relationship in the scene graph has a long-tail effect, that is, most existing scene graph models are good at fitting predicates with high frequency in the dataset, but poor in learning visual relationships with fewer labeled instances.

Specifically, the existing scene graph generation methods have stable performance for frequently occurring predicate categories, but are not ideal for noisy labels in the dataset and difficult-to-learn relationship categories.

Although the existing research enhances the visual relationship detection ability of the model by improving the data set, such as using the generated missing labels to train the scene graph and alleviating the semantic ambiguity in visual relationship detection through probabilistic modeling, none of them considers the large-scale process. The problem of noisy labels is common in human-annotated datasets, and the visual relationship detection effect for datasets containing noisy labels needs to be improved

Images