Method and device for generating scene graph

Abstract

Disclosed is a generation method and generation device of a scene graph, the generation method comprising: acquiring image data and text data; obtaining a feature vector of the image data through a Faster R-CNN target detector, and by analyzing the feature vector Fusion is carried out to obtain the visual feature vector; based on the pre-trained fastText model, the word vector of the text data is obtained, and the word vector is fused to obtain the semantic feature vector; by the visual feature vector and the semantic The feature vectors are matched to obtain the predicted value of the visual relationship, and the predicted value of the visual relationship is learned symmetrically by using the cross-entropy function and the reverse cross-entropy function to obtain the final predicted value of the visual relationship, thereby generating a scene graph. This generative method can effectively reduce the label noise problem of human-annotated datasets.

Description

technical field [0001] The present disclosure generally relates to the field of computer vision, and more specifically, relates to a method and device for generating a scene graph based on symmetric learning. Background technique [0002] The rapid development of the field of computer vision has made many breakthroughs in visual tasks such as image classification, semantic segmentation and visual relationship detection in a short period of time. Network (Region-based CNN, R-CNN) and fully convolutional network (Fully Convolutional Network, FCN), driven. [0003] On this basis, the research on image understanding has gradually developed from low-level feature extraction to high-level semantic learning. The next step is to infer the semantic relationship between multiple objects, thereby promoting the development of multimodal tasks, such as visual question answering, image description and visual commonsense reasoning. Among them, the emergence of Scene Graph Generation (SGG...

AI Technical Summary

Problems solved by technology

However, in practical engineering, the cost of collecting and annotating large-scale datasets is high, and manual annotation is error-prone, and even high-quality datasets may contain noisy labels

At the same time, the visual visual relationship in the scene graph has a long-tail effect, that is, most existing scene graph models are good at fitting predicates with high frequency in the dataset, but poor in learning visual relationships with fewer labeled instances.

Specifically, the existing scene graph generation methods have stable performance for frequently occurring predicate categories, but are not ideal for noisy labels in the dataset and difficult-to-learn relationship categories.

Although the existing research enhances the visual relationship detection ability of the model by improving the data set, such as using the generated missing labels to train the scene graph and alleviating the semantic ambiguity in visual relationship detection through probabilistic modeling, none of them considers the large-scale process. The problem of noisy labels is common in human-annotated datasets, and the visual relationship detection effect for datasets containing noisy labels needs to be improved

Images