A visual relationship detection method and device based on high-order semantic structure of scene graph

Abstract

The present invention proposes a visual relationship detection method and device based on the high-order semantic structure of the scene graph. The algorithm includes predicting the category and position of all objects in the picture, outputting the visual feature vector corresponding to each object, and analyzing all detected objects. Each pairing operation is performed for each two, based on the pairing result, the joint visual feature vector is extracted, and the position is encoded to obtain the position encoding; the categories of all objects are input into the hierarchical semantic clustering algorithm, and the high-level semantics corresponding to each object are obtained after processing feature vectors; carry out semantic encoding to the output of the hierarchical semantic clustering algorithm; generate relational classifier weights; combine the visual feature vectors, the joint visual feature vectors and the position codes into a unified feature vector, using the The weight of the relationship classifier performs a dot product operation on the unified feature vector, and finally obtains the conditional probability of the relationship between each two objects as a scene graph.

Description

technical field [0001] The invention relates to the field of image processing, in particular to a visual relationship detection method and device based on a high-order semantic structure of a scene graph. Background technique [0002] The main goal of the visual relation detection task is to identify and localize the content of the visual triadic relation (subject, relation, object) present in the image. Recognition refers to identifying the category attributes of the target object, and localization is to return the bounding box of the target object. Understanding a visual scene is often more than recognizing individual objects, and even a perfect object detector would struggle to perceive the subtle differences between a person feeding a horse and a person standing next to it. Learning the rich semantic relations between these objects is what visual relation detection is all about. The key to a deeper understanding of the visual scene is to build a structured representati...

AI Technical Summary

Problems solved by technology

[0007] In order to solve the technical problem that it is difficult to obtain the correct type and quantity of annotations and complete triplet annotations, the present invention proposes a visual relationship detection method and device based on the high-order semantic structure of the scene graph

Images