Semantic vision positioning method and device based on multi-modal graph convolutional network

Abstract

The invention provides a semantic vision positioning method and device based on a multi-modal graph convolutional network. The method comprises the steps: obtaining an input picture and corpus description; extracting multi-scale visual features of an input picture by using a convolutional neural network, and encoding and embedding spatial coordinate information to obtain spatial perception visualfeatures; analyzing the corpus description to construct a semantic structure diagram, encoding each node word vector in the semantic structure diagram, and learning diagram node semantic features through a multilayer perceptron; fusing the spatial perception visual features and the graph node semantic features to obtain multi-modal features of each node in the semantic structure graph; spreading relationship information of nodes in the semantic structure chart through a graph convolution network, and learning visual semantic relationships under the guidance of semantic relationships; and performing semantic visual position reasoning to obtain a visual position of the semantic information. According to the method, context semantic information is combined when ambiguous semantic elements areprocessed, and visual positioning can be guided by utilizing semantic relation information.

Description

technical field [0001] The present invention relates to the field of computers, in particular to a semantic visual positioning method and device based on a multimodal graph convolutional network. Background technique [0002] Realizing the communication between humans and machines in the real world, enabling machines to understand visual scenes described by natural language, is a basic but very challenging problem in the field of artificial intelligence. The basis of this problem is to enable the machine to locate semantic elements in the visual scene, that is, given a natural language description of a visual scene, the machine must be able to locate the corresponding semantic element in the visual scene. In recent years, the task of semantic visual localization has received extensive attention and has been developed rapidly, achieving excellent performance. However, existing solutions locate noun phrases one by one in the picture, do not model the semantic relationship bet...

AI Technical Summary

Problems solved by technology

Generally speaking, methods that lack semantic relationship modeling between noun phrases are often difficult to obtain ideal visual positioning when dealing with ambiguous or ambiguous semantic elements. It is necessary to study fine semantic relationship modeling methods for noun phrases so that they can be used in Semantic Visual Localization Guided by Semantic Structure Information

[0003] Existing solutions mainly focus on the fusion of visual features and corpus features, reconstructing the corpus from the corresponding visual regions of the corpus, or roughly combining semantic context information. They lack the exploration of semantic relationship information between noun phrases in the corpus, so they are limited. Semantic elements that need to combine information such as context and semantic relations for visual position reasoning

Images