Multi-modal classification method based on graph convolutional neural network

Abstract

The invention discloses a multi-modal classification method based on a graph convolutional neural network. The method comprises the following steps: 1, ; the method comprises the following steps: (1)firstly, a user needs to prepare an object library, each object comprises a mode, a category mark is provided for a small number of objects in the library through a manual marking method, the objectswith the category marks are called initial marked training data, and the objects with the category marks and a large number of remaining unmarked objects form a training data set together; according to the method, graph structure information of different modes is comprehensively considered through the innovative multi-mode graph convolutional neural network, and trainable weights are distributed in each layer of the multi-mode graph convolutional neural network, so that representation learned by each mode can gradually consider structure information of other modes.

Description

technical field [0001] The invention belongs to the technical field of artificial intelligence in computer science and technology, and specifically relates to a multimodal classification method based on graph convolutional neural network. Background technique [0002] In recent years, more and more multi-modal data have appeared in practical applications. For example, multimedia data in the Internet often contains multiple modal information: video, image, and text information appearing around; Status information: the text information of the web page itself and the hyperlink information linked to the web page. These multimodal data contain huge economic value, and using these multimodal data can often obtain better results than single modality data. For example, in user content recommendation based on information flow, different modal information (such as pictures and texts) in the information flow can be considered simultaneously to recommend content of interest to users. ...

AI Technical Summary

Problems solved by technology

On the other hand, the graph convolutional neural network can embed graph structure information into the neural network, and is suitable for processing large-scale data, but it cannot be directly applied to multi-modal scenarios. Objects in practical applications often have multi-modal information, but the traditional multi-modal method only trains the learner separately on multiple modalities and then integrates them. This way is easy to ignore the useful structural information in different modalities. Therefore, we propose a graph-based convolutional neural network The multimodal classification method of

Images