Multi-modal knowledge graph construction method

Abstract

The invention discloses a multi-modal knowledge graph construction method, and relates to the knowledge engineering technology in the field of big data. The method is realized through the following technical scheme: firstly, extracting multi-modal data semantic features based on a multi-modal data feature representation model, constructing a pre-training model-based data feature extraction model for texts, images, audios, videos and the like, and respectively finishing single-modal data semantic feature extraction; secondly, projecting different types of data into the same vector space for representation on the basis of unsupervised graph, attribute graph, heterogeneous graph embedding and other modes, so as to realize cross-modal multi-modal knowledge representation; on the basis of the above work, two maps needing to be fused and aligned are converted into vector representation forms respectively, then based on the obtained multi-modal knowledge representation, the mapping relation of entity pairs between knowledge maps is learned according to priori alignment data, multi-modal knowledge fusion disambiguation is completed, decoding and mapping to corresponding nodes in the knowledge maps are completed, and a fused new atlas, entities and attributes thereof are generated.

Description

technical field [0001] The invention relates to knowledge engineering technology in the field of artificial intelligence, in particular to a method for constructing a multimodal knowledge map. Background technique [0002] With the continuous development of artificial intelligence technology, knowledge graph, as the pillar of knowledge in the field of artificial intelligence, has attracted extensive attention from academia and industry for its powerful knowledge representation and reasoning capabilities. Traditional hand-designed image features are cumbersome. With the development of network and technology, traditional recognition methods can no longer meet people's needs when dealing with massive images generated under the background of big data. In recent years, knowledge graphs have been widely used in fields such as semantic search, question answering, and knowledge management. As a means of knowledge representation and storage, knowledge graph is considered to solve th...

AI Technical Summary

Problems solved by technology

The multimodal map formed in this way has the following problems: the dependence and correspondence between different modal features are not considered at the source, so that the final fusion result cannot well describe the various associations contained in the multimodal data itself.

Images