Multi-modal knowledge graph construction method

A knowledge map and construction method technology, applied in knowledge expression, semantic tool creation, unstructured text data retrieval, etc., can solve the problem of not considering the dependence and correspondence of different modal features, and not being able to describe multimodal data well Various related issues, to achieve the effect of improving knowledge credibility and usability, rich knowledge types, and high knowledge credibility

Pending Publication Date: 2021-01-08
10TH RES INST OF CETC
View PDF0 Cites 56 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The multimodal map formed in this way has the following problems: the dependence and correspondence between different modal features are not c

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-modal knowledge graph construction method
  • Multi-modal knowledge graph construction method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0017] refer to figure 1 . According to the present invention, adopt following steps:

[0018] Multi-modal data semantic feature extraction: extract multi-modal data semantic features based on multi-modal data feature representation model, construct text, image, audio and video data feature extraction models based on pre-trained models, and complete single-mode data based on feature extraction models Semantic feature extraction of dynamic data, semantic feature extraction of text data, image feature extraction, video feature extraction, textual description information extraction and textual description of image data, textual description information extraction of video;

[0019] Multimodal knowledge representation: Based on unsupervised graph embedding, attribute graph embedding, heterogeneous graph embedding, etc., different types of data are projected into the same vector space for representation, realizing cross-modal multimodal knowledge representation;

[0020] Multi-mod...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a multi-modal knowledge graph construction method, and relates to the knowledge engineering technology in the field of big data. The method is realized through the following technical scheme: firstly, extracting multi-modal data semantic features based on a multi-modal data feature representation model, constructing a pre-training model-based data feature extraction model for texts, images, audios, videos and the like, and respectively finishing single-modal data semantic feature extraction; secondly, projecting different types of data into the same vector space for representation on the basis of unsupervised graph, attribute graph, heterogeneous graph embedding and other modes, so as to realize cross-modal multi-modal knowledge representation; on the basis of the above work, two maps needing to be fused and aligned are converted into vector representation forms respectively, then based on the obtained multi-modal knowledge representation, the mapping relation of entity pairs between knowledge maps is learned according to priori alignment data, multi-modal knowledge fusion disambiguation is completed, decoding and mapping to corresponding nodes in the knowledge maps are completed, and a fused new atlas, entities and attributes thereof are generated.

Description

technical field [0001] The invention relates to knowledge engineering technology in the field of artificial intelligence, in particular to a method for constructing a multimodal knowledge map. Background technique [0002] With the continuous development of artificial intelligence technology, knowledge graph, as the pillar of knowledge in the field of artificial intelligence, has attracted extensive attention from academia and industry for its powerful knowledge representation and reasoning capabilities. Traditional hand-designed image features are cumbersome. With the development of network and technology, traditional recognition methods can no longer meet people's needs when dealing with massive images generated under the background of big data. In recent years, knowledge graphs have been widely used in fields such as semantic search, question answering, and knowledge management. As a means of knowledge representation and storage, knowledge graph is considered to solve th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06N5/02G06F16/36
CPCG06N5/022G06N5/027G06F16/367
Inventor 代翔崔莹李春豹杨露黄刘刘鑫潘磊
Owner 10TH RES INST OF CETC
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap