Method for semantic query of large-scale knowledge graph based on spark

Abstract

The invention discloses a method for semantic query of a large-scale knowledge graph based on spark. The method comprises the steps that (1) entities and relations in each triple are replaced by corresponding id; (2) a layered sub-graph index is established based on categories and relations and the index is stored in a hdfs document; (3) operations related to sparql query are translated by an spark operation primitive language; (4) different score functions are distributed according to characteristics of each triple mode, and an execution sequence of each triple mode in the sparql query is determined; and (5) according to the execution sequences of the triple modes and the spark operation primitive language, query and linkage are executed, and then a linkage result is analyzed by a mapping list and then returned. According to the invention, efficient query of massive semantic data is supported; extendibility is very high; and the method has very high practical values for query and application based on large-scale semantic data.

Description

technical field [0001] The present invention relates to computer storage query technology, in particular to a large-scale knowledge map semantic query method based on spark. Background technique [0002] With the rapid development of the Semantic Web, the amount of semantic data has shown explosive growth, and a large amount of semantic data in RDF format has been published by researchers in academia and industry. For example, Google's knowledge map has more than 600 million entities and 20 billion facts (2012), and the wikidata project also contains more than 20 million pages, each page contains a large number of triplet facts, YAGO and DBPedia also Contains more than 100 million records, and the Linked OpenData (LOD) project has released more than 2,700 data sources, containing a total of more than 130 billion RDF triples (2016). How to effectively store queries on large-scale RDF knowledge graphs is a thorny problem faced by many researchers, and this has always been con...

AI Technical Summary

Problems solved by technology

Traditional RDF-based storage query engines improve query performance by building subject-predicate-object permutation indexes, but at the same time, this also consumes a large amount of storage space, and most of these engines are based on stand-alone machines, which are relatively scalable. Poor, only suitable for small-scale RDF data storage query, facing the current massive semantic data, there are problems such as insufficient computing performance and scalability

A small number of existing distributed processing engines such as hadoopRDF partially solve the scalability problem with the help of distributed platforms, but due to the iterative execution characteristics of sparql queries, their performance is still severely constrained, and cannot meet large-scale semantic data queries Real-time requirements, so its practicability is greatly restricted, and a high-performance storage query engine facing massive semantic data is urgently needed to change this dilemma

Images