Partitioning and parallel distribution processing method of super-large scale RDF graph data

Abstract

The invention discloses a partitioning and parallel distribution processing method of super-large scale RDF graph data. The method comprises the following steps: preprocessing original RDF graph data, generating a corresponding Hash dictionary file and reshaped three-column list data, and converting the reshaped three-column list data into an associated matrix M; creating a hypergraph model of the associated matrix M, wherein the subject, the predicate and the object of the M in the hypergraph model are hyperedges, and data associated with the hyperedges is hyperedge data; judging whether the RDF graph data is a connected graph or an unconnected graph; if it is the unconnected graph, partitioning the unconnected graph into a plurality of connected graphs; on the basis of the hypergraph model and concurrent breadth traversal, equidistantly placing the hyperedge data on a path, classifying and ranking the hyperedge data, uniformly partitioning the hyperedge data into K portions to be put into K slave nodes; and establishing mapping relationships among the hyperedge data and the slave nodes. The partitioning and parallel distribution processing method of the super-large scale RDF graph data is quick in partitioning speed, high in partitioning quality, balance in data and task loads, high in parallelism of query processing, and fast in query processing speed.

Description

technical field [0001] The invention belongs to the field of big data processing, and more specifically relates to a method for dividing and parallel distribution processing of ultra-large-scale RDF graph data. Background technique [0002] Resource Description Framework (RDF) is the core of the entire Semantic Web system structure, and it is widely used to describe various information resources on the Internet. With the continuous growth of RDF data, the processing on a single machine has become incapable, so the RDF data must be divided into multiple machines for processing. [0003] For the division of ultra-large-scale RDF graph data, commonly used methods include heuristic division and parallel hierarchical division. For the heuristic method, an objective function is generally provided, and then the division is carried out around the optimal direction of this function, but the selection of the objective function is more difficult. For parallel hierarchical division, i...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the above deficiencies or improvement needs of the prior art, the present invention provides a method and system for the division and parallel distribution processing of ultra-large-scale graph data. And equally divide the super-edge data on the path, so as to take into account the uniformity of data distribution and the balance of task load, and through the use of bit-block transmission and pipeline processing methods, it solves the long division time and the division of existing division methods. Problems with low quality and uneven task load

Images