Large-scale associated data division method and system based on attribute graph

Abstract

The invention discloses a large-scale associated data division method and system based on an attribute graph, and belongs to the field of data division. The method comprises the steps of (1) building the attribute graph according to the to-be-divided associated data; (2) obtaining the initial vertexes of all paths in the attribute graph so as to obtain an initial vertex set; (3) traversing the initial vertex set, taking the traversed initial vertex as a path starting point, obtaining the paths meeting the constraints between path attributes in the attribute graph, and forming a path group, thereby obtaining a plurality of path groups after the traversing is finished; (4) dividing the associated data by taking the path group as a unit so as to obtain k divided blocks, wherein k is the number of machine nodes in the distributed graph data management system. According to the present invention, the cross-partition communication during the data query process can be reduced, and the data query efficiency is improved.

Description

technical field [0001] The invention belongs to the field of data division, and more specifically relates to a large-scale associated data division method and system based on an attribute graph. Background technique [0002] Due to the rapid development of fields such as social network analysis, machine learning, and data mining, Linked Data is experiencing explosive growth. Linked Data is a specification recommended by the Internet Consortium (W3C) to publish and connect various data, information and knowledge. Due to the increase in the amount of data, the semantic relationship between linked data has also become very complicated. [0003] As the scale of linked data continues to expand, it becomes increasingly difficult to perform storage operations on a single node, and the storage capacity of a single computing node is far lower than the growth of data. At present, the main solution to the problem that a single node cannot handle large data is to divide large-scale da...

AI Technical Summary

Problems solved by technology

Although in the early stage of data partitioning, the method of hash partitioning can obtain balanced partitioning very quickly, but the operation of graph data in the later stage will become very time-consuming

Although the hash partition method can indeed distribute the data relatively evenly among the partitions, the hash partition method does not take into account the structure of the graph between the data, so it is very likely that the data with a high degree of semantic correlation or close association will not be Dividing the data into the same partition not only invalidates the semantic information between the data, but also must perform a costly distributed join (join) operation in the later query to obtain the final output by merging the intermediate results in the query process. Involves a lot of cross-partition communication in

Data operations in the late stage of the hash division method will cause very high costs, and operations involving parallel operations will cause very huge communication overhead

Images