A large-scale linked data partition method and system based on attribute graph

Abstract

The invention discloses a large-scale associated data division method and system based on an attribute graph, belonging to the field of data division, including: (1) constructing an attribute graph according to the associated data to be divided; (2) obtaining the starting point of all paths in the attribute graph (3) Traversing the starting vertex set, taking the traversed starting vertex as the starting point of the path, obtaining the path in the attribute graph that satisfies the constraints between the path attributes and forming a path group, thus in After the traversal, a plurality of path groups are obtained; (4) the associated data is divided in units of path groups to obtain k division blocks; where k is the number of machine nodes in the distributed graph data management system. The invention can reduce cross-partition communication in the process of data query and improve the efficiency of data query.

Description

technical field [0001] The invention belongs to the field of data division, and more specifically relates to a large-scale associated data division method and system based on an attribute graph. Background technique [0002] Due to the rapid development of fields such as social network analysis, machine learning, and data mining, Linked Data is experiencing explosive growth. Linked Data is a specification recommended by the Internet Consortium (W3C) to publish and connect various data, information and knowledge. Due to the increase in the amount of data, the semantic relationship between linked data has also become very complicated. [0003] As the scale of linked data continues to expand, it becomes increasingly difficult to perform storage operations on a single node, and the storage capacity of a single computing node is far lower than the growth of data. At present, the main solution to the problem that a single node cannot handle large data is to divide large-scale da...

AI Technical Summary

Problems solved by technology

Although in the early stage of data partitioning, the method of hash partitioning can obtain balanced partitioning very quickly, but the operation of graph data in the later stage will become very time-consuming

Although the hash partition method can indeed distribute the data relatively evenly among the partitions, the hash partition method does not take into account the structure of the graph between the data, so it is very likely that the data with a high degree of semantic correlation or close association will not be Dividing the data into the same partition not only invalidates the semantic information between the data, but also must perform a costly distributed join (join) operation in the later query to obtain the final output by merging the intermediate results in the query process. Involves a lot of cross-partition communication in

Data operations in the late stage of the hash division method will cause very high costs, and operations involving parallel operations will cause very huge communication overhead

Images