Distributed graph data partitioning and copying method based on load driving

Abstract

The invention provides a distributed graph data partitioning and copying method based on workload driving. The method comprises the following steps of: respectively storing data in a graph data set into a server cluster through a hash table method; constructing an active vertex set in a vertex data set by combining the workload information of a user and the load judgment of a source server, then scoring each active vertex by combining a server cluster according to a scoring function, constructing a target server scoring list, and determining a target migration server through the load judgment of the target server; and combining the active vertex set with the corresponding target migration server, judging whether the active vertexes are high-reading vertexes or not through a threshold value, and if the active vertexes are not the high-reading vertexes, further dynamically adjusting copy data of the active vertexes through the maximum replication factor. According to the method, the position of the graph vertex data is dynamically adjusted by utilizing the characteristic of dynamic change of the workload, so that the query efficiency is improved, and low delay and high throughput of query are ensured.

Description

technical field [0001] The invention belongs to the field of computers, and in particular relates to a load-driven distributed graph data division and replication method. Background technique [0002] In recent years, with the continuous expansion of social network and Wanwei graph data, and the continuous expansion of multi-user access requirements, it is difficult for a common single database server to meet multi-user high-quality access services under existing resources. The traditional solution is to use vertical expansion and complete replication of data, and the resulting high cost is unsatisfactory, so a distributed data storage method emerges as the times require. Distributed data storage is based on cheap server clusters for horizontal partition expansion and partial replication backup, and provides concurrent graph data processing. In the face of distributed parallel data storage, how to reasonably store graph data in different nodes, so that users can improve the...

AI Technical Summary

Problems solved by technology

[0005] Another problem is the disaster recovery of distributed graph data. In order to prevent data loss due to system operation errors or system failures, we need to perform data backup, that is, data replication

The traditional solution is to perform a full backup of all data according to a fixed replication factor, resulting in expensive network and storage overhead

Images