Distributed efficient parallel loading method capable of keeping janus Graph data consistency

Abstract

The invention discloses an efficient parallel loading method capable of keeping janus Graph real-time data consistency, and belongs to the field of distributed graph databases. The method comprises the steps of firstly, constructing a distributed structure; creating two empty distributed queues; then, receiving and analyzing the data in real time, and storing the data into a queue I; the data processing module takes out and calls data with corresponding IDs in the distributed index module one by one and loads the data into a graph database, unique marks of points where the IDs cannot be called are stored in a second queue, the point processing module judges whether the IDs corresponding to the marks can be obtained or not, If yes, the point processing module continues to obtain the next mark for judgment; otherwise, loading each mark into the graph database, and generating a corresponding ID; meanwhile, the corresponding relation between the S and the ID is stored; and the cluster management module finds the main node and distributes tasks to the sub-nodes, and the sub-nodes process data in respective distributed queue partitions in parallel. According to the invention, the parallel loading of real-time data is improved while the data consistency is ensured.

Description

technical field [0001] The invention belongs to the field of distributed graph databases, in particular to a distributed and efficient parallel loading method capable of maintaining the consistency of janusGraph data. Background technique [0002] With the continuous development of computer technology and the continuous improvement of informatization, the amount of data is increasing rapidly, and the data structure is gradually becoming more complex. Traditional relational databases are difficult to use in many scenarios, so various non-relational databases have been born. . [0003] Graph database is a kind of non-relational database, which is good at storing various relational network data. Among many graph databases, janusGraph is a very good distributed graph database with high scalability. By expanding the size of the cluster, it can linearly Increase the upper limit of graph storage to support the storage and retrieval of super large graphs. [0004] In many scenario...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the inefficiency and insecurity of the graph database when processing large-flow real-time data, the present invention provides a distributed and efficient parallel loading method that can maintain the consistency of janusGraph data. By increasing the number of nodes, the loading speed can be increased nearly linearly. Greatly improve data loading efficiency while maintaining data consistency

Images