Knowledge graph distributed mass data importing method based on load balancing

Abstract

The invention relates to a knowledge graph distributed mass data import method based on load balancing, and belongs to the technical field of knowledge graph data import, the method comprises the following steps: S1, building a Spark distributed computing cluster; s2, solving a jar packet dependency conflict and a version conflict between the Janusgraph and the Spark, and carrying out a connection test by using a SparkGraphCompute interface; s3, investigating the format of an input file which is processed by the Spark in batch graph data import operation, and generating data according to the file format; s4, adjusting resource distribution of worker nodes of the Spark cluster according to the data import integrity and the data import speed, and optimizing the import speed; and S5, carrying out segmentation and load balancing on the super nodes in the data, and accelerating the importing of the super node data. According to the method provided by the invention, the data import efficiency can be greatly improved through distributed calculation, parallelization of super node data import is realized through a load balancing method based on node segmentation, and finally efficient import of the super node data is achieved.

Description

technical field [0001] The invention belongs to the technical field of knowledge graph data import, and in particular relates to a method for importing distributed mass data of knowledge graphs based on load balancing. Background technique [0002] A graph database (GDB) is a database that uses a graph structure for semantic queries, and it uses nodes, edges, and attributes to represent and store data. The key concept of the system is the graph, which directly associates data items in storage with collections of data nodes and edges representing relationships between nodes. A graph database is a non-relational database to address the limitations of existing relational databases. Especially for relational databases, complex multi-table joint query operations are required in most application scenarios (for example, searching for friends of friends of users). A large number of multi-table joint query operations, when there are many records in the table, the calculation amount...

AI Technical Summary

Problems solved by technology

[0008] For the first Janusgraph data import solution, there will be a problem of extremely low efficiency. It may even take several days to import billion-level data, which cannot meet the demand for massive data blowout in many application scenarios; for the second data import solution , Embedded Janusgraph is to open an embedded JanusGraph graph instance from a JVM-based user application

In this case, JanusGraph is part of the user application. Once the application is closed, the graph instance will also be deleted, and data persistence cannot be achieved. For the third data import solution, it is mainly for the Janusgraph storage backend Cassandra to write in batches. However, the storage backend supported by Janusgraph also supports HBase, BerkleyDB, etc. in addition to Cassandra. HBase, as the top project of Apache, provides high reliability, high performance, column storage, and scalability for massive data storage. The basis of scaling and real-time reading and writing has a wide range of applications, but the third solution cannot support all storage backends such as HBase

And the above three schemes do not take into account that the super node cannot be split and it is difficult to parallelize, thereby reducing the impact of data import performance

Images