A method for fast batch loading of Janusgraph data

Abstract

The present invention relates to a method for fast batch loading of Janusgraph data, comprising the following steps: a data preparation stage: data of vertices and edges to be batch loaded into a graph are saved in a Hive table for shard storage, so as to facilitate parallel reading by a Spark computing engine, and a Schema structure of Janusgraph graph data to be loaded is created, and a mapping relationship between the Hive table and the Schema structure is configured; an HBase data loading stage: vertices and edges in a Hive table are read in parallel by a Spark computing engine, an RDD data set is constructed according to a storage structure and an encoding method of Janusgraph in an HBase database, and the RDD data set is loaded into an HBase database corresponding to the Janusgraph storage in the form of an HFile file; and an Elasticsearch index construction stage: vertices and edges in a Hive table are read in parallel by a Spark computing engine, attributes of vertices or edges to be indexed are extracted, an RDD data set is constructed according to a storage structure of Janusgraph in an Elasticsearch index, and the RDD data set is written in parallel into an Elasticsearch index.

Description

Technical Field

[0001] The present invention relates to the field of computer science big data technology, and in particular to a method for fast batch loading of distributed graph database Janusgraph data. Background Art

[0002] A graph is a mathematical logical object that represents the relationship between entities. In mathematics, it is represented as a two-tuple G=(V, E), which itself consists of N vertices (V=vertex) and M edges (E=edge). Each vertex corresponds to multiple edges (≤M), and each edge connects two vertices. Edges can have directions. If the edges contained in the graph have directions, it is called a directed graph, otherwise it is an undirected graph.

[0003] Graph database is a type of NoSQL database, a non-relational database that uses graph theory to store relationship information between entities. The most common example is the relationship between people in social networks. Relational databases are not very effective in storing "relational" data. Their queries are complex, slow, and beyond expectations. The unique design of graph databases makes up for this shortcoming.

[0004] Janusgraph is an open source distributed graph database. It is widely used in the field of graph data analysis due to its advantages such as good versatility, high performance, and open source code. Janusgraph supports Cassandra, HBase and other databases as storage to save complete graph structure data; it supports Elasticsearch, Solr and other databases as indexes, and can realize real-time retrieval of indexed vertices and edges. Since HBase, Elasticsearch and other databases are widely used in the field of big data and have excellent performance, this solution is mainly aimed at scenarios where HBase clusters are used as storage and Elasticsearch clusters are used as indexes.

[0005] Based on this scenario, the existing Janusgraph has the following problems when loading data:

[0006] (1) Loading data into the graph through the API interface provided by Janusgraph will be submitted as a transaction, which increases the performance overhead of data loading. However, offline data loading does not require transactions to ensure data consistency.

[0007] (2) The API provided by Janusgraph calls the API of the HBase database to save data to the HBase database. During this period, HBase will frequently perform flush, compact, and split operations, resulting in a large amount of unnecessary resource consumption, thereby reducing the efficiency of storage. Moreover, if the API call speed of the HBase database exceeds the writing capacity of the HBase database, some data may be lost, resulting in the problem of missing graph data.

[0008] (3) The API interface provided by Janusgraph calls the API interface of the HBase database to save data, and then calls the API interface of the Elasticsearch index to build the index after returning successfully. This serial writing method cannot fully utilize the cluster resources of HBase data and Elasticsearch index. Summary of the invention

[0009] The purpose of the present invention is to use HBase cluster as graph data storage and Elasticsearch cluster as index scenario for Janusgraph open source distributed graph database, which is particularly suitable for fast batch loading of ultra-large-scale massive data of more than 100 million. The present invention makes full use of cluster resources to improve the performance of batch loading of data into Janusgraph, solves the problem of slow batch loading of massive data, and solves the problem that part of the data may be lost due to parallel writing of massive data through the API interface of HBase database, resulting in missing graph data, and provides a method for fast batch loading of data in distributed graph database Janusgraph.

[0010] In order to achieve the above-mentioned object of the invention, the embodiment of the present invention provides the following technical solutions:

[0011] A method for quickly loading Janusgraph data in batches comprises the following steps:

[0012] Data preparation stage: Save the data of vertices and edges that need to be loaded into the graph in batches to the Hive table for shard storage, so that the Spark computing engine can read them in parallel, create the Schema structure of the graph data to be loaded into Janusgraph, and configure the mapping relationship between the Hive table and the Schema structure;

[0013] HBase data loading phase: Use the Spark computing engine to read the vertices and edges in the Hive table in parallel, build the first RDD data set according to the storage structure and encoding method of Janusgraph in the HBase database, and load the first RDD data set in the form of HFile files into the HBase database corresponding to the Janusgraph storage;

[0014] Elasticsearch index building phase: Use the Spark computing engine to read the vertices and edges in the Hive table in parallel, extract the attributes of the vertices or edges that need to be indexed, build the second RDD data set according to the storage structure of Janusgraph in the Elasticsearch index, and write the second RDD data set to the Elasticsearch index in parallel.

[0015] In the above solution, graph data is written into the HBase database through the HFile batch loading method of the HBase database, and the Elasticsearch index is built through the Spark computing engine, thereby bypassing the transaction overhead of calling the Janusgraph API interface.

[0016] Furthermore, the step of saving the data of vertices and edges that need to be loaded into the graph in batches to a Hive table for shard storage so as to facilitate parallel reading by the Spark computing engine includes:

[0017] Divide the data that needs to be loaded into the graph in batches into vertices and edges, forming vertex Hive tables and edge Hive tables respectively, and perform shard storage on each vertex Hive table and edge Hive table;

[0018] Use the Spark computing engine to assign globally unique vertex IDs to all vertices in all vertex Hive tables, replace the associated vertices of all edges in all edge Hive tables with the assigned vertex IDs, and then assign globally unique edge IDs to all edges in all edge Hive tables.

[0019] Furthermore, the steps of creating the Schema structure of the graph data to be loaded into Janusgraph and configuring the mapping relationship between the Hive table and the Schema structure include:

[0020] Create a Schema structure for the graph data to be loaded in Janusgraph, which contains PropertyKey (attribute), VertexLabel (vertex label), EdgeLabel (edge ​​label), and Mixed index;

[0021] Create a configuration file to configure the mapping relationship between the Hive table names of vertices and edges, the fields and the labels and attributes of Janusgraph.

[0022] In the above solution, the API interface provided by Janusgraph is not used for data loading, so the Janusgraph internal IDs of the vertices and edges of the Hive table need to be pre-allocated. It is necessary to create the Janusgraph Schema structure and configure the mapping relationship between the Hive table and the Schema structure so that data can be loaded into the graph according to the configured mapping relationship during data loading and index construction.

[0023] Furthermore, the step of using the Spark computing engine to read the vertices and edges in the Hive table in parallel and constructing the first RDD data set according to the storage structure and encoding method of Janusgraph in the HBase database includes:

[0024] Connect to the Schema structure created by Janusgraph, obtain the relevant information of PropertyKey (attribute), VertexLabel (vertex label), EdgeLabel (edge ​​label), and read the mapping relationship;

[0025] Use the Spark computing engine to read the data in the vertex Hive table and edge Hive table with assigned IDs in parallel, convert and encode them according to the storage structure and encoding method of Janusgraph in the HBase database, so that each data contains the row key, column cluster, column name, and value of the HBase database, and merge all the converted and encoded data into the first RDD data set.

[0026] Furthermore, the step of loading the first RDD data set in the form of an HFile file into an HBase database corresponding to the Janusgraph storage includes:

[0027] Read the row key partition information corresponding to the HBase database stored in Janusgraph, repartition the generated first RDD data set, and sort it in the order of row key, column cluster, and column name within the partition, and then save the first RDD data set in the form of HFile files in the HDFS distributed file system;

[0028] Read the HFile file in the HDFS distributed file system and load the HFile file into the HBase database corresponding to the Janusgraph storage.

[0029] In the above scheme, the Spark computing engine is used in the HBase data loading phase to read the vertices and edges in the Hive table in parallel, and the RDD data set is constructed according to the data storage structure and encoding method of Janusgraph in the HBase database. The RDD data set is partitioned and sorted according to the partitioning method of the HBase database to generate HFile files, ensuring that the HFile files generated by a single partition will not span multiple partitions of the HBase database, and ensuring the storage order of a single partition data to ensure that no errors will occur or performance will be affected when the HFile files are loaded into the HBase database, and then all generated HFile files are loaded into the HBase database.

[0030] Since HFile files are used to load the HBase database instead of the HBase database API interface to load data, transaction overhead caused by the Janusgraph API interface is avoided, and data loading and index construction can be executed in parallel, greatly improving the overall loading efficiency of Janusgraph data.

[0031] Furthermore, the specific steps of the Elasticsearch index construction phase include:

[0032] Connect to the created Mixed index information, which includes the index name, index type, label, index attribute name, and read the configured mapping relationship;

[0033] Use the Spark computing engine to read the vertex Hive table and edge Hive table prepared in the data preparation phase in parallel, use the index information to make a judgment, if a vertex or edge has a configured index, extract the required index attribute data, and build the second RDD data set according to the index storage structure of Janusgraph in Elasticsearch;

[0034] The second RDD data set of extracted and transformed vertices or edges is written to the Elasticsearch index in parallel according to the Elasticsearch index storage location of Janusgraph.

[0035] In the above scheme, since the data is loaded using the HFile batch loading method of the HBase database instead of the API interface of Janusgraph, the index data needs to be written separately. To ensure the efficiency of index writing, the present invention uses Spark to read and load the index data in parallel. In the Elasticsearch index construction phase, Spark is used to read the vertices and edges in the Hive table in parallel, extract the attributes of the vertices or edges that need to be indexed, and build the second RDD data set according to the index storage structure of Janusgraph in Elasticsearch, and then the second RDD data set is written in parallel to the designated storage location of Janusgraph in Elasticsearch.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] (1) The present invention writes graph data into the HBase database by using the HFile file batch loading method of the HBase database, and constructs the Elasticsearch index through the Spark computing engine, thereby bypassing the transaction overhead of calling the API interface of Janusgraph.

[0038] (2) The present invention writes graph data into the HBase database through the HFile file batch loading method of the HBase database, bypassing the unnecessary resource consumption caused by the API interface of the HBase database, and at the same time solving the problem that part of the data may be lost due to the parallel writing of massive data through the API interface of the HBase database, resulting in the loss of graph data.

[0039] (3) The HBase database loading and Elasticsearch index construction of the present invention can be executed simultaneously, that is, stage (two) and stage (three) can be carried out simultaneously, making full use of the HBase cluster and Elasticsearch cluster resources, thereby greatly improving the performance of batch loading of massive data into Janusgraph.

[0040] (4) The HFile file generated by the conversion of the present invention can be used as a backup snapshot of the graph data for storage. The efficiency of redoing the graph can be greatly improved through the HFile file. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 The figure is a flow chart of a method for rapid batch loading of Janusgraph data according to the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0045] Embodiment 1:

[0046] The present invention is particularly suitable for batch loading of massive data of more than 100 million into Janusgraph, using HBase cluster as storage and Elasticsearch cluster as index, which can significantly improve the performance of data loading into the graph. In a production environment, the method of the present invention can be used to quickly batch load massive stock data into the graph, and then incremental data can be updated into the graph through the API interface of Janusgraph, and the data consistency of storage and index can be guaranteed through the transactions provided by Janusgraph. The present invention is implemented by the following technical solutions, such as Figure 1 As shown, a method for fast batch loading of Janusgraph data includes the following steps:

[0047] (I) Data preparation phase: Save the data of vertices and edges that need to be loaded into the graph in batches to the Hive table for shard storage, so that the Spark computing engine can read them in parallel, create the Schema structure of the graph data to be loaded into Janusgraph, and configure the mapping relationship between the Hive table and the Schema structure.

[0048] Step 1-1, divide the data that needs to be loaded into the graph in batches into vertices and edges, form vertex Hive tables and edge Hive tables respectively, and perform shard storage on each vertex Hive table and edge Hive table.

[0049] In step 1-2, use the Spark computing engine to assign globally unique vertex IDs to all vertices in all vertex Hive tables, replace the associated vertices of all edges in all edge Hive tables with the assigned vertex IDs, and then assign globally unique edge IDs to all edges in all edge Hive tables.

[0050] Steps 1-3, create the Schema structure of the graph data to be loaded in Janusgraph, and the Schema structure includes PropertyKey (attribute), VertexLabel (vertex label), EdgeLabel (edge ​​label), and Mixed index.

[0051] Steps 1-4, create a configuration file, configure the mapping relationship between the Hive table name of the vertex and edge, the field and the label and attribute of Janusgraph.

[0052] (ii) HBase data loading phase: Use the Spark computing engine to read the vertices and edges in the Hive table in parallel, build the first RDD data set according to the storage structure and encoding method of Janusgraph in the HBase database, and load the first RDD data set in the form of HFile files into the HBase database corresponding to the Janusgraph storage.

[0053] Step 2-1, connect the Schema structure created in step 1-3 of the data preparation phase, obtain the relevant information of PropertyKey (attribute), VertexLabel (vertex label), EdgeLabel (edge ​​label), and read the mapping relationship configured in step 1-4 of the data preparation phase.

[0054] Step 2-2, use the Spark computing engine to read in parallel the data in the vertex Hive table and edge Hive table whose IDs have been assigned in step 1-2 of the data preparation phase, and convert and encode them according to the storage structure and encoding method of Janusgraph in the HBase database.

[0055] Use the globally unique vertex ID assigned in step 1-2 of the data preparation phase as the internal vertex ID of Janusgraph, and the globally unique edge ID assigned as the internal edge ID of Janusgraph; at the same time, use the vertex and edge information of step 1-1 to encode all data according to the Janusgraph encoding method, so that each data contains the row key, column cluster, column name, and value of the HBase database, and merge all the converted and encoded data into a first RDD data set.

[0056] In the process of converting the globally unique vertex ID into the row key of the HBase database, the last N bits of the vertex ID are used as the partition of the Janusgraph vertex, thereby avoiding the performance problem caused by the unbalanced partition of the HBase database and the first RDD data set.

[0057] Step 2-3, read the row key partition information corresponding to the HBase database stored in Janusgraph, repartition the first RDD data set generated in step 2-2, and sort it in the order of row key, column cluster, and column name within the partition, and then save the first RDD data set in the form of HFile files in the HDFS distributed file system.

[0058] If the partition position of the HBase database is smaller than the partition position of the Janusgraph vertex, the first RDD data set can be repartitioned based on the partition information of the HBase database, and each partition can be further divided into smaller partitions, thereby ensuring that the converted single HFile file shard data belongs to the same HBase database partition. At the same time, the number of partitions of the first RDD data set is increased, the amount of data in each partition is reduced, the resource consumption of sorting the partitions of the first RDD data set is reduced, and the sorting time is shortened.

[0059] Step 2-4, read the HFile file in the HDFS distributed file system and load the HFile file into the HBase database corresponding to the Janusgraph storage.

[0060] (III) Elasticsearch index building phase: Use the Spark computing engine to read the vertices and edges in the Hive table in parallel, extract the attributes of the vertices or edges that need to be indexed, build the second RDD data set according to the storage structure of Janusgraph in the Elasticsearch index, and write the second RDD data set to the Elasticsearch index in parallel.

[0061] Step 3-1, connect the Mixed index information created in step 1-3 of the data preparation phase, which contains information such as index name, index type, label, index attribute name, and read the mapping relationship configured in step 1-4.

[0062] Step 3-2, use the Spark computing engine to read the vertex Hive table and edge Hive table prepared in step 1-2 of the data preparation phase in parallel, use the index information of step 3-1 to make a judgment, if a vertex or an edge has a configured index, extract the required index attribute data, and build the second RDD data set according to the index storage structure of Janusgraph in Elasticsearch.

[0063] In step 3-3, the second RDD data set of the extracted and converted vertices or edges is written in parallel to the Elasticsearch index according to the Elasticsearch index storage location of Janusgraph.

[0064] The present invention uses HBase database as the storage of Janusgraph graph data, uses Elasticsearch index as the index of Janusgraph graph data, and uses Spark computing engine to complete the tasks of data preparation, data loading and data indexing. Spark computing engine submits the tasks to YARN cluster for execution, and uses HDFS and Hive cluster to save original data and intermediate data. HBase, YARN, HDFS and Hive are all deployed in the same Hadoop cluster, and Hadoop cluster and Elasticsearch cluster are independently deployed on different servers.

[0065] This solution writes graph data into the HBase database through the HFile file batch loading method of the HBase database, and builds the Elasticsearch index through the Spark computing engine, thereby bypassing the transaction overhead of the API structure of Janusgraph and the unnecessary resource consumption caused by the API interface of the HBase database. At the same time, it solves the problem that some data may be lost when writing massive data in parallel through the API interface of the HBase database, resulting in missing graph data. Moreover, the HBase database loading and the Elasticsearch index construction can be executed at the same time, that is, stage (two) and stage (three) can be carried out at the same time, making full use of the resources of the HBase cluster and the Elasticsearch cluster, thereby greatly improving the performance of batch loading of massive data into Janusgraph.

[0066] Embodiment 2:

[0067] Based on Example 1, an enterprise investment relationship graph is used as an example of data that needs to be batch loaded into the graph. It is assumed that the enterprise basic information table enterprise_info is used as the vertex Hive table, and the enterprise investment relationship table enterprise_invest_relation is used as the edge Hive table.

[0068] The enterprise_info table data sample is as follows:

[0069] Company Name Unified Social Information Code for Enterprises A Information Technology Co., Ltd. 123*456 B Network Technology Co., Ltd. 789*147

[0070] The following is a sample of enterprise_invest_relation table data:

[0071]

[0072] (I) Data preparation stage

[0073] Step 1-1, import the enterprise_info table and the enterprise_invest_relation table into Hive for distributed storage to form vertex Hive tables and edge Hive tables.

[0074] In step 1-2, use the Spark computing engine to assign globally unique vertex IDs to all vertices in the vertex Hive table enterprise_info, assign globally unique edge IDs to all edges in the edge Hive table enterprise_invest_relation, and replace the vertices in the edge Hive table with the assigned vertex IDs.

[0075] After the replacement is completed, the data sample of the enterprise_info table is as follows:

[0076] Vertex ID Company Name Unified Social Information Code for Enterprises 1 A Information Technology Co., Ltd. 123*456 2 B Network Technology Co., Ltd. 789*147

[0077] The following is a sample of enterprise_invest_relation table data:

[0078]

[0079]

[0080] Steps 1-3, create the Schema structure of the graph data to be loaded in Janusgraph. In this example, the PropertyKeys that need to be created include "Company Name", "Company Unified Social Information Code" and "Investment Ratio", the VertexLabel only has "Company", and the EdgeLabel only has "Investment". Since companies need to be searched based on "Company Name" or "Company Unified Social Information Code", a Mixed index needs to be created for "Company Name" and "Company Unified Social Information Code".

[0081] Steps 1-4, create a configuration file, configure the mapping relationship between the Hive table name of the vertex and edge, the field and the label and attribute of Janusgraph. In this example, you need to configure the enterprise_info table to be mapped to the "Enterprise" VertexLabel (vertex label) of Janusgraph, the enterprise_invest_relation table to be mapped to the "Investment" EdgeLabel (edge ​​label) of Janusgraph, and map the "Investor ID" to the starting vertex ID of the "Investment" EdgeLabel (edge ​​label) of Janusgraph, and map the "Investee ID" to the ending vertex ID of the "Investment" EdgeLabel (edge ​​label) of Janusgraph. Map the "Enterprise Name" and "Enterprise Unified Social Information Code" fields of the enterprise_info table and the "Investment Ratio" field of the enterprise_invest_relation table to the corresponding PropertyKey (attribute) of Janusgraph.

[0082] (II) HBase data loading phase

[0083] Step 2-1, connect the Schema structure configured in step 1-3, and read the mapping relationship configured in step 1-4.

[0084] Step 2-2, use the Spark computing engine to read in parallel the enterprise_info table and enterprise_invest_relation table that have been assigned IDs in step 1-2 of the data preparation phase, and use the information obtained in step 2-1 to convert and encode the data of the enterprise_info table and the enterprise_invest_relation table according to the storage structure and encoding method of Janusgraph in the HBase database, so that each data contains the row key, column cluster, column name, and value of the HBase database, and merge all the converted and encoded data into a first RDD data set.

[0085] Step 2-3, read the row key partition information corresponding to the HBase database stored in Janusgraph, repartition the first RDD data set generated in step 2-2 according to the row key partitioning method of the HBase database, and sort it in the order of row key, column cluster, and column name within the partition, and then save the first RDD data set in the form of HFile files in the HDFS distributed file system.

[0086] Step 2-4, read the HDFS distributed file system and load the HFile file into the HBase database corresponding to the Janusgraph storage.

[0087] (III) Elasticsearch index construction phase

[0088] Step 3-1, connect the Mixed index information created for the "Enterprise Name" and "Enterprise Unified Social Information Code" PropertyKey (attribute) in step 1-3, and read the mapping relationship configured in step 1-4.

[0089] Step 3-2, use the Spark computing engine to read the data of the two Hive tables enterprise_info and enterprise_invest_relation prepared in step 1-2, use the Mixed index information in step 3-1 to make judgments, extract the "enterprise name" and "enterprise unified social information code" data of the enterprise_info table for which the index needs to be created, and build the second RDD data set according to the storage structure of Janusgraph in the Elasticsearch index.

[0090] In step 3-3, the second RDD data set of the extracted and converted vertices or edges is written in parallel to the Elasticsearch index according to the Elasticsearch index storage location of Janusgraph.

[0091] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for fast batch loading of Janusgraph data, characterized in that: The following steps are involved: Data preparation stage: Save the data of vertices and edges that need to be loaded into the graph in batches to the Hive table for shard storage, so that the Spark computing engine can read them in parallel, create the Schema structure of the graph data to be loaded into Janusgraph, and configure the mapping relationship between the Hive table and the Schema structure; HBase data loading phase: Use the Spark computing engine to read the vertices and edges in the Hive table in parallel, build the first RDD data set according to the storage structure and encoding method of Janusgraph in the HBase database, and load the first RDD data set in the form of HFile files into the HBase database corresponding to the Janusgraph storage; Elasticsearch index building phase: Use the Spark computing engine to read the vertices and edges in the Hive table in parallel, extract the attributes of the vertices or edges that need to be indexed, build the second RDD data set according to the storage structure of Janusgraph in the Elasticsearch index, and write the second RDD data set to the Elasticsearch index in parallel.

2. A method for rapid batch loading of Janusgraph data according to claim 1, characterized in that: The step of saving the data of vertices and edges that need to be loaded into the graph in batches to the Hive table for shard storage so as to facilitate parallel reading by the Spark computing engine includes: Divide the data that needs to be loaded into the graph in batches into vertices and edges, forming vertex Hive tables and edge Hive tables respectively, and perform shard storage on each vertex Hive table and edge Hive table; Use the Spark computing engine to assign globally unique vertex IDs to all vertices in all vertex Hive tables, replace the associated vertices of all edges in all edge Hive tables with the assigned vertex IDs, and then assign globally unique edge IDs to all edges in all edge Hive tables.

3. A method for rapid batch loading of Janusgraph data according to claim 2, characterized in that: The steps of creating the Schema structure of the graph data to be loaded into Janusgraph and configuring the mapping relationship between the Hive table and the Schema structure include: Create a Schema structure for the graph data to be loaded into Janusgraph, which contains attributes, vertex labels, edge labels, and Mixed indexes; Create a configuration file to configure the mapping relationship between the Hive table names of vertices and edges, the fields and the labels and attributes of Janusgraph.

4. A method for rapid batch loading of Janusgraph data according to claim 3, characterized in that: The step of using the Spark computing engine to read the vertices and edges in the Hive table in parallel and constructing the first RDD data set according to the storage structure and encoding method of Janusgraph in the HBase database includes: Connect to the Schema structure created by Janusgraph, obtain relevant information about attributes, vertex labels, and edge labels, and read the mapping relationship; Use the Spark computing engine to read the data in the vertex Hive table and edge Hive table with assigned IDs in parallel, convert and encode them according to the storage structure and encoding method of Janusgraph in the HBase database, so that each data contains the row key, column cluster, column name, and value of the HBase database, and merge all the converted and encoded data into the first RDD data set.

5. A method for rapid batch loading of Janusgraph data according to claim 4, characterized in that: The step of loading the first RDD data set in the form of an HFile file into the HBase database corresponding to the Janusgraph storage includes: Read the row key partition information corresponding to the HBase database stored in Janusgraph, repartition the generated first RDD data set, and sort it in the order of row key, column cluster, and column name within the partition, and then save the first RDD data set in the form of HFile files in the HDFS distributed file system; Read the HFile file in the HDFS distributed file system and load the HFile file into the HBase database corresponding to the Janusgraph storage.

6. A method for rapid batch loading of Janusgraph data according to claim 3, characterized in that: The specific steps of the Elasticsearch index construction phase include: Connect to the created Mixed index information, which includes the index name, index type, label, index attribute name, and read the configured mapping relationship; Use the Spark computing engine to read the vertex Hive table and edge Hive table prepared in the data preparation phase in parallel, use the index information to make a judgment, if a vertex or edge has a configured index, extract the required index attribute data, and build the second RDD data set according to the index storage structure of Janusgraph in Elasticsearch; The second RDD data set of extracted and transformed vertices or edges is written to the Elasticsearch index in parallel according to the Elasticsearch index storage location of Janusgraph.

Images