A method and system for fast data loading in graph-relational hybrid storage databases

Abstract

This invention relates to a method and system for fast data loading in a graph-relational hybrid storage database. The method mainly includes: reading a loading configuration file and converting the configuration information in the configuration file into in-memory table objects; prioritizing the table objects to be loaded according to their type, with the priority being relational tables > graphs > label tables > relational tables; and parsing the data source file, processing the data, processing the graph data, and merging the data according to the priority. This invention can reliably load data from multiple relational tables and multiple graphs, making it widely applicable. This invention eliminates the need to specify separate start and end vertices; it allows direct specification of fixed column attributes in the data to automatically parse and obtain start and end vertex information, simplifying the data loading process. This invention processes each resource block independently, reducing unnecessary resource contention and improving data loading performance.

Description

Technical Field

[0001] This invention relates to the field of data loading technology for graph databases, and in particular to a method and system for fast data loading for graph-relational hybrid storage databases. Background Technology

[0002] Graph databases are a type of NoSQL database that primarily uses graph theory to store information about relationships between entities. There are three basic concepts in graph databases: vertices generally indicate an entity; edges represent relationships between entities, usually directed edges; and attributes describe certain information about vertices or edges. The modeling approach in graph databases differs from traditional relational databases. It directly models entities and relationships from the real world, without the high level of abstraction found in relational databases, making it simpler and easier to understand. In terms of complex relationship query capabilities, graph databases are suitable for analyzing multi-level complex relationships and are better suited to today's massive data processing scenarios.

[0003] This graph-relational hybrid storage database supports both relational and graph database storage. The graph model used is an attribute graph, meaning vertices with the same attribute belong to the same label table, and edges with the same attribute belong to the same relation table. Label and relation tables must belong to the same graph, and a graph can have multiple labels and relations. Multiple graphs can exist in the system. Each graph is logically isolated from the others.

[0004] The graph-relational hybrid storage database uses a combination of indexes, volume logs (VLOGs), and native graph arrays for storage. The index stores the offset of a key-value pair within the VLOG, allowing for a globally unique key-value pair to be identified based on its offset. The VLOG stores the actual key-value data, written sequentially and including information such as version numbers, similar to a write-ahead log. The native graph array contains adjacency information for nodes and edges, maintained separately and indexed using IDs (array indices).

[0005] Currently, most mainstream graph database loading tools cannot cover the loading logic of graph-relational hybrid storage databases. This is mainly reflected in:

[0006] (1) Limited data loading function: Existing graph database loading tools are not compatible with loading data from relational databases and graph databases, and cannot meet the loading needs of various types of data.

[0007] (2) The data loading logic of graph databases is complex: Before loading data between graph databases and other heterogeneous data sources, the source data needs to be transformed in a complex manner according to the graph database model. The process is complicated and has low reusability.

[0008] (3) The data loading performance advantage of online graph-relational hybrid storage database is insufficient: When loading data online, it is necessary to consider resource competition in various scenarios, which consumes a lot of effort to control resource competition, resulting in low loading throughput.

[0009] Therefore, overcoming the shortcomings of existing technologies and solving the aforementioned technical problems has become a critical technical challenge that the industry urgently needs to address. Summary of the Invention

[0010] In view of the above-mentioned defects or improvement needs of existing technologies, and in order to solve the problems existing in current tool-based data loading, this invention provides a method and system for fast data loading of graph-relational hybrid storage databases. Its purpose is to be compatible with loading relational data and graph database data, simplify the data loading process, and optimize data loading performance.

[0011] The embodiments of the present invention adopt the following technical solutions:

[0012] In a first aspect, the present invention provides a method for fast data loading of a graph-relational hybrid storage database, comprising: reading a loading configuration file and converting the configuration information in the loading configuration file into a memory table object;

[0013] Based on the type of the table object, the table objects that need to be loaded are prioritized, with the priority being relational table > chart > label table > relational table;

[0014] Based on priority levels, the data source files are parsed, data is processed, graph data is processed, and data is merged.

[0015] Furthermore, when parsing, processing, graph data, and merging data from data source files according to priority levels, the processing of tables of the next priority can only begin after the processing of tables of the current priority has been completed.

[0016] Furthermore, the process of parsing the data source file, processing the data, processing the graph data, and merging the data specifically includes:

[0017] Read the data source file, parse it, and generate in-memory records that are a hybrid graph-relationship storage method;

[0018] Write the memory records to the VLOG and index;

[0019] Convert the data from the chart to the native graph array and update the adjacency information;

[0020] Multiple index files generated during data processing are combined into a single large global index file.

[0021] Furthermore, the process of writing memory records into the VLOG and index specifically includes:

[0022] Write memory records to VLOG: Encode the memory records byte by byte into a binary stream; add the binary stream to the VLOG file to obtain the offset position of this memory record in the VLOG; save the offset position of the record in the VLOG in the memory record;

[0023] Write memory records to the index: Write memory records to the index's memory cache; when the number of memory cache entries reaches a certain threshold, persist the index; retrieve a memory record from the index cache, encode the key of the memory record and its offset in the VLOG into a binary stream by bytes, and write it to the index file; repeat the process of retrieving and writing memory records until all index cache entries have been traversed.

[0024] Furthermore, the process of converting the chart data to a native graph array and updating the adjacency information specifically includes graph data processing of the label table and graph data processing of the relationship table, wherein:

[0025] The graph data processing of the label table includes: assigning a vertex ID to each record of the vertex data in the label table; accessing the native graph array based on the vertex ID and updating the information of the record corresponding to the vertex array ID;

[0026] The graph data processing of the relation table includes: assigning an edge ID to each data entry of the relation table; parsing the vertex key from the edge value and querying the label table to obtain the IDs of the starting vertex and the ending vertex; accessing the original graph array based on the edge ID and updating the information of the record corresponding to the edge array ID; accessing the original graph array based on the starting vertex ID and updating the edge information of the record corresponding to the starting vertex array ID; simultaneously adding a record to the vertex-edge adjacency array to record the association between the starting vertex and the edge; and accessing the original graph array based on the ending vertex ID and updating the edge information of the record corresponding to the ending vertex array ID, simultaneously adding a record to the vertex-edge adjacency array to record the association between the ending vertex and the edge.

[0027] Furthermore, it also includes a multi-threaded processing mechanism, which configures a parsing thread group, a data processing thread group, a graph data processing thread group, and a merging thread group. Each thread group contains multiple processing threads, and these threads execute concurrently. Specifically:

[0028] Configure the data file segment size using the "-B" parameter, configure the number of parsing threads using the "-p" parameter, and configure the number of data processing threads using the "-d" parameter;

[0029] Retrieve data of a specified segment size from the data file, generate a Seg, distribute the Seg to an idle thread obtained from the parsing thread group, and repeat this process until all the contents of the data file have been traversed;

[0030] The parsing thread receives and processes the Seg, generates Records, and distributes the Records to idle threads obtained from the data processing thread group;

[0031] The data processing thread receives and processes Records, generates VLOGs and index records, and distributes Records to idle threads obtained from the graph data processing thread group; for non-graph data, the index records are distributed to idle threads obtained from the data merging thread group.

[0032] The graph data processing thread receives and processes Records, distributing indexed records to idle threads obtained from the data merging thread group.

[0033] Furthermore, it also includes a resource reservation mechanism, which specifically includes:

[0034] Configure the reserved resource limit using the parameter "-m";

[0035] When the parsing thread processes the data, it allocates memory resources all at once according to the configuration.

[0036] When parsing data segments, the parsing thread converts the data into memory records and writes them sequentially to the reserved resources. When the reserved memory resources have been used up, the current resources are submitted to the subsequent process for processing, and then new resources are requested to continue processing.

[0037] Furthermore, it also includes a resource independent processing mechanism: each reserved resource can only be processed by one thread at a time.

[0038] Furthermore, it also includes a load balancing mechanism, which advances some processing steps in the data processing, graph data processing, and data merging process to the parsing processing thread. These advanced processing steps include:

[0039] Find the hash table: In the graph data processing flow, find the hash table of the vertices;

[0040] Vertex HASH generation: In the graph data processing flow, the HASH value of the vertices is calculated.

[0041] Maintaining the HASH of vertices: In the graph data processing flow, vertices are added to the corresponding HASH table;

[0042] Vertex ID and edge ID allocation: In the graph data processing flow, vertex IDs and edge IDs are allocated;

[0043] Determine the starting vertex ID and ending vertex ID of the edge: In the graph data processing flow, the starting vertex ID and ending vertex ID of the edge are obtained by looking up the hash table and hash value.

[0044] On the other hand, the present invention provides a fast data loading system for graph-relational hybrid storage databases, used to implement the fast data loading method for graph-relational hybrid storage databases as described in the first aspect. The system includes a configuration file parsing module, a table object management module, a data source file parsing module, a data processing module, a graph data processing module, and a data merging module, wherein:

[0045] The configuration file parsing module is used to read the loading configuration file and generate a list of tables to be loaded;

[0046] The table object management module is used to classify the tables that need to be loaded according to their type;

[0047] The data source file parsing module is used to read the data source file and parse it to generate specific records that are stored in a graph-relationship hybrid storage.

[0048] The data processing module is used to index specific records and persist VLOGs;

[0049] The graph data processing module is used to maintain and persist the native graph array based on the data of labels and relationships.

[0050] The data merging module is used to clean up intermediate data and complete the persistence of the final data.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] 1. The data loading of this invention can be compatible with loading data from multiple relational tables and multiple charts, and has a wide range of applications.

[0053] 2. When loading edge data, the present invention does not require specifying the start vertex and end vertex separately. Instead, it can directly specify the attributes of fixed columns in the data, and automatically parse and obtain the information of the start vertex and end vertex, thus simplifying the data loading process.

[0054] 3. In the data loading process, each resource block is processed independently, which reduces unnecessary resource competition and improves the performance of data loading. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0056] Figure 1 This is a flowchart of a method for fast data loading in a graph-relational hybrid storage database according to Embodiment 1 of the present invention;

[0057] Figure 2 This is an extended flowchart of step 103 provided in Embodiment 1 of the present invention;

[0058] Figure 3 This is a flowchart of data loading into a relational table provided in Embodiment 1 of the present invention;

[0059] Figure 4 This is a flowchart of data loading into a chart provided in Embodiment 1 of the present invention;

[0060] Figure 5 This is a schematic diagram of a data fast loading system module structure for a graph-relational hybrid storage database provided in Embodiment 2 of the present invention;

[0061] Figure 6 This is a flowchart of the configuration file parsing module provided in Embodiment 2 of the present invention;

[0062] Figure 7 This is a flowchart of the table object management module provided in Embodiment 2 of the present invention;

[0063] Figure 8 This is a flowchart of the data source file parsing module provided in Embodiment 2 of the present invention;

[0064] Figure 9 This is a flowchart of the data processing module provided in Embodiment 2 of the present invention;

[0065] Figure 10 This is a flowchart of the graph data processing module provided in Embodiment 2 of the present invention;

[0066] Figure 11 This is a flowchart of the data merging module provided in Embodiment 2 of the present invention;

[0067] Figure 12 This is the first example model diagram provided in Embodiment 2 of the present invention;

[0068] Figure 13 This is a second example model diagram provided in Embodiment 2 of the present invention;

[0069] Figure 14This is an example diagram of multi-threaded processing provided in Embodiment 3 of the present invention;

[0070] Figure 15 This is a schematic diagram of a data loading device structure for a graph-relational hybrid storage database provided in Embodiment 4 of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining this invention and are not intended to limit this invention. Furthermore, the embodiments described herein are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. In addition, the technical features of the various embodiments or individual embodiments provided by this invention can be arbitrarily combined to form feasible technical solutions, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0072] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0073] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0074] Example 1:

[0075] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for fast data loading in a graph-relational hybrid storage database, and the specific steps are as follows.

[0076] Step 101: Read the loading configuration file and convert the configuration information in the loading configuration file into a memory table object.

[0077] Step 102: Based on the type of the table objects, prioritize the table objects that need to be loaded, with the priority being relational tables > charts > label tables > relational tables.

[0078] Step 103: Based on the priority level, parse the data source files, process the data, process the graph data, and merge the data. In this step, processing of the next priority table can only begin after the current priority table has been processed.

[0079] refer to Figure 2For step 103 of this preferred embodiment, parsing the data source file, processing the data, processing the graph data, and merging the data may specifically include the following steps.

[0080] Step 1031: Read the data source file and parse it to generate in-memory records that are a hybrid graph-relationship storage.

[0081] Step 1032: Write memory records to VLOG and index. This step is divided into writing memory records to VLOG and writing memory records to index. Writing memory records to VLOG includes: encoding the memory record byte-by-byte into a binary stream; adding the binary stream to the VLOG file to obtain the offset position of the memory record in VLOG; and storing the offset position of the record in VLOG in the memory record. Writing memory records to index includes: writing the memory record to the index's memory cache; persisting the index after the number of entries in the index's memory cache reaches a certain threshold; retrieving a memory record from the index cache, encoding the key of the memory record and its offset position in VLOG byte-by-byte into a binary stream, and writing it to the index file; repeating the memory record retrieval and writing process until all index cache entries have been traversed.

[0082] Step 1033: Convert the graph data to the native graph array and update the adjacency information. This step specifically includes graph data processing for the label table and graph data processing for the relation table. The graph data processing for the label table includes: assigning a vertex ID to each record of the vertex data in the label table; accessing the native graph array based on the vertex ID and updating the information of the record corresponding to the vertex array ID. The graph data processing for the relation table includes: assigning an edge ID to each record of the edge data in the relation table; parsing the vertex key from the edge value and querying the label table to obtain the IDs of the starting and ending vertices; accessing the native graph array based on the edge ID and updating the information of the record corresponding to the edge array ID; accessing the native graph array based on the starting vertex ID and updating the edge information of the record corresponding to the starting vertex array ID; simultaneously adding a record to the vertex-edge adjacency array to record the association between the starting vertex and the edge; accessing the native graph array based on the ending vertex ID and updating the edge information of the record corresponding to the ending vertex array ID; simultaneously adding a record to the vertex-edge adjacency array to record the association between the ending vertex and the edge.

[0083] Step 1034: Combine the multiple index files generated during data processing into a large global index file.

[0084] Using the above method, this preferred embodiment can be compatible with both relational data and graph database data, simplifying the data loading process and optimizing data loading performance.

[0085] Based on the above methods, refer to Figure 3In this preferred embodiment, the data loading process for a relational table includes the following steps.

[0086] Step 201: Read the loading configuration file to obtain all tables and attributes that need to be loaded.

[0087] Step 202: Obtain the table structure directly from the data source file and create a table object.

[0088] Step 203: Further read the content of each row from the data source file, generate records, and write them into the graph-relational hybrid storage database.

[0089] refer to Figure 4 In this preferred embodiment, the data loading process for the chart includes the following steps.

[0090] Step 301: Read the loading configuration file to obtain all tables and attributes that need to be loaded.

[0091] Step 302: Based on the table type, prioritize the tables that need to be loaded. The priority is: chart > label table > relationship table.

[0092] Step 303: Prioritize chart processing: Create a chart object based on the configured chart name.

[0093] Step 304: Further Processing of the Label Table: Obtain the label structure directly from the data source file and create label objects. Further Processing of the Label Table: Read the content of each row from the data source file, generate records, and write them to the graph-relational hybrid storage database. Further Processing of the Label Table: Each record in the label corresponds to a vertex; a vertex ID needs to be assigned, and the original graph array information needs to be updated.

[0094] Step 305: Further Processing of the Relationship Table: Directly read the relationship structure from the data source file and create relationship objects. Further Processing of the Relationship Table: Read the content of each row from the data source file, generate records, and write them to the graph-relationship hybrid storage database. Further Processing of the Relationship Table: Each record in the relationship corresponds to an edge, containing the edge's start and end point attributes. The key information of the starting and ending vertices can be directly parsed to obtain their keys, and the IDs of the starting and ending vertices can be retrieved. Based on the retrieved vertex IDs, update the original graph array information, and simultaneously update the adjacency information of the edges. Further Processing of the Relationship Table: Each record in the relationship corresponds to an edge, requiring the assignment of edge IDs, updating the original graph array information, and simultaneously updating the adjacency information of the edges.

[0095] In summary, the data loading in this preferred embodiment is compatible with loading data from multiple relational tables and multiple charts, making it widely applicable. When loading edge data, this preferred embodiment does not require specifying the start and end vertices separately; instead, it allows direct specification of the attributes of fixed columns in the data, automatically parsing and obtaining the information of the start and end vertices, thus simplifying the data loading process. Furthermore, in this preferred embodiment, each resource block is processed independently during data loading, reducing unnecessary resource contention and improving data loading performance.

[0096] Example 2:

[0097] Based on the fast data loading method for graph-relational hybrid storage databases provided in Embodiment 1, Embodiment 2 provides a fast data loading system for graph-relational hybrid storage databases to implement the above method, such as... Figure 5 As shown, the system includes a configuration file parsing module, a table object management module, a data source file parsing module, a data processing module, a graph data processing module, and a data merging module. Specifically: the configuration file parsing module reads the loading configuration file and generates a list of tables to be loaded; the table object management module classifies the tables to be loaded according to their type; the data source file parsing module reads the data source file and parses it to generate specific records for graph-relationship hybrid storage; the data processing module indexes and persists the specific records using VLOG; the graph data processing module maintains and persists the native graph array based on label and relationship data; and the data merging module cleans up intermediate data and completes the persistence of the final data.

[0098] For details, please refer to Figure 6 As shown, the main function of the configuration file parsing module is to read the loaded configuration file, convert the configuration information in the configuration file into in-memory table objects, and output them to the table object management module. The loaded configuration file can configure multiple different similar tables. After the attributes of each table are parsed, a unique table in-memory object is obtained.

[0099] Specifications for loading configuration files:

[0100] [XXX] -- The filename of the data source to be imported.

[0101] name = XXX -- Name. The table name for a relational table, the diagram name, the label name, and the relation name.

[0102] type = XXX -- type. Relational tables, charts, label tables, relational tables.

[0103] graph = XXX -- graph name. For label tables and relationship tables, it is necessary to specify the graph to which they belong.

[0104] begin = XXX -- starting label name. For relational tables, it is necessary to specify their starting label.

[0105] end = XXX -- Termination label name. For relational tables, a termination label needs to be specified.

[0106] Example: Figure 12 The contents of the load.ini configuration file for the model shown are as follows:

[0107] [table0.csv]

[0108] type = table

[0109] name = table0

[0110] [table1.csv]

[0111] type = table

[0112] name = table1

[0113] [graph_1.csv]

[0114] type=graph

[0115] name = graph_1

[0116] [graph_1_label_0.csv]

[0117] type = label

[0118] name = label_0

[0119] graph = graph_1

[0120] [graph_1_label_1.csv]

[0121] type = label

[0122] name = label_1

[0123] graph = graph_1

[0124] [graph_2.csv]

[0125] type=graph

[0126] name = graph_2

[0127] [graph_2_label_0.csv]

[0128] type=label

[0129] name=label_0

[0130] graph=graph_2

[0131] [graph_2_label_1.csv]

[0132] type=label

[0133] name=label_1

[0134] graph=graph_2

[0135] [graph_2_relation0.csv]

[0136] type=relation

[0137] name=relation0

[0138] graph=graph_2

[0139] begin=label_1

[0140] end=label_1

[0141] [graph_2_relation1.csv]

[0142] type=relation

[0143] name=relation1

[0144] graph=graph_2

[0145] begin=label_0

[0146] end=label_1

[0147] [graph_2_relation2.csv]

[0148] type=relation

[0149] name=relation2

[0150] graph=graph_2

[0151] begin=label_1

[0152] end=label_0

[0153] [graph_2_relation3.csv]

[0154] type=relation

[0155] name=relation3

[0156] graph = graph_2

[0157] begin = label_0

[0158] end = label_0

[0159] Based on the above steps, users can configure data loading for various database types without developing any code.

[0160] refer to Figure 7 As shown, the main function of the table object management module is to classify the tables to be loaded according to their type. The priority is: relational tables > charts > label tables > relational tables. Subsequent processing flows are all based on this hierarchy. Tables at the same level can be processed concurrently, and processing of the next level of tables can only begin after the current level has been completed.

[0161] The implementation process is as follows: During configuration file parsing, new table objects are generated; based on the type of the table object, it is placed into the table object list of the corresponding layer; the table objects in the relational table layer are traversed, and data source files are parsed and data is processed; the table objects in the chart layer are traversed, and data source files are parsed, data is processed, and chart data is processed; the table objects in the tag table layer are traversed, and data source files are parsed, data is processed, and chart data is processed; the table objects in the relational table layer are traversed, and data source files are parsed, data is processed, and chart data is processed.

[0162] Example: Figure 12 The hierarchical information of the table objects in the model shown is as follows:

[0163] Table layers: table1, table2

[0164] graph layers: graph_1, graph_2

[0165] label layer: label_0(graph_1), label_1(graph_1), label_0(graph_2), label_1(graph_2)

[0166] relation layer: relation0(graph_2), relation1(graph_2), relation2(graph_2), relation3(graph_2)

[0167] refer to Figure 8 As shown, the main function of the data source file parsing module is to read the data source file and parse it to generate in-memory records in a graph-relational hybrid storage manner. This module needs to traverse all records in the data source file, read the specific data from the file, and transform the read data according to the attribute type of the table fields to generate in-memory records.

[0168] In the data source file, the attribute information of the table needs to be described in the first line. The data source file parsing module needs to read the data in the first line and parse out the specific information such as the attribute name and attribute data type of the table. When processing the data in the subsequent process, the file data will be converted into memory records based on the attribute data type obtained at this time.

[0169] The attribute information is described as follows:

[0170] :name1:type1,:name2:type2,name3:type3,name4:type4

[0171] `name` represents the attribute name. `type` represents the attribute data type. The attribute name and attribute data type must be separated by a first fixed separator (attribute name and attribute type separator). Different attributes must be separated by a second fixed separator (attribute separator), and this second separator must be different from the first fixed separator. `:` before the attribute name indicates that this attribute is a key attribute column of the table.

[0172] Supported attribute data types: Integer, Long, Float, Double, String, Binary, Date, Boolean, Number.

[0173] Parsing the first line of the data source file: Read the first character; if it is ':', it indicates a key attribute; continue reading the string until an attribute name and attribute type separator are read, the resulting string is the attribute name; continue reading the string until an attribute separator is read, the resulting string is the attribute type; save the attribute name, attribute type, and whether it is a key attribute to the attribute array of the table object; repeat the above steps until all characters in the first line have been traversed.

[0174] The conversion of records from the data source file to records in memory is as follows: Read the contents of the data source file until an attribute delimiter is read, and the obtained string is the attribute value; query the attribute array of the table object to obtain the corresponding attribute information; according to the attribute type of the attribute information, convert the obtained string into a numerical value to obtain the memory data; if the obtained string is the last character of a line, save the current record as a complete memory record; repeat the above steps until all characters in the data source file have been traversed.

[0175] Example: The data source file table1.csv contains the following:

[0176] :key1:Integer,:key2:Long,value1:Integer,value2:Long,value3:Float

[0177] 0,0,0,0,0.000000

[0178] 0,1,1,1,1.000000

[0179] 0,2,2,2,2.000000

[0180] 0,3,3,3,3.000000

[0181] After parsing and processing the first row, the attribute information of table1 can be obtained as follows:

[0182]

[0183] After the data is parsed by the data source, the resulting memory records are as follows:

[0184]

[0185]

[0186] refer to Figure 9 As shown, the main function of the data processing module is to write memory records into the index and VLOG. The data processing module writes the key and offset of the memory record into the index.

[0187] Writing memory records to a VLOG mainly involves the following steps: encoding the memory record into a binary stream byte by byte; adding the binary stream to the VLOG file to obtain the offset position of this record in the VLOG; and saving the offset position of the record in the VLOG in the memory record.

[0188] Writing in-memory records to the index mainly involves the following steps: writing the in-memory record to the index's memory cache; persisting the index after the number of records in the index's memory cache reaches a certain threshold; retrieving a record from the index cache; encoding the record's key and its offset in the VLOG into a binary stream by bytes and writing it to the index file; repeating the process of "retrieving a record from the index cache; encoding the record's key and its offset in the VLOG into a binary stream by bytes and writing it to the index file" until all index caches have been traversed.

[0189] Example: The contents recorded in memory are as follows:

[0190]

[0191] After data processing, the data in the VLOG is as follows:

[0192]

[0193]

[0194] The first column shows the offset position recorded in the VLOG.

[0195] After updating the memory record and writing the memory record to the index, the data in the index cache is:

[0196]

[0197] After the index is persisted, the contents of the index file are:

[0198] Integer(0) Long(0) 1000 Integer(1) Long(1) 1028 Integer(2) Long(2) 1056 Integer(3) Long(3) 1084

[0199] refer to Figure 10 As shown, the main function of the graph data processing module is to convert the graph data into the native graph array and update the adjacency information.

[0200] Graph data processing for the label table: For the vertex data in the label table, the graph data processing module needs to assign a vertex ID to each record; based on the vertex ID, access the native graph array and update the information of the record corresponding to the vertex array ID.

[0201] Example of label representation: Figure 13 The label table data source file for the model shown contains the following:

[0202] label_0.csv

[0203] :key1:Integer,value1:Integer,value2:Long

[0204] 0,0,0

[0205] 1,1,1

[0206] 2,2,2

[0207] 3,3,3

[0208] label_1.csv

[0209] :key1:Long,value1:Integer,value2:Long

[0210] 0,0,0

[0211] 1,1,1

[0212] 2,2,2

[0213] 3,3,3

[0214] The VLOG data obtained after data processing is as follows:

[0215] 2000 Integer(0) Integer(0) Long(0) 2016 Integer(1) Integer(1) Long(1) 2032 Integer(2) Integer(2) Long(2) 2048 Integer(3) Integer(3) Long(3) 2068 Long(0) Integer(0) Long(0) 2088 Long(1) Integer(1) Long(1) 2108 Long(2) Integer(2) Long(2) 2128 Long(3) Integer(3) Long(3)

[0216] The first column shows the offset position recorded in the VLOG.

[0217] The contents of the obtained index file are:

[0218]

[0219]

[0220] After updating the native graph array, the resulting vertex array is recorded as follows:

[0221] offset Entering edge Going out of bounds 0 2000 NA NA 1 2016 NA NA 2 2032 NA NA 3 2048 NA NA 4 2068 NA NA 5 2088 NA NA 6 2108 NA NA 7 2128 NA NA

[0222] The first column is the vertex array index, and the second column is the VLOG offset.

[0223] Graph data processing for relational tables: For edge data in the relational table, the graph data processing module needs to assign an edge ID to each data entry; parse the vertex key from the edge value, and query the label table to obtain the IDs of the starting and ending vertices; based on the edge ID, access the native graph array and update the information of the record corresponding to the edge array ID; based on the starting vertex ID, access the native graph array and update the edge information of the record corresponding to the starting vertex array ID; simultaneously, add a record to the vertex-edge adjacency array to record the association between the starting vertex and the edge. Based on the ending vertex ID, access the native graph array and update the edge information of the record corresponding to the ending vertex array ID; simultaneously, add a record to the vertex-edge adjacency array to record the association between the ending vertex and the edge.

[0224] Example of relation representation: Figure 13The data source file containing the relational tables of the model shown is as follows:

[0225] relation_0.csv

[0226] :key1:Long,begin_key:Integer,out_key:Long,value3:Float

[0227] 0,1,0,0.000000

[0228] 1,1,2,1.000000

[0229] 2,1,3,2.000000

[0230] relation_1.csv

[0231] :key1:Long,begin_key:Integer,out_key:Long,value3:Long

[0232] 0,0,0,0

[0233] 1,0,1,1

[0234] 2,0,2,2

[0235] 3,0,3,3

[0236] relation_2.csv

[0237] :key1:Long,begin_key:Integer,out_key:Long,value3:Integer

[0238] 0,1,1,0

[0239] 1,1,2,1

[0240] 2,1,3,2

[0241] relation_3.csv

[0242] :key1:Long,begin_key:Integer,out_key:Long,value3:Integer

[0243] 0,0,1,0

[0244] 1,0,2,1

[0245] 2,0,3,2

[0246] The VLOG data obtained after data processing is as follows:

[0247]

[0248]

[0249] The first column shows the offset position recorded in the VLOG.

[0250] The contents of the obtained index file are:

[0251] Long(0) 3000 Long(1) 3024 Long(2) 3048 Long(0) 3076 Long(1) 3104 Long(2) 3132 Long(3) 3160 Long(0) 3184 Long(1) 3204 Long(2) 3228 Long(0) 3252 Long(1) 3276 Long(2) 3300

[0252] After updating the original graph array, the edge array records are as follows:

[0253]

[0254]

[0255] The first column is the edge array index, and the second column is the VLOG offset.

[0256] After updating the vertex array, the record is as follows:

[0257] offset Entering edge Going out of bounds 0 2000 NA 3,4,5,6,10,11,12 1 2016 7,10 NA 2 2032 8,11 NA 3 2048 9,12 NA 4 2068 3 0 5 2088 4 0,1,2,7,8,9 6 2108 1,5 NA 7 2128 2,9 NA

[0258] The first column is the vertex array index, and the second column is the VLOG offset.

[0259] refer to Figure 11 As shown, the main function of the data merging module is to combine multiple index files generated during data processing into a large global index file. Since loading large amounts of data typically results in multiple index files, data merging is necessary to combine the numerous small index files into a single large and comprehensive index file, facilitating rapid service provision.

[0260] Example:

[0261] The contents of index file 1 are as follows (the index file in the data processing module example):

[0262] Integer(0) Long(0) 1000 Integer(1) Long(1) 1028 Integer(2) Long(2) 1056 Integer(3) Long(3) 1084

[0263] The contents of index file 2 are as follows (index file in the tag table data processing example):

[0264] Integer(0) 2000 Integer(1) 2016 Integer(2) 2032 Integer(3) 2048 Long(0) 2068 Long(1) 2088 Long(2) 2108 Long(3) 2128

[0265] The contents of index file 3 are as follows (index file in the relational table data processing example):

[0266]

[0267]

[0268] The content of the merged index file is as follows:

[0269]

[0270]

[0271] In summary, the data loading in this preferred embodiment is compatible with loading data from multiple relational tables and multiple charts, making it widely applicable. When loading edge data, this preferred embodiment does not require specifying the start and end vertices separately; instead, it allows direct specification of the attributes of fixed columns in the data, automatically parsing and obtaining the information of the start and end vertices, thus simplifying the data loading process. Furthermore, in this preferred embodiment, each resource block is processed independently during data loading, reducing unnecessary resource contention and improving data loading performance.

[0272] Example 3:

[0273] Based on the data loading method and system for graph-relational hybrid storage databases provided in Embodiments 1 and 2, this Embodiment 3 also provides several optimization mechanisms for fast data loading, including a multi-threaded processing mechanism, a resource reservation mechanism, a resource independent processing mechanism, and a load balancing mechanism. The multi-threaded processing mechanism is mainly used to segment the data source file and distribute the file segments to multiple threads for concurrent processing. The resource reservation mechanism is mainly used to limit the amount of resources used; resources are reserved during data parsing, and when the resource threshold is reached during data processing, resources are switched directly, with new data written directly to new resource blocks and old resource blocks persisted. The resource independent processing mechanism is mainly used to ensure that data in each resource database is processed independently by a single thread, and each resource is persisted to a separate file, avoiding resource contention. The load balancing mechanism is mainly used to split time-consuming processing flows (such as finding the starting and ending vertices of an edge) and distribute them to shorter processing flows, balancing the processing time of each module's threads and achieving load balancing between processing flows.

[0274] Specifically, the multi-threaded processing mechanism provided in this embodiment is adopted to improve data loading performance. This multi-threaded processing mechanism is configured with a parsing thread group, a data processing thread group, a graph data processing thread group, and a merging thread group. Each thread group contains multiple processing threads, which execute concurrently. The specific implementation process includes: configuring the data file segment size using the parameter "-B", configuring the number of parsing threads using the parameter "-p", and configuring the number of data processing threads using the parameter "-d"; retrieving data of the specified segment size from the data file, generating Seg, and distributing the Seg to idle threads obtained from the parsing thread group, repeating this process until all content of the data file has been traversed; the parsing thread receives and processes the Seg, generates Records, and distributes the Records to idle threads obtained from the data processing thread group; the data processing thread receives and processes the Records, generates VLOG and index records, and distributes the Records to idle threads obtained from the graph data processing thread group; for non-graph data, the index records are distributed to idle threads obtained from the data merging thread group; the graph data processing thread receives and processes the Records, and distributes the index records to idle threads obtained from the data merging thread group.

[0275] Example: Load a data source file that is 1024K in size.

[0276] The parameter "-B 128" configures the data file segment size to 128K. The parameter "-p2" configures the number of parsing threads to 2. The parameter "-d 4" configures the number of data processing threads to 4. The number of graph data processing threads and data merging threads is the same as the number of data processing threads, both being 4. Figure 14 As shown.

[0277] The first thread to run is:

[0278] Parsing Thread 1 Seg 1[0,128K) Record 1 Parsing Thread 2 Seg 2[128K,256K) Record 2

[0279] The thread that runs in the second round is:

[0280] Parsing Thread 1 Seg 3 [256K, 384K) Record 3 Parsing Thread 2 Seg 4 [384K, 512K) Record 4 Data processing thread 1 Record 1 Index 1 Data processing thread 2 Record 2 Index 2

[0281] The thread that runs in the third round is:

[0282]

[0283]

[0284] The thread that runs in the fourth round is:

[0285] Parsing Thread 1 Seg 7 [896K, 1024K) Record 7 Data processing thread 1 Record 5 Index 5 Data processing thread 2 Record 6 Index 6 Graph data processing thread 1 Record 3 Graph Data Processing Thread 2 Record 4 Merge Thread 1 Index 1, Index 2 Index 1-1 Merge Thread 2 Index 3, Index 4 Index 1-2

[0286] The thread that runs in the fifth round is:

[0287] Data processing thread 1 Record 7 Index 7 Graph data processing thread 1 Record 5 Graph Data Processing Thread 2 Record 6 Merge Thread 1 Index 5, Index 6 Index 1-3 Merge Thread 2 Index 3, Index 4 Index 1-2

[0288] The thread that runs in the sixth round is:

[0289] Graph data processing thread 1 Record 7 Merge Thread 1 Index 5, Index 6 Index 1-3 Merge Thread 2 Index 1-1, Index 1-2 Index 1-4

[0290] The thread that runs in the seventh round is:

[0291] Merge Thread 1 Index 1-3, Index 1-4 Index 1-5

[0292] The resource reservation mechanism provided in this embodiment is adopted to reduce the performance overhead caused by frequent memory resource requests and improve resource utilization. Specifically, this mechanism includes: configuring a reserved resource limit using the parameter "-m"; requesting memory resources all at once according to the configuration during parsing thread processing; when parsing data segments, converting the data into memory records and writing them directly and sequentially into the reserved resources; and when the reserved memory resources have been exhausted, submitting the current resources to subsequent processes for processing, and then re-requesting resources to continue processing. Furthermore, to improve resource utilization, this embodiment expands the data during parsing, enabling subsequent processing flows to directly reuse resources, thereby reducing unnecessary resource overhead.

[0293] The independent resource processing mechanism provided in this embodiment is adopted to reduce the performance overhead caused by resource contention. Each reserved resource can only be processed by one thread at the same time, ensuring that there is no contention and simplifying the processing flow.

[0294] For the load balancing mechanism provided in this embodiment, since the system throughput is determined by the process with the lowest performance, a load balancing mechanism between modules is adopted to improve the processing capacity of the loading system. The loading processing flow mainly includes parsing, data processing, graph data processing, and data merging. Among these, the performance of parsing is far superior to that of other processes. Therefore, this embodiment of the invention chooses to advance some processing steps in the data processing, graph data processing, and data merging processes to the parsing processing thread. By reducing the performance of the parsing process, the performance of other modules is improved, so that the load between modules is balanced, thereby improving the system throughput. The aforementioned advanced processing steps mainly include looking up the hash table, generating the hash of vertices, maintaining the hash of vertices, allocating vertex IDs and edge IDs, and determining the starting vertex ID and ending vertex ID of edges.

[0295] For looking up the hash table: To more quickly find the starting and ending vertices when adding edges, the loading system of this invention maintains a hash table of vertices. This hash table needs to be found during the graph data processing flow.

[0296] For vertex hash generation: To more quickly find the starting and ending vertices when adding edges, the loading system of this invention maintains a vertex hash table. The hash values ​​of the vertices need to be calculated during the graph data processing flow.

[0297] For maintaining the HASH of vertices: In the graph data processing flow, it is necessary to calculate the HASH value of the vertices and add the vertices to the corresponding HASH table.

[0298] For the allocation of vertex IDs and edge IDs: In the graph data processing flow, vertex IDs and edge IDs need to be allocated.

[0299] To determine the starting and ending vertex IDs of an edge: In the graph data processing flow, it is necessary to obtain the starting and ending vertex IDs of the edge. These can be obtained by looking up the hash table or hash values.

[0300] Example 4:

[0301] Based on the data loading method and system for graph-relational hybrid storage databases provided in Embodiments 1 to 3 above, the present invention also provides a data loading device for graph-relational hybrid storage databases that can be used to implement the above methods and module processes, such as... Figure 15 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The data loading device for a graph-relational hybrid storage database in this embodiment includes one or more processors 21 and a memory 22. Figure 15 Take a processor 21 as an example.

[0302] Processor 21 and memory 22 can be connected via a bus or other means. Figure 15 Taking the example of a connection between China and Israel via a bus.

[0303] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the data fast loading method and module flow for graph-relational hybrid storage databases in Embodiments 1 and 2. The processor 21 executes various functional applications and data processing of the data fast loading device for graph-relational hybrid storage databases by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the data fast loading method and module flow for graph-relational hybrid storage databases in Embodiments 1 and 3.

[0304] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0305] The program instructions / modules are stored in memory 22. When executed by one or more processors 21, they perform the fast data loading method for graph-relational hybrid storage database described in Embodiments 1 to 3 above. For example, they perform the methods described above. Figures 1-4 The steps shown.

[0306] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0307] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fast data loading in a graph-relational hybrid storage database, characterized in that, include: Read the loading configuration file and convert the configuration information in the loading configuration file into a memory table object; Based on the type of the table object, the table objects that need to be loaded are prioritized, with the priority being relational table > chart > label table > relational table; Based on priority levels, the data source files are parsed, data is processed, graph data is processed, and data is merged. This includes: reading the data source files and parsing them to generate in-memory records that are a hybrid storage of graph and relation; writing the in-memory records to VLOG and index; converting the graph data to the native graph array and updating the adjacency information; and merging the multiple index files generated during data processing into a large global index file.

2. The method for fast data loading in a graph-relational hybrid storage database according to claim 1, characterized in that, When parsing, processing, graph data, and merging data based on priority levels, the processing of tables of the next priority can only begin after the processing of tables of the current priority has been completed.

3. The method for fast data loading in a graph-relational hybrid storage database according to claim 1, characterized in that, The specific steps of writing memory records to the VLOG and index include: Write memory records to VLOG: Encode the memory records byte by byte into a binary stream; add the binary stream to the VLOG file to obtain the offset position of this memory record in the VLOG; save the offset position of the record in the VLOG in the memory record; Write memory records to the index: Write memory records to the index's memory cache; after the number of memory cache entries reaches a set threshold, persist the index; retrieve a memory record from the index cache, encode the key of the memory record and its offset position in the VLOG into a binary stream by bytes, and write it to the index file; repeat the process of retrieving and writing memory records until all index cache entries have been traversed.

4. The method for fast data loading in a graph-relational hybrid storage database according to claim 1, characterized in that, The process of converting the chart data to a native graph array and updating adjacency information specifically includes graph data processing of the label table and graph data processing of the relationship table, wherein: The graph data processing of the label table includes: assigning a vertex ID to each record of the vertex data in the label table; accessing the native graph array based on the vertex ID and updating the information of the record corresponding to the vertex array ID; The graph data processing of the relation table includes: assigning an edge ID to each data entry of the relation table; parsing the vertex key from the edge value and querying the label table to obtain the IDs of the starting vertex and the ending vertex; accessing the original graph array based on the edge ID and updating the information of the record corresponding to the edge array ID; accessing the original graph array based on the starting vertex ID and updating the edge information of the record corresponding to the starting vertex array ID; simultaneously adding a record to the vertex-edge adjacency array to record the association between the starting vertex and the edge; and accessing the original graph array based on the ending vertex ID and updating the edge information of the record corresponding to the ending vertex array ID, simultaneously adding a record to the vertex-edge adjacency array to record the association between the ending vertex and the edge.

5. The method for fast data loading in a graph-relational hybrid storage database according to any one of claims 1-4, characterized in that, It also includes a multi-threaded processing mechanism, which is configured with a parsing thread group, a data processing thread group, a graph data processing thread group, and a merging thread group. Each thread group contains multiple processing threads, and these threads execute concurrently. Specifically: Configure the data file segment size using the "-B" parameter, configure the number of parsing threads using the "-p" parameter, and configure the number of data processing threads using the "-d" parameter; Retrieve data of a specified segment size from the data file, generate a Seg, distribute the Seg to an idle thread obtained from the parsing thread group, and repeat this process until all the contents of the data file have been traversed; The parsing thread receives and processes the Seg, generates Records, and distributes the Records to idle threads obtained from the data processing thread group; The data processing thread receives and processes Records, generates VLOGs and index records, and distributes Records to idle threads obtained from the graph data processing thread group; for non-graph data, the index records are distributed to idle threads obtained from the data merging thread group. The graph data processing thread receives and processes Records, distributing indexed records to idle threads obtained from the data merging thread group.

6. The method for fast data loading in a graph-relational hybrid storage database according to claim 5, characterized in that, It also includes a resource reservation mechanism, which specifically includes: Configure the reserved resource limit using the parameter "-m"; When the parsing thread processes the data, it allocates memory resources all at once according to the configuration. When parsing data segments, the parsing thread converts the data into memory records and writes them sequentially to the reserved resources. When the reserved memory resources have been used up, the current resources are submitted to the subsequent process for processing, and then new resources are requested to continue processing.

7. The method for fast data loading in a graph-relational hybrid storage database according to claim 6, characterized in that, It also includes a resource independent processing mechanism: each reserved resource can only be processed by one thread at a time.

8. The method for fast data loading in a graph-relational hybrid storage database according to claim 5, characterized in that, It also includes a load balancing mechanism, which advances some processing steps in the data processing, graph data processing, and data merging process to the parsing processing thread. These advanced processing steps include: Find the hash table: In the graph data processing flow, find the hash table of the vertices; Vertex HASH generation: In the graph data processing flow, the HASH value of the vertices is calculated; Maintaining the HASH of vertices: In the graph data processing flow, vertices are added to the corresponding HASH table; Vertex ID and edge ID allocation: In the graph data processing flow, vertex IDs and edge IDs are allocated; Determine the starting vertex ID and ending vertex ID of the edge: In the graph data processing flow, the starting vertex ID and ending vertex ID of the edge are obtained by looking up the hash table and hash value.

9. A system for fast data loading in a graph-relational hybrid storage database, used to implement the method for fast data loading in a graph-relational hybrid storage database as described in any one of claims 1-8, characterized in that, It includes a configuration file parsing module, a table object management module, a data source file parsing module, a data processing module, a graph data processing module, and a data merging module, among which: The configuration file parsing module is used to read the loading configuration file and generate a list of tables to be loaded; The table object management module is used to classify the tables that need to be loaded according to their type; The data source file parsing module is used to read the data source file and parse it to generate in-memory records that are stored in a hybrid graph-relationship manner; The data processing module is used to write memory records into the index and VLOG; The graph data processing module is used to convert the graph data into a native graph array and update the adjacency information. The data merging module is used to combine multiple index files generated during data processing into a large global index file.

Images