Optimization method and device for full data import of graph database based on memory database

Abstract

The invention discloses an optimization method and device for full data import of a graph database based on a memory database, relates to the technical field of data import of the graph database, and solves the problem of extra graph database query overhead caused by source point and target point query operation required during edge data import. The method comprises the following steps: introducing a mapping relationship between a primary key and an internal identifier of a cache point of a memory database, and directly querying and obtaining the internal identifier of a source point and a target point from the memory database when constructing a GQL statement for edge data import; and meanwhile, the data importing task is asynchronously processed in a producer-consumer mode, so that the additional query operation of the graph database server during edge data importing is avoided, the communication frequency between the client and the graph database server is effectively reduced, the network overhead is reduced, and the data importing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data import into a graph database, and in particular to an optimization method and device for importing full data into a graph database based on an in-memory database. Background Art

[0002] A graph database is a database that stores and queries data in a graph structure and is widely used in social networks, recommendation systems, knowledge graphs, and other fields. In a graph database, data is usually stored in the form of nodes and edges, where nodes represent entities and edges represent the relationship between entities.

[0003] In the prior art, graph databases require that edge data must depend on point data. That is, when inserting an edge data into a graph database, the source node and target node data associated with the edge data must already exist in the graph database. Therefore, edge data import into a graph database generally includes the following steps:

[0004] 1. When importing point data, the graph database server generates a unique internal identifier (internalId) for each point;

[0005] 2. When importing edge data, you need to query the corresponding internal identifier (internalId) from the graph database based on the primary key of the source and target points of the edge. The number of internalId queries required depends on the number of partitions to which the primary keys of the source and target points in a batch of edge data are divided.

[0006] 3. Map the queried internal identifier (internalId) to the edge, and then write the underlying edge.

[0007] However, the above methods in the prior art have the following problems:

[0008] 1. Performance bottleneck: Each time edge data is imported, the graph database query engine needs to query the source and target points, which adds additional network communication and database query overhead;

[0009] 2. Synchronous waiting: When the database service endpoint data is written, the internalId is generated and the point data is written synchronously. When the edge data is written, the internal of the source and target points is queried and the edge data is written synchronously, which leads to low import efficiency. Summary of the invention

[0010] The purpose of this application is to overcome the problems of additional graph database query overhead caused by the need to query source and target points when importing edge data in the prior art, as well as low import efficiency caused by synchronous waiting, and to provide an optimization method and device for importing full data of a graph database based on an in-memory database.

[0011] In a first aspect, a method for optimizing full data import of a graph database based on an in-memory database is provided, comprising:

[0012] Start a first producer thread, wherein the first producer thread is used to process source data and construct a first GQL statement for batch writing points, send the first GQL statement to the graph database server, and after receiving a response from the graph database server, put the response result into a first queue and continue to process the next batch of source data, wherein the response result includes the primary key and internal identifier of the point written in the current batch;

[0013] Start a first consumer thread, wherein the first consumer thread is used to obtain a response result from the first queue, iteratively parse the response result to obtain a mapping relationship between the primary key and the internal identifier of all points in the current batch, and write the mapping relationship into a memory database;

[0014] When writing edge data, the primary key set of the source point and the target point is extracted from the edge data, the corresponding internal identifier is queried from the memory database according to the primary key set, and the second GQL statement for writing the edge data is constructed according to the mapping relationship between the queried primary key and the internal identifier.

[0015] In some possible implementations, when importing point data, the client configures in the first GQL statement to allow the graph database server to return the primary keys and internal identifiers of the points written in batches to the client, so that the response results returned by the graph database server include the primary keys and internal identifiers of the points written in the current batch.

[0016] In some possible implementations, when writing edge data:

[0017] Start a second producer thread, wherein the second producer thread is used to extract a primary key set of a source point and a target point in the edge data, query a corresponding internal identifier from the memory database according to the primary key set, construct a second GQL statement for writing the edge data according to a mapping relationship between the queried primary key and the internal identifier, and put the constructed second GQL statement into a second queue;

[0018] Start a second consumer thread, wherein the second consumer thread is used to obtain a second GQL statement from the second queue, send the obtained second GQL statement to the graph database server and receive an execution result;

[0019] During the edge data import process, the producer-consumer model is used to asynchronously process edge data import tasks, which reduces the synchronization waiting time during the edge data import process and significantly improves the efficiency of edge data import.

[0020] In some possible implementations, the first producer thread and the first consumer thread process data asynchronously.

[0021] In some possible implementations, the second producer thread and the second consumer thread process data asynchronously.

[0022] In a second aspect, an optimization device for importing full data of a graph database based on a memory database is provided, comprising:

[0023] A first production module is used to start a first producer thread, wherein the first producer thread is used to process source data and construct a first GQL statement for batch writing points, send the first GQL statement to the graph database server, and after receiving a response from the graph database server, put the response result into a first queue and continue to process the next batch of source data, wherein the response result includes the primary key and internal identifier of the point written in the current batch;

[0024] A first consumption module, used to start a first consumer thread, wherein the first consumer thread is used to obtain a response result from the first queue, iteratively parse the response result to obtain a mapping relationship between the primary key and the internal identifier of all points in the current batch, and write the mapping relationship into a memory database;

[0025] The edge data writing module is used to extract the primary key set of the source point and the target point from the edge data when writing the edge data, query the corresponding internal identifier from the memory database according to the primary key set, and construct a second GQL statement for writing the edge data according to the mapping relationship between the queried primary key and the internal identifier.

[0026] In some possible implementations, the edge data writing module includes:

[0027] A second production module is used to start a second producer thread, wherein the second producer thread is used to extract a primary key set of a source point and a target point in the edge data, query a corresponding internal identifier from the memory database according to the primary key set, and construct a second GQL statement for writing the edge data according to a mapping relationship between the queried primary key and the internal identifier, and put the constructed second GQL statement into a second queue;

[0028] The second consumption module is used to start a second consumer thread, wherein the second consumer thread is used to obtain a second GQL statement from the second queue, send the obtained second GQL statement to the graph database server and receive the execution result.

[0029] In some possible implementations, the first producer thread and the first consumer thread process data asynchronously, and the second producer thread and the second consumer thread process data asynchronously.

[0030] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes steps for executing the method in any one of the implementations of the first aspect described above.

[0031] In a fourth aspect, an electronic device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements a method as in any one of the implementations in the first aspect above.

[0032] The present application has the following beneficial effects: when importing data into a graph database, by introducing a memory database to cache the mapping relationship between the primary key of a point and an internal identifier (internalId), there is no need to query the internal identifiers (internalId) of the source and target points from the graph database server when importing edge data, thereby avoiding additional query operations on the graph database server when importing edge data, effectively reducing the number of communications between the client and the graph database server, and reducing network overhead; at the same time, the producer-consumer mode is used to asynchronously process point data import tasks, reducing the synchronization waiting time during the point data import process, significantly improving the efficiency of point data import, and being applicable to the full data import scenario of the graph database, with broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1It is a flowchart of an optimization method for importing full data of a graph database based on an in-memory database according to Example 1 of the present application;

[0036] Figure 2 It is a flowchart of point data import optimization in the optimization method for full data import of a graph database based on a memory database in Example 1 of the present application;

[0037] Figure 3 It is a flowchart of edge data import optimization in the optimization method for full data import of a graph database based on a memory database in Example 1 of the present application;

[0038] Figure 4 It is a structural block diagram of an optimization device for importing full data of a graph database based on an in-memory database according to Embodiment 2 of the present application;

[0039] Figure 5 It is a schematic diagram of the internal structure of the electronic device of Example 4 of the present application.

[0040] Reference numerals:

[0041] 100, first production module; 200, first consumption module; 300, edge data writing module; 301, second production module; 302, second consumption module. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example 1

[0044] like Figure 1 As shown, Example 1 of the present application involves an optimization method for importing full data into a graph database based on an in-memory database, including two parts: optimization of point data import and optimization of edge data import.

[0045] like Figure 2 As shown, in the optimization of point data import, when importing point data, the client configures the first GQL statement for writing point data to allow the graph database server to return the primary key and internalId of the points written in batches to the client. The client starts a producer thread, which is responsible for processing the source data in batches to construct the first GQL statement for writing point data, and sends the first GQL statement to the graph database server. After receiving the response from the graph database server, the producer puts the response result into the first queue and continues to process the next batch of source data.

[0046] The client starts a consumer thread, which is responsible for obtaining the response results of the graph database server from the first queue, iteratively parsing the response results, obtaining the mapping relationship between the primary key and internalId of all points in the current batch, and writing the mapping relationship key-value pairs into the memory database, where the memory database includes but is not limited to redis, dragonfly and other memory databases.

[0047] Taking redis as an example of the in-memory database, the specific implementation code for point data writing is as follows: / / Create a producer val producer = Future { try { iterator.grouped(nodeConfig.batchSize).foreach(nodes =>{ / / Construct a GQL statement for batch writing points val gql = writer.getGql(nodes, true) if (gql != null) { / / Send the GQL statement to the graph database server val (result, host) = graphProvider.execute(gql) if (result.isSucceeded) { / / Create a consumer val consumer = Future { / / Get the response result from the queue val (result, endFlag) = queue.take() if ((result == null&&producer.isCompleted) || endFlag.equals("NEBULA")) { Then, the response result is iteratively parsed to obtain the mapping relationship between the primary key and the internal identifier of all points in the current batch, and the mapping relationship is written into the memory database. Specifically, iterative parsing is a traversal parsing process of all data result sets contained in the response result according to the access requirements. The parsing here refers to obtaining all data rows (each row is a node) from the response result, and extracting the value of the node primary key and the value of the internal identifier from the node, so that the mapping relationship between the node primary key and the internal identifier (internalId) can be obtained through the iterative parsing process, and then the obtained mapping relationship between the node primary key and the internal identifier (internalId) can be saved in the local memory database. The implementation code is as follows: val nodePkToElementIdMap = new mutable.HashMap[String, Long] / / Iterate the loop to parse the nodes in the response result while (result.hasNext) { / / Parse the node data type val node = result.next().get(0).asNode / / Get the mapping relationship between the node primary key and the internal identifier (internalId) and put it into the local Map nodePkToElementIdMap.put(nodeTypeMap(nodeConfig.name) +node.getProperties.asScala(nodeDesc.nodePkNames.head).toString, node.getId) } / / Write the mapping between the node's primary key and internal identifier (internalId) to the in-memory database ThirdDbUtils.batchWrite(pool, nodePkToElementIdMap) } } }(ExecutionContext.fromExecutor(executor)) like Figure 3As shown in the figure, in the optimization of edge data import, when importing edge data, the client queries the internal identifiers corresponding to these primary keys from the local memory database based on the primary keys of the source and target points in the edge data, and then uses the special write edge statement provided by the graph database to directly construct the source and target points based on the internal identifiers of the points, without having to query through the graph database service to obtain the internal identifiers of the source and target points.

[0048] For the import of edge data, the client starts two producer threads, which are responsible for extracting the primary key sets of the source and target points in the edge data, querying the internalIds corresponding to these primary keys from the memory database according to the primary key sets, and mapping the internalIds of the points to the edge data to construct the second GQL statement, where the second GQL statement is used to write the edge data, and the constructed second GQL statement is placed in the second queue.

[0049] The client starts a consumer thread, which is responsible for obtaining the second GQL statement for writing edge data from the second queue, and sending the second GQL statement to the graph database service for actual writing of edge data. Through the consumer mode, when sending the second GQL statement to the graph database server, the thread does not need to synchronously wait for the query of the internalId of the source and target points and the construction of the second GQL statement. It can directly obtain the second GQL statement that has been processed by the producer from the second queue, avoiding a lot of synchronous waiting time and increasing the number of requests sent to the graph database server. At the same time, it avoids the overhead of multiple internal queries on the source and target points on the graph database server.

[0050] Taking redis as an example, the specific implementation code for creating the second producer is as follows: / / Create a producer val producerTasks = producerList.map { case (edgeGroupList) => Future { try { edgeGroupList.foreach { edges => / / Extract the primary key set of source and target points from the edge data set val nodePks = getPkNodes(edges) / / Query the internalId set corresponding to the primary key set from the memory database to obtain the mapping Map of primary key and internalId val nodePkToInternalIdMap = ThirdDbUtils.batchRead(jedisPool, nodePks.toList) / / Map the edges in batches using the primary key and internalId of the node. val (dirtyRecords, gql) = writer.getGqlForInitialImport(edges, nodeTypeMap) if (gql != null) { / / Put the write edge GQL statement into the queue queue.put((gql, edges.size - dirtyRecords.size)) } } finally In the above embodiment, the mapping relationship between the primary key and the internalId of the node managed by the graph database server is cached by referencing the memory database. When writing edge data, there is no need to query the internal identifiers of the source and target points from the graph database server. The internal identifiers of the source and target points can be obtained directly from the local memory database, thereby avoiding additional query operations of the graph database server when importing edge data, effectively reducing the number of communications between the client and the graph database server, and reducing network overhead.

[0051] In order to reduce synchronization time, the producer-consumer model is used to asynchronously process data when importing vertex and edge data.

[0052] Specifically, when importing point data, the operations of constructing the first GQL statement for writing the point and sending the first GQL statement request to the graph database server are executed by a separate thread. This part of the operation does not need to wait for the parsing response result and writing the mapping relationship to the memory database. After the graph database server returns the response result, the response result can be directly placed in the first queue. At the same time, the operations of parsing the response result and writing the mapping relationship to the memory database are also executed by a separate thread. The thread obtains the response result from the first queue for processing, and does not need to synchronously wait for the response of the graph database server.

[0053] When importing edge data, the operation of querying the corresponding internalId from the memory database according to the primary keys of the source and target points of the edge and constructing the second GQL statement is performed by two separate threads. After this part of the operation is processed, the generated second GQL statement can be directly placed in the second queue without waiting for the execution result of the second GQL statement on the graph database server. At the same time, the operation of obtaining the second GQL statement and sending the second GQL statement to the graph database server is also executed by a separate thread. This thread obtains the second GQL statement from the second queue and sends it to the graph database server. After the execution is completed, the next second GQL statement is immediately taken from the second queue and sent to the graph database server, without waiting for the previous query of the internal identifier from the memory database and the construction of the second GQL statement.

[0054] In this embodiment, this asynchronous processing method can maximize the use of the client's CPU resources without waiting for the return of IO. Therefore, asynchronous processing can reduce the synchronization waiting time, improve the overall import performance, improve the utilization rate of the client CPU, and significantly improve the efficiency of data import.

[0055] Example 2

[0056] like Figure 4 As shown, an optimization device for importing full data of a graph database based on a memory database according to Embodiment 2 of the present application includes: A first production module 100 is used to start a first producer thread, wherein the first producer thread is used to process source data and construct a first GQL statement for batch writing points, send the first GQL statement to the graph database server, and after receiving a response from the graph database server, put the response result into a first queue and continue to process the next batch of source data, wherein the response result includes the primary key and internal identifier of the point written in the current batch; A first consumption module 200 is used to start a first consumer thread, wherein the first consumer thread is used to obtain a response result from the first queue, iteratively parse the response result to obtain a mapping relationship between the primary key and the internal identifier of all points in the current batch, and write the mapping relationship into a memory database; The edge data writing module 300 is used to extract the primary key set of the source point and the target point from the edge data when writing the edge data, query the corresponding internal identifier from the memory database according to the primary key set, and construct a second GQL statement for writing the edge data according to the mapping relationship between the queried primary key and the internal identifier.

[0057] Specifically, the edge data writing module 300 includes: The second production module 301 is used to start a second producer thread, wherein the second producer thread is used to extract a primary key set of a source point and a target point in the edge data, query a corresponding internal identifier from the memory database according to the primary key set, and construct a second GQL statement for writing the edge data according to a mapping relationship between the queried primary key and the internal identifier, and put the constructed second GQL statement into a second queue; The second consumption module 302 is used to start a second consumer thread, wherein the second consumer thread is used to obtain a second GQL statement from the second queue, send the obtained second GQL statement to the graph database server and receive the execution result.

[0058] When importing vertex and edge data, the producer-consumer mode is used to process data asynchronously. This asynchronous processing method can maximize the use of the client's CPU resources and does not need to wait for IO returns. Therefore, asynchronous processing can reduce the synchronization waiting time, thereby improving the client CPU utilization and data import efficiency.

[0059] It should be noted that other specific implementation methods of the optimization device for full data import of a graph database based on an in-memory database in this embodiment can be found in the specific implementation methods of the optimization method for full data import of a graph database based on an in-memory database mentioned above. To avoid redundancy, they will not be repeated here.

[0060] Example 3

[0061] A computer-readable storage medium according to Embodiment 3 of the present application, wherein the computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing a method in any one of the implementations in Embodiment 1 of the present application; Among them, the computer-readable storage medium can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM); the computer-readable storage medium can store program code, and when the program stored in the computer-readable storage medium is executed by the processor, the processor is used to execute the steps of the method in any one of the implementation methods in Example 1 of the present application.

[0062] Example 4

[0063] like Figure 5 As shown, an electronic device involved in Embodiment 4 of the present application includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, a method in any one of the implementations in Embodiment 1 of the present application is implemented; Among them, the processor can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or an integrated circuit to execute relevant programs to implement the method in any one of the implementation methods in Example 1 of the present application.

[0064] The processor may also be an integrated circuit electronic device with signal processing capability. In the implementation process, each step of the method in any implementation of Embodiment 1 of the present application may be completed by an integrated logic circuit of hardware in the processor or by instructions in software form.

[0065] The above-mentioned processor may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the functions required to be performed by the unit included in the data processing device of the embodiment of the present application in combination with its hardware, or executes the method in any one of the implementation modes in the embodiment 1 of the present application.

[0066] The above are only preferred specific implementations of the present application; however, the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved ideas of the present application within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An optimization method for importing full data of a graph database based on an in-memory database, characterized in that: include: Start a first producer thread, wherein the first producer thread is used to process source data and construct a first GQL statement for batch writing points, send the first GQL statement to the graph database server, and after receiving a response from the graph database server, put the response result into a first queue and continue to process the next batch of source data, wherein the response result includes the primary key and internal identifier of the point written in the current batch; Start a first consumer thread, wherein the first consumer thread is used to obtain a response result from the first queue, iteratively parse the response result to obtain a mapping relationship between the primary key and the internal identifier of all points in the current batch, and write the mapping relationship into a memory database; When writing edge data, the primary key set of the source point and the target point is extracted from the edge data, the corresponding internal identifier is queried from the memory database according to the primary key set, and the second GQL statement for writing the edge data is constructed according to the mapping relationship between the queried primary key and the internal identifier.

2. According to claim 1, the optimization method for importing full data of a graph database based on a memory database is characterized in that: When the client imports point data, it configures in the first GQL statement that it allows the graph database server to return the primary key and internal identifier of the points written in batches to the client.

3. According to claim 1, the optimization method for importing full data of a graph database based on a memory database is characterized in that: When writing edge data: Start a second producer thread, wherein the second producer thread is used to extract a primary key set of a source point and a target point in the edge data, query a corresponding internal identifier from the memory database according to the primary key set, construct a second GQL statement for writing the edge data according to a mapping relationship between the queried primary key and the internal identifier, and put the constructed second GQL statement into a second queue; A second consumer thread is started, wherein the second consumer thread is used to obtain a second GQL statement from the second queue, send the obtained second GQL statement to the graph database server and receive an execution result.

4. The optimization method for importing full data of a graph database based on a memory database according to claim 1 is characterized in that: The first producer thread and the first consumer thread process data in an asynchronous manner.

5. The optimization method for importing full data of a graph database based on a memory database according to claim 3 is characterized in that: The second producer thread and the second consumer thread process data in an asynchronous manner.

6. An optimization device for importing full data of a graph database based on a memory database, characterized in that: include: A first production module is used to start a first producer thread, wherein the first producer thread is used to process source data and construct a first GQL statement for batch writing points, send the first GQL statement to the graph database server, and after receiving a response from the graph database server, put the response result into a first queue and continue to process the next batch of source data, wherein the response result includes the primary key and internal identifier of the point written in the current batch; A first consumption module, used to start a first consumer thread, wherein the first consumer thread is used to obtain a response result from the first queue, iteratively parse the response result to obtain a mapping relationship between the primary key and the internal identifier of all points in the current batch, and write the mapping relationship into a memory database; The edge data writing module is used to extract the primary key set of the source point and the target point from the edge data when writing the edge data, query the corresponding internal identifier from the memory database according to the primary key set, and construct a second GQL statement for writing the edge data according to the mapping relationship between the queried primary key and the internal identifier.

7. The optimization device for importing full data of a graph database based on a memory database according to claim 6, characterized in that: The edge data writing module comprises: A second production module is used to start a second producer thread, wherein the second producer thread is used to extract a primary key set of a source point and a target point in the edge data, query a corresponding internal identifier from the memory database according to the primary key set, and construct a second GQL statement for writing the edge data according to a mapping relationship between the queried primary key and the internal identifier, and put the constructed second GQL statement into a second queue; The second consumption module is used to start a second consumer thread, wherein the second consumer thread is used to obtain a second GQL statement from the second queue, send the obtained second GQL statement to the graph database server and receive the execution result.

8. The optimization device for importing full data of a graph database based on a memory database according to claim 7, characterized in that: The first producer thread and the first consumer thread process data in an asynchronous manner, and the second producer thread and the second consumer thread process data in an asynchronous manner.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing the method according to any one of claims 1 to 5.

10. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the method as claimed in any one of claims 1 to 5.

Images