Graph data importing method of graph database

Abstract

The invention relates to a graph data importing method of a graph database, and belongs to the technical field of indexing and storage of structured data. The method comprises the following steps: creating a graph data import task, carrying out id division on the data import task in sequence, then executing the data import task, after the first data import task is completed, stopping all uncompleted data import tasks, and taking the completed data import task as a standard data import task; carrying out conflict detection on uncompleted data import tasks, re-importing the tasks with detected conflicts according to set retry times and interval time, marking the standard data import tasks as successful after all conflict detection is completed, setting the version of the graph database as the numerical value id + 1 of the first task, waiting for the first task to take effect, and completing data import task processing; and then continuing to execute the data import task which is not completed, and repeating the steps until no data import task needs to be executed. According to the method, the atomicity of the batch parallel imported graph data of the graph database is ensured, and meanwhile, the system blockage is reduced.

Description

Technical Field

[0001] The present invention relates to a graph data import method for a graph database, and belongs to the technical field of indexing and storage of structured data (G06F16 / 22). Background Art

[0002] Graph data is a data structure composed of vertices and edges, where vertices represent entities and edges represent relationships between entities. A graph database is a database system that can store and manage graph data. Graph databases focus on optimizing the storage and query of graph data, and their practical applications often involve importing raw data from external heterogeneous data sources into graph databases. Especially in data-sensitive graph analysis businesses, the atomicity of graph data import is a key requirement, that is, the graph data import operation needs to ensure that all graph data are effective at the same time or all are ineffective, to avoid the intermediate state where some graph data are successfully imported but not effective.

[0003] Traditional graph database graph data import methods include native data import, temporary graph import, and large transaction import. Native data import does not consider the atomicity of batch import, that is, after some graph data is successfully imported, the relevant graph data will take effect immediately. This means that if the import operation eventually fails, there will be some dirty data in the graph database. Temporary graph import first imports the data into a temporary graph, and then renames the temporary graph to the original graph after success. Although this method ensures the atomicity of the import of the full amount of data, it does not support access to the original data and cannot support parallel import operations. Large transaction import submits the entire graph data batch as a large transaction. In theory, it can provide atomicity guarantees and support parallel import, but because the transaction is too large, the conflict detection overhead is high, and it will block online transactions and affect the performance of the business system.

[0004] Therefore, there is an urgent need for an atomic batch parallel import method with less blocking when importing graph data. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to reduce blocking while ensuring the atomicity of batch parallel import of graph data into a graph database.

[0006] The present invention proposes a technical solution to solve the above technical problem: a method for importing graph data into a graph database, comprising the following steps:

[0007] Step 1: Establish a graph data import task for importing the graph data into the graph database, wherein the graph data import task contains at least one graph data, and assigns a numerical id to all graph data import tasks in increasing order; and at the same time, set the number of retries and the retry interval of the graph data import task;

[0008] Step 2: Map the primary key of each graph data import task to obtain the primary key of each graph data import task;

[0009] Step 3: construct a conflict detector, which consists of a hash function and multiple Bloom filters;

[0010] Step 4: Execute the graph data import task. When the first graph data import task is completed, suspend the unfinished graph data import tasks; and use the first graph data import task as the first standard graph data import task.

[0011] Step 4.1: The hash function maps the primary key of the first standard graph data import task to the corresponding Bloom filter slot, and sets the corresponding Bloom filter slot to indicate that the primary key already exists;

[0012] Step 4.2: The conflict detector performs conflict detection on the graph data import tasks that have not been completed in step 4 respectively;

[0013] The hash function maps the primary key of the unfinished graph data import task to the corresponding Bloom filter slot; if the Bloom filter slot corresponding to any primary key of the unfinished graph data import task has been set, a conflict is detected, and all the data that has been imported into the graph database by the unfinished graph data import task is deleted; the unfinished graph data import task that has detected a conflict is re-executed according to the retry number and retry interval time, and if a conflict is still detected after the retry number is reached, it will not be re-executed;

[0014] If the Bloom filter slots corresponding to all primary keys of the unfinished graph data import tasks are not set, no conflict is detected, and the data that has been imported into the graph database by the unfinished graph data import tasks is retained;

[0015] Step 5: Mark the first standard graph data import task as successful, and set the version of the graph database to the numerical id of the first standard graph data import task + 1;

[0016] If the numerical id of the first standard graph data import task is the minimum numerical id, the version of the graph database is effective;

[0017] If the numerical id of the first standard graph data import task is not the minimum numerical id, the version of the graph database will not be effective until there is no graph data import task with a numerical id smaller than the numerical id of the standard graph data import task that needs to be executed to take effect again;

[0018] Step 6: Repeat steps 4-5, and when the second graph data import task is completed, suspend the unfinished graph data import task; and use the second graph data import task as the second standard graph data import task;

[0019] The first standard graph data import task in steps 4-5 is changed to the second standard graph data import task, and the rest is the same;

[0020] By analogy, steps 4-5 are repeated continuously until the last graph data import task is completed and the repetition stops, the last graph data import task is used as the last standard graph data import task, and steps 4-5 are completed. Furthermore, when querying the graph data in the graph database, the graph data in the first graph database version in descending order among all effective graph database versions is used as the queryable graph data.

[0021] The beneficial effects of the present invention are as follows: since a graph data import task is created for the graph data and a numerical id is assigned to the graph data import task in increasing order, whenever a graph data import task is completed, the import is paused and conflict detection is performed, and then the graph database version is validated by a sequential validation mechanism, which effectively ensures the atomicity of parallel batch import of graph data. In addition, a conflict detector composed of a hash function and multiple Bloom filters is used to perform conflict detection on the data primary key of the graph data import task. This detection scheme is efficient and avoids a large amount of performance overhead, reducing system congestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The graph data importing method of the graph database of the present invention is further explained below in conjunction with the accompanying drawings.

[0023] Figure 1 It is a schematic diagram of the conflict detector structure of the graph data import method of the graph database of the embodiment. DETAILED DESCRIPTION

[0024] Example

[0025] A method for importing graph data into a graph database in this embodiment includes the following steps:

[0026] Step 1: Create a graph data import task for importing graph data into the graph database. The graph data import task contains at least one graph data. All graph data import tasks are assigned numerical ids in increasing order. At the same time, set the number of retries and retry interval of the graph data import task.

[0027] In this embodiment, three graph data import tasks are established, and the numerical ids are assigned as 1, 2, and 3;

[0028] Set the number of retries for the graph data import task to 1 and the retry interval to 10s.

[0029] Step 2: Map the primary key of each graph data import task to obtain the primary key of each graph data import task;

[0030] The primary key (pk) of the graph data is the unique identifier of each node or edge in the graph data. This embodiment uses the prior art snowflake algorithm to generate different globally unique identifiers id as the primary key of the graph data.

[0031] Step 3: Construct a conflict detector, which consists of a hash function and multiple Bloom filters;

[0032] The schematic diagram of the conflict detector structure is as follows: Figure 1 As shown. A hash function is a mathematical function used to convert data (such as the primary key of a vertex or edge in the graph data) into an identifier of a fixed length, which is called a "hash value" or "hash value". A bloom filter is a probabilistic data structure used to check whether an element is in a set. Its characteristic is that it can efficiently check whether an element exists, but there may be "false positives", that is, it is judged that the element exists but is actually not in the set, but it will not be "missed". Based on this feature, the number of retries and the retry interval of the graph data import task are set to avoid "false positives";

[0033] The hash function calculates an integer value based on the input primary key, and then takes the value modulo the length of the Bloom filter's bit array, so that the calculation result is mapped to a valid slot in the bit array.

[0034] Step 4: Execute the graph data import task. When the first graph data import task is completed, suspend the graph data import tasks that have not been completed; and use the first graph data import task as the first standard graph data import task;

[0035] After executing three graph data import tasks for a period of time in this embodiment, all graph data in the graph data import task with a numerical id of 1 has been imported into the graph data, that is, the graph data import task has been completed, and the unfinished graph data import tasks 2 and 3 are suspended; the graph data import task 1 is used as the first standard graph data import task;

[0036] Step 4.1: The hash function maps the primary key of the first standard graph data import task to the corresponding Bloom filter slot, and sets the corresponding Bloom filter slot to indicate that the primary key already exists;

[0037] Step 4.2: The conflict detector performs conflict detection on the graph data import tasks that have not been completed in step 4;

[0038] The hash function maps the primary key of the unfinished graph data import task to the corresponding Bloom filter slot; if the Bloom filter slot corresponding to any primary key of the unfinished graph data import task has been set, a conflict is detected, and all the data that has been imported into the graph database by the unfinished graph data import task is deleted; the unfinished graph data import task that has detected a conflict is re-executed according to the retry count and retry interval time, and will not be re-executed if a conflict is still detected after the retry count is reached;

[0039] If the Bloom filter slots corresponding to all primary keys of the unfinished graph data import tasks are not set, no conflict is detected, and the data that has been imported into the graph database by the unfinished graph data import tasks is retained;

[0040] In this embodiment, the Bloom filter slot corresponding to one of the primary keys of graph data import task 2 has been set, indicating that graph data import task 2 conflicts with graph data import task 1. All data that has been imported into the graph database by graph data import task 2 is deleted, and graph data import task 2 is executed again after 10 seconds.

[0041] The Bloom filter slots corresponding to all primary keys of graph data import task 3 are not set, indicating that there is no conflict, and the data imported into the graph database by graph data import task 3 is retained.

[0042] Step 5: Mark the first standard graph data import task as successful, and set the version of the graph database to the numerical id of the first standard import task + 1;

[0043] If the numerical id of the standard graph data import task is the minimum numerical id, the version of the graph database will take effect;

[0044] If the numerical id of the standard graph data import task is not the minimum numerical id, the graph database version will not take effect until there is no graph data import task with a numerical id smaller than the numerical id of the standard graph data import task that needs to be executed and then it will take effect;

[0045] Since the numerical id of the first standard graph data import task is 1, which is the minimum numerical id, the version of the effective graph database is 2.

[0046] Step 6: Repeat steps 4-5. When the second graph data import task is completed, suspend the unfinished graph data import task; use the second graph data import task as the second standard graph data import task;

[0047] The first standard graph data import task in steps 4-5 is changed to the second standard graph data import task, and the rest is the same;

[0048] And so on, keep repeating steps 4-5 until the last graph data import task is completed and stop repeating. The last graph data import task is used as the last standard graph data import task, and steps 4-5 are completed.

[0049] The second graph data import task completed in this embodiment is the graph data import task with the numerical id of 3, and the graph data import task 2 that has been re-executed and not yet completed in step 4.2 is paused; and the graph data import task 3 is used as the second standard graph data import task;

[0050] Repeat steps 4-5. During this period, it is still detected that the Bloom filter slot corresponding to any primary key of graph data import task 2 has been set, that is, there is still a conflict after reaching the number of retries, so graph data import task 2 is no longer executed.

[0051] Since the graph data import task 2 is no longer executing the import, the numerical id of the second standard graph data import task is 3, which is the minimum numerical id, so the effective graph database version is 4.

[0052] Since there are no other graph data import tasks to be executed, that is, graph data import task 3 is the last graph data import task, the repetition is stopped.

[0053] At this time, a user client queries the data in the graph database. The effective graph database versions are 2 and 4. The graph data in the first graph database version in reverse order, that is, 4, is provided for the user client to query.

[0054] The above description is only a preferred embodiment of the present invention, but the present invention is not limited thereto. All equivalent replacements or equivalent changes of the concepts and technical solutions according to the present invention should be included in the protection scope of the present invention.

Claims

1. A method for importing graph data into a graph database, characterized in that The steps include: Step 1: Establish a graph data import task for importing the graph data into the graph database, wherein the graph data import task contains at least one graph data, and assigns a numerical id to all graph data import tasks in increasing order; and at the same time, set the number of retries and the retry interval of the graph data import task; Step 2: Map the primary key of each graph data import task to obtain the primary key of each graph data import task; Step 3: construct a conflict detector, which consists of a hash function and multiple Bloom filters; Step 4: Execute the graph data import task. When the first graph data import task is completed, suspend the graph data import tasks that have not been completed; and use the first graph data import task as the first standard graph data import task; Step 4.1: The hash function maps the primary key of the first standard graph data import task to the corresponding Bloom filter slot, and sets the corresponding Bloom filter slot to indicate that the primary key already exists; Step 4.2: The conflict detector performs conflict detection on the graph data import tasks that have not been completed in step 4 respectively; The hash function maps the primary key of the unfinished graph data import task to the corresponding Bloom filter slot; If the Bloom filter slot corresponding to any primary key of the unfinished graph data import task has been set, a conflict is detected, and all the data that has been imported into the graph database by the unfinished graph data import task is deleted; the unfinished graph data import task that has detected a conflict is re-executed according to the retry count and retry interval. If a conflict is still detected after the retry count is reached, it will not be re-executed; If the Bloom filter slots corresponding to all primary keys of the unfinished graph data import tasks are not set, no conflict is detected, and the data that has been imported into the graph database by the unfinished graph data import tasks is retained; Step 5: Mark the first standard graph data import task as successful, and set the version of the graph database to the numerical id of the first standard graph data import task + 1; If the numerical id of the first standard graph data import task is the minimum numerical id, the version of the graph database is effective; If the numerical id of the first standard graph data import task is not the minimum numerical id, the version of the graph database will not be effective until there is no graph data import task with a numerical id smaller than the numerical id of the standard graph data import task that needs to be executed to take effect again; Step 6: Repeat steps 4-5, and when the second graph data import task is completed, suspend the unfinished graph data import task; and use the second graph data import task as the second standard graph data import task; The first standard graph data import task in steps 4-5 is changed to the second standard graph data import task, and the rest is the same; And so on, steps 4-5 are repeated continuously until the last graph data import task is completed and the repetition stops, the last graph data import task is used as the last standard graph data import task, and steps 4-5 are completed.

2. The method for importing data into a graph database according to claim 1, characterized in that: When querying the graph data in the graph database, the graph data in the first graph database version in descending order among all valid graph database versions is used as the queryable graph data.

Images