GQL graph database data importing method and device

Abstract

The invention discloses a GQL graph database data import method and device, relates to the technical field of graph database data import, and solves the problems that in the prior art, a GQL graph database cannot realize breakpoint resume; in order to solve the problems of repeated data import, computing resource waste and easy data loss caused by the fact that data needs to be imported again after a task of importing the data into a GQL graph database is interrupted, the method comprises the following steps: partitioning a data source, obtaining a data import completion state of a partition in a last import task, performing batch data import on the partition which is not imported, and obtaining a data import completion state of the partition in the last import task; and the task state information of the completely imported partitions is recorded, so that breakpoint resuming is facilitated, the breakpoint resuming function of data importing of the GQL graph database is realized, repeated importing caused by task importing interruption is avoided, the data importing efficiency and stability are remarkably improved, and the data importing efficiency is greatly improved especially in a distributed environment. And the problems of large data volume, network instability and the like can be effectively solved.

Description

Technical Field

[0001] The present application relates to the technical field of graph database data import, and in particular to a GQL graph database data import method and device. Background Art

[0002] With the development of big data and distributed computing technologies, graph databases have shown significant advantages in processing complex relational data. However, in practical applications, the data import process of distributed graph databases often faces problems such as large data volume, unstable network, and diverse data sources, which makes the import task easy to fail. Traditional data import methods need to restart the import after the task is interrupted, which not only wastes a lot of computing resources for large-scale data sets, but also may cause data to be imported repeatedly or lost, resulting in poor stability, poor performance, and poor accuracy. Therefore, how to achieve breakpoint resumption during data import, avoid repeated import, and improve import efficiency has become a technical problem that needs to be solved urgently.

[0003] The Chinese patent with publication number CN113704277A discloses a database-based breakpoint resume method and related devices, which use the storage capacity of the database to record the breakpoint position, where the breakpoint position is the row number of the database source table or the array number of the database source table. The database transaction is executed by obtaining the database transaction to be processed, reading the breakpoint record, and identifying the corresponding processing position from the breakpoint position, and updating the breakpoint position identifier at each submission operation, which affects the database performance due to frequent database access. In addition, this patent is for transactional databases. The import of an import batch of data is transactional, and either all imports succeed or all imports fail. It is not suitable for GQL graph databases without transaction capabilities. Summary of the invention

[0004] The purpose of the present application is to overcome the problems in the prior art that the GQL graph database cannot realize breakpoint resumption, and the task of importing data into the GQL graph database needs to be restarted after the data is interrupted, resulting in repeated data import, waste of computing resources and easy data loss, and to provide a GQL graph database data import method and device.

[0005] In a first aspect, a method for importing data into a GQL graph database is provided, comprising:

[0006] Partition the data source according to the preset partitioning strategy;

[0007] Read the task status information of the last import task under the specified path to obtain the data import completion status of the partition;

[0008] Import the data in the partitions whose data import completion status is incomplete into the GQL graph database in batches;

[0009] The task status information of the partition that has been imported is stored in the specified path, and the task status information includes: source data index, vertex edge type, partition ID, data import completion status and import status data.

[0010] In some possible implementations, the method further includes: in the process of batch importing data in partitions into the GQL graph database, splitting the batch data that failed to be imported into single data for concurrent retry of import, and recording the single data that failed to be imported.

[0011] In some possible implementations, data sources are identified and classified, and memory partitions are performed based on characteristics and data volumes of the data sources.

[0012] In some possible implementations, the import status data includes: the number of source data records, the amount of point data that has been imported, the amount of edge data that has been imported, the amount of point data that has failed to be imported, the amount of edge data that has failed to be imported, the amount of point data that has been imported and has an impact on the GQL graph database, and the amount of edge data that has been imported and has an impact on the GQL graph database.

[0013] In some possible implementations, batch importing data in partitions whose data import completion status is incomplete into the GQL graph database includes:

[0014] Read the data in the partition whose import completion status is incomplete;

[0015] Convert the read data into GQL statements for batch import;

[0016] The GQL statement is sent to the graph database service, and the graph database service executes the GQL statement to import data.

[0017] In some possible implementations, during the process of importing data into the GQL graph database, the user is shown the amount of data completed in the last import task at the start of the data import, the import progress of the current import task, and the number of queries per second.

[0018] In a second aspect, a GQL graph database data import device is provided, comprising:

[0019] Partitioning module, used to partition the data source according to the preset partitioning strategy;

[0020] The last breakpoint status reading module is used to read the task status information of the last import task under the specified path to obtain the data import completion status of the partition;

[0021] The breakpoint resume module is used to batch import data from partitions whose data import completion status is incomplete into the GQL graph database;

[0022] The task status recording module is used to store the task status information of the partition that has been imported to the specified path. The task status information includes: source data index, vertex edge type, partition ID, data import completion status and import status data.

[0023] In some possible implementations, it also includes: a split retry module, which is used to split the batch data that failed to be imported into single data for concurrent retry import during the process of batch importing data in partitions into the GQL graph database, and record the single data that failed to be imported.

[0024] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes steps for executing the method in any one of the implementations of the first aspect described above.

[0025] In a fourth aspect, an electronic device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements a method as in any one of the implementations in the first aspect above.

[0026] This application has the following beneficial effects:

[0027] 1. This application implements the breakpoint-resume function of GQL graph database data import by partitioning the source data in memory and recording the completion status and import progress of each partition, avoiding repeated imports caused by interruptions in import tasks, and significantly improving the efficiency and stability of data import. Especially in a distributed environment, it can effectively deal with problems such as large data volume and unstable network, providing strong technical support for the application of GQL graph database.

[0028] 2. This application handles batch errors by splitting batch data into single data and then retrying them concurrently, thereby minimizing the amount of failed data and further improving the efficiency of data import. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 It is a flowchart of the GQL graph database data import method of Example 1 of the present application;

[0032] Figure 2 It is a structural block diagram of a GQL graph database data import device of Example 3 of the present application;

[0033] Figure 3 It is a schematic diagram of the internal structure of the electronic device of Example 5 of the present application.

[0034] Reference numerals:

[0035] 100, partition module; 200, last breakpoint status reading module; 300, breakpoint resume module; 400, task status recording module. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Example 1

[0038] like Figure 1 As shown, a method for importing data into a GQL graph database according to Embodiment 1 of the present application includes:

[0039] S100: partition the data source according to a preset partitioning strategy.

[0040] Specifically, different data sources are identified and classified, and memory partitions are performed based on the characteristics and data volume of the data source. For example, for a file data source stored on the HDFS file system, each BLOCK is a partition (BLOCK is a concept in HDFS, and files are divided into different BLOCK blocks by HDFS and stored on different data nodes of the distributed file system); for data stored in the graph database NebulaGraph, each part of the NebulaGraph database point type or edge type data is a partition. When the data in the data source is subsequently imported into the GQL graph database, the data will be read and processed separately according to the partitions.

[0041] S200, reading the task status information of the last import task under the specified path to obtain the data import completion status of the partition.

[0042] Specifically, in the configuration file of the import tool, the user can configure a specified path or a new path as the specified address for storing the task status information of the last imported task. When the import tool reads the task status information of the last imported task, it obtains the task status information of the last imported task by reading the file in the specified path and parses the JSON data in the file into the task status structure defined in the import tool. The implementation code is as follows:

[0043] def readTaskInfo(spark: SparkSession, checkPointConfig:CheckPointConfigEntry): List[ExchangeTaskInfo] = {

[0044] val targetPath = new Path(checkPointConfig.checkPointPath)

[0045] var fileSystem: FileSystem = null

[0046] / / Generate a file system object based on the storage path for breakpoint resume configured by the user in the configuration file

[0047] if (checkPointConfig.checkPointPath.startsWith("file: / / ")) {

[0048] fileSystem = FileSystem.getLocal(new Configuration())

[0049] } else {

[0050] fileSystem = targetPath.getFileSystem(new Configuration())

[0051] }

[0052] / / The path does not exist, that is, there is no status information that can be resumed at the current breakpoint

[0053] if (!fileSystem.exists(targetPath)) {

[0054] return List()

[0055] }

[0056] / / The breakpoint-resume status file under the path does not exist, that is, there is currently no status information that can be resumed.

[0057] if (fileSystem.listStatus(targetPath).length == 0) {

[0058] return List()

[0059] }

[0060] import spark.implicits._

[0061] / / Read the breakpoint resume file and convert the JSON format data into ExchangeTaskInfo breakpoint resume task information

[0062] val checkPointData = spark.read.json(checkPointConfig.checkPointPath).as[ExchangeTaskInfo]

[0063] checkPointData

[0064] .collect()

[0065] .toList

[0066] }

[0067] Among them, the task status structure includes: source data index, vertex and edge type, partition id, completion, and import status data. The import status data includes: the number of source data records (number of rows), the amount of point data that has been imported, the amount of edge data that has been imported, the amount of point data that has failed to be imported, the amount of edge data that has failed to be imported, the amount of point data that has been imported and has an impact on the graph database, and the amount of edge data that has been imported and has an impact on the graph database.

[0068] S300 batch-imports the data in the partitions whose data import completion status is incomplete into the GQL graph database.

[0069] Specifically, the user can choose whether to enable breakpoint resume in the configuration file. If the user enables breakpoint resume in the configuration file, when the import task is restarted, the breakpoint status structure is constructed according to the breakpoint status of the previous import task: Map[sourceIndex -> Map[partitionId -> stats]], which stores the data source index and the status of the current data source import task. The import task status is also a key-value mapping relationship, that is, the import status corresponding to each partition of the current data source. The import status includes the number of source data records (number of rows), the amount of point data that has been imported, the amount of edge data that has been imported, the amount of point data that has failed to be imported, the amount of edge data that has failed to be imported, the amount of point data that has been imported and has an impact on the graph database, and the amount of edge data that has been imported and has an impact on the graph database. The partitions of the data source stored in this structure are the completed partition data. Then this import task will not import the completed partition data, but only the unfinished partition data. That is to say, in this application, the breakpoint resume is performed on a partition basis. This application uses data partitions as the granularity, and realizes the breakpoint resume function by recording the import status and progress of each partition, thereby avoiding repeated imports and data loss, and avoiding frequent recording of import status.

[0070] If the user does not enable breakpoint resume in the configuration file, the above breakpoint status structure is empty, which means that all partitions of the data source are considered to be in an unfinished state. This ensures that the subsequent data import processing logic remains consistent and can be imported based on the breakpoint status structure. When the breakpoint status structure is empty, all partition data of the data source needs to be imported without breakpoint resume.

[0071] In this embodiment, in the breakpoint resume import task, it is determined whether a partition of a data source has completed import according to the above-mentioned breakpoint status structure. If completed, the partition is skipped and data reading, processing and importing into the database are not performed. If not completed, data reading, processing and importing into the database of a partition of the data source are performed.

[0072] Secondly, during the breakpoint-resume process, by printing the amount of data completed by the last import task at the start of the breakpoint, the import progress of the current task, and the import qps (i.e., the number of queries per second), and by reading the partition status information in the memory of each job of the distributed import task in real time, the current import progress is printed to ensure that users can understand the progress of the import task in real time.

[0073] S400. Store the task status information of the partition that has been imported to a specified path, where the task status information includes: source data index, vertex edge type, partition ID, data import completion status, and import status data.

[0074] Specifically, during the import process of this task, the source data is partitioned according to the preset partitioning strategy, and the amount of source data processed by the current partition, the amount of point data that has been imported, the amount of edge data that has been imported, the amount of point data that has failed to be imported, the amount of edge data that has failed to be imported, the amount of point data that has been imported and has an impact on the graph database, and the amount of edge data that has been imported and has an impact on the graph database are cumulatively recorded in the partition of each data source. After all the data in the current partition is processed, the current source data index, point edge type, partition ID, data import completion status, and import status data are recorded as task status information to the specified path set by the user in the configuration file, so that if the import task is accidentally interrupted, the task status information of the import task can be read through the specified path after the next import task is started, thereby identifying the partitions that have been imported and those that have not been imported.

[0075] In this embodiment, in order to minimize the data that fails to be imported, in the process of batch importing the data in the partition into the GQL graph database, the batch data that failed to be imported is split into single data for concurrent retry import. The single data that failed to be imported again is recorded in the memory, and when the failed data stored in the memory reaches a certain number, it is flushed to the disk. This application supports batch data import, and when a batch data import failure is detected during the import process, the batch data is split into single data for concurrent import, minimizing the amount of import failure data caused by dirty data.

[0076] After the local import is completed, the single data that failed to be imported will be recorded in the form of a GQL statement in the specified directory configured by the user in the configuration file, for example: in the app_id / failed directory of the error.path directory configured in the configuration file. This embodiment also provides a re-direction function, allowing the user to specify the task id to be re-directed, and re-import the data GQL file of the failed directory under the specified app_id in the error.path directory, thereby improving the performance and reliability of the import by only importing failed data.

[0077] In this embodiment, the data partition is used as the granularity for breakpoint resumption to avoid frequent recording of completed progress status and improve the performance of data import. In addition, for GQL graph databases that do not support transaction functions, different data writing modes can be selected according to business needs. For example, if the data already existing in the GQL graph database is not written again, the INSERT OR IGNORE mode can be configured in the configuration file. If the data already existing in the GQL graph database is overwritten, the INSERT OR REPLACE mode can be configured in the configuration file. Therefore, the breakpoint resumption of this embodiment does not rely on the transaction capabilities of the graph database.

[0078] Example 2

[0079] The application process of the present application is described with a specific embodiment. Assume that there is a person.csv file of size 100M containing 1 million data items that needs to be imported into the GQL graph database. And assume that the import has been executed and the task is suspended midway, that is, the import failure is simulated, and the GQL graph database data import method of the present application is adopted, including:

[0080] S100, the user configures a data source. If there is only one data source, person.csv, in the configuration file, the index of the data source is 0; the person.csv file is divided into 10 partitions according to the data size, and each partition contains 10M data.

[0081] S200, in the configuration file, the user configures the specified path for storing task status information as / tmp / checkpoint / , then the import tool reads the path, under which the task status information of the last import task is recorded, and the read status information is converted into an ExchangeTaskInfo structure. In this import task, there will be a log output indicating the amount of data that has been completed in the last import.

[0082] S300, convert the information of the ExchangeTaskInfo structure into a Map key-value pair structure, namely:

[0083] 0->{(1->Stats),(2->Stats),(3->Stats)}.

[0084] When executing the import, when the import tool processes the data of partition 1, partition 2, and partition 3, it knows based on the Map key-value pair structure that the data import of these three partitions has been completed, and then skips the data reading, data processing, and data import of these three partitions.

[0085] When the import tool processes the data of partitions 4, 5, 6, 7, 8, 9, and 10, since these partitions have not yet been imported, these partitions will read, process, and import data normally.

[0086] S400, the task status information of the three partitions 1-3 has been recorded in the path / tmp / checkpoint / . For the seven partitions 4-10, when a partition is written, the task status information of the partition's import completion will continue to be recorded in the path configured by the user in the configuration file: / tmp / checkpoint / .

[0087] During the data import process, failed batches of data will be split into individual records for concurrent retry import. At this time, the individual records that failed to be imported will be recorded in the path configured by the user in the configuration file. For example, if the user configures error.path to be / tmp / errors, the GQL statements that failed to be imported will be stored in the / tmp / errors / app-2024010112345-0001 / failed / directory. These stored failed statements can also be loaded and imported again through the import tool.

[0088] Example 3

[0089] like Figure 2 As shown, a GQL graph database data import device involved in Example 3 of the present application includes:

[0090] A partitioning module 100 is used to partition the data source according to a preset partitioning strategy;

[0091] The last breakpoint status reading module 200 is used to read the task status information of the last import task under the specified path to obtain the data import completion status of the partition;

[0092] The breakpoint resume module 300 is used to batch import the data in the partitions whose data import completion status is incomplete into the GQL graph database;

[0093] The task status recording module 400 is used to store the task status information of the partition that has been imported to a specified path, wherein the task status information includes: source data index, vertex edge type, partition ID, data import completion status and import status data.

[0094] In a further embodiment, a split-retry module is also included, which is used to split the batch data that failed to be imported into single data for concurrent retry import during the process of batch importing data in partitions into the GQL graph database, and record the single data that failed to be imported.

[0095] It should be noted that other specific implementations of the GQL graph database data import device in this embodiment can refer to the specific implementations of the above-mentioned GQL graph database data import method. In order to avoid redundancy, they will not be repeated here.

[0096] Example 4

[0097] A computer-readable storage medium according to Embodiment 4 of the present application, wherein the computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing a method in any one of the implementations in Embodiment 1 of the present application;

[0098] Among them, the computer-readable storage medium can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM); the computer-readable storage medium can store program code, and when the program stored in the computer-readable storage medium is executed by the processor, the processor is used to execute the steps of the method in any one of the implementation methods in Example 1 of the present application.

[0099] Example 5

[0100] like Figure 3 As shown, an electronic device involved in Embodiment 5 of the present application includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, a method in any one of the implementations in Embodiment 1 of the present application is implemented;

[0101] Among them, the processor can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the method in any one of the implementation methods in Example 1 of the present application.

[0102] The processor may also be an integrated circuit electronic device with signal processing capability. In the implementation process, each step of the method in any implementation of Embodiment 1 of the present application may be completed by an integrated logic circuit of hardware in the processor or by instructions in software form.

[0103] The above-mentioned processor may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the functions required to be performed by the unit included in the data processing device of the embodiment of the present application in combination with its hardware, or executes the method in any one of the implementation modes in the embodiment 1 of the present application.

[0104] The above are only preferred specific implementations of the present application; however, the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved ideas of the present application within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for importing data into a GQL graph database, characterized in that: include: Partition the data source according to the preset partitioning strategy; Read the task status information of the last import task under the specified path to obtain the data import completion status of the partition; Import the data in the partitions whose data import completion status is incomplete into the GQL graph database in batches; The task status information of the partition that has been imported is stored in the specified path, and the task status information includes: source data index, vertex edge type, partition ID, data import completion status and import status data.

2. The GQL graph database data import method according to claim 1, characterized in that: Also includes: When importing data in partitions into a GQL graph database in batches, the batch data that failed to be imported is split into individual pieces of data for concurrent retry import, and the individual pieces of data that failed to be imported are recorded.

3. The GQL graph database data import method according to claim 1, characterized in that: Identify and classify data sources, and perform memory partitioning based on the characteristics and data volume of the data source.

4. The GQL graph database data import method according to claim 1, characterized in that: The import status data includes: the number of source data records, the amount of point data that has been imported, the amount of edge data that has been imported, the amount of point data that has failed to be imported, the amount of edge data that has failed to be imported, the amount of point data that has been imported and has an impact on the GQL graph database, and the amount of edge data that has been imported and has an impact on the GQL graph database.

5. The GQL graph database data import method according to claim 4, characterized in that: Import the data in the partitions whose data import completion status is incomplete into the GQL graph database in batches, including: Read the data in the partition whose import completion status is incomplete; Convert the read data into GQL statements for batch import; The GQL statement is sent to the graph database service, and the graph database service executes the GQL statement to import data.

6. The method for importing GQL graph database data according to any one of claims 1 to 5, characterized in that: During the data import process into the GQL graph database, the user is shown the amount of data completed in the last import task at the start of the data import, the import progress of the current import task, and the number of queries per second.

7. A device for importing data into a GQL graph database, characterized in that: include: Partitioning module, used to partition the data source according to the preset partitioning strategy; The last breakpoint status reading module is used to read the task status information of the last import task under the specified path to obtain the data import completion status of the partition; The breakpoint resume module is used to batch import data from partitions whose data import completion status is incomplete into the GQL graph database; The task status recording module is used to store the task status information of the partition that has been imported to the specified path. The task status information includes: source data index, vertex edge type, partition ID, data import completion status and import status data.

8. The GQL graph database data import device according to claim 7, characterized in that: Also includes: The split-retry module is used to split the batch data that failed to be imported into single data for concurrent retry import during the process of batch importing data in partitions into the GQL graph database, and record the single data that failed to be imported.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing the method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the method as claimed in any one of claims 1 to 6 when executed by the processor.

Images