Method and system for importing data of graph database, electronic device, and medium

Abstract

Provided in the present invention are a method and system for importing data of a graph database, an electronic device, and a medium. The system comprises an import task initiation module, an import task execution module and an import task monitoring module, wherein the import task initiation module is used for: configuring graph database information, configuring a graph data format conversion script, reading a data file row by row, and calling an API to submit an import task; the import task execution module is used for: receiving the import task by means of the API, allocating the task to a data import cluster, reading graph database configuration information, reading a data dynamic processing script, analyzing data to obtain a graph database insertion statement, and executing the insertion statement to write the data into a graph database; and the import task monitoring module is used for: collecting import execution information, summarizing the data, and generating an import task execution condition report. The method of the present invention supports efficient import of large-scale graph data, is simple to implement, supports data files in different formats, and can allow a user to check import progress at any time.

Description

Method, system, electronic device, and medium for importing graph database data Technical Field

[0001] The present application relates to the field of graph data processing, and in particular to a method, system, electronic device, and medium for importing graph database data. Background Art

[0002] With the rapid development of the Internet, graph databases are attracting more and more attention from enterprises. Compared with traditional relational databases, graph databases have more efficient and faster data processing capabilities and more efficient management capabilities for massive and complex data.

[0003] As graph processing capabilities gradually improve, the scale of graph databases is also getting larger and larger. For graph databases with hundreds of millions of points and billions of edges, how to efficiently import data into the graph database is an extremely challenging task.

[0004] In addition, due to the complexity of graph structure and the lack of a unified graph data format standard, these graph data formats are different, which increases the complexity of importing these graph data into the graph database.

[0005] In the related art, different data sources are converted into data sets in a unified column format through a data access module, and then the data input format, conversion rules, and data output format are defined through a parameter configuration module to realize the import of data from different data sources and in different formats. For the data source types already supported by the system, this method does not require a lot of coding to support data in different formats. However, when a new data source needs to be connected, the data access module code needs to be modified to realize the conversion of the new data source format to the platform's unified column format. Moreover, this method does not consider solving the performance issues of data conversion and import execution when importing large amounts of data.

[0006] Another approach (see patent publication number: CN 114647689 A) involves converting data into a file in a pre-set format required by the graph database. The pre-set file is then loaded into the graph database's data storage directory. The pre-set file in the data storage directory is then imported into the graph space using commands provided by the graph database. This approach can address resource preemption and service unavailability issues associated with large-scale imports in distributed graph databases, but it suffers from the following issues:

[0007] (1) The cost of writing a data format processing program is high, especially when the data file is large, and a distributed processing program is needed to improve the speed of data format processing.

[0008] (2) Poor versatility. Different graph databases require different preset file formats, and not all graph databases provide methods for loading preset format data files.

[0009] In addition, another method is to implement parallel import through distributed computing frameworks such as Spark, or to write a set of strategies to enable the master node to distribute the edge files to each node, and each node to convert the format and import the data stored in the node. Take NebulaGraph Exchange, the official import tool provided by the nebula graph database, as an example. This tool has built-in import methods for 13 common data sources. To use this tool, you need to install Spark and configure the environment first, and execute the officially provided compiled exchange jar package through the sprak-submit command. If you want to support the import of new data sources, you need to modify the exchange code according to the data format and recompile it. This method uses the Spark tool to implement distributed import of graph data, which improves the efficiency of import, but there are several problems:

[0010] (1) The import task initiator needs to install the Spark tool and perform complex configuration.

[0011] (2) Poor versatility: When adding a new data source, the Spark task code needs to be modified and repackaged.

[0012] (3) Submitting the entire import task at once through the spark-submit command makes debugging difficult, especially when there is some non-standard data in the data file, which cannot be discovered and intervened in real time during the import process.

[0013] (4) Since the entire import task is submitted at once, if the task is unexpectedly interrupted, the import can only be restarted. For ultra-large-scale graph data import tasks, the cost is very high.

[0014] (5) It is inconvenient to check the import progress.

[0015] Summary of the Invention

[0016] In response to the shortcomings of the existing technology, the present invention provides a graph database data import method, system, electronic device, and medium based on task submission, task distribution, data parsing through dynamic scripts, and real-time viewing of import progress.

[0017] According to a first aspect of an embodiment of the present invention, a graph database data import system is provided, comprising:

[0018] The import task initiation module is used to configure graph database information, configure graph data format conversion scripts, read data files line by line, and call the API to submit the import task;

[0019] The import task execution module is used to receive import tasks through the API and distribute the tasks to the data import cluster through the message server;

[0020] The graph data import cluster includes multiple servers, each of which is used to parse the distributed task data using a data processing dynamic script according to the graph data format conversion script to obtain a graph database insert statement, and then execute the insert statement based on the graph database information to write the data into the graph database;

[0021] The import task monitoring module is used to collect import execution data, summarize the import data in real time, and generate import task execution reports.

[0022] According to a second aspect of an embodiment of the present invention, a method for importing data into a graph database is provided, the method comprising:

[0023] Step S1: Initiate the import task module, configure the graph database information, configure the data format conversion script, read the data file line by line, and call the API to submit the import task;

[0024] In step S2, the import task execution module receives the import task through the API, and distributes the task to the graph data import cluster through the message server. The servers in the graph data import cluster use the data processing dynamic script to parse the distributed task data according to the graph data format conversion script to obtain the graph database insert statement, and then execute the insert statement according to the graph database information to write the data into the graph database.

[0025] Step S3: The import task monitoring module collects import execution status data, summarizes the import status data in real time, and generates an import task execution status report.

[0026] Furthermore, the step S1 includes:

[0027] Step S101: The import task initiation module configures the graph database information, including the import task ID, graph database IP address, user name, and graph space name, and saves it to the cloud configuration server;

[0028] Step S102: Configuring the graph data format conversion script, including importing the task ID and the data format conversion script content, and saving it to the cloud configuration server;

[0029] Step S103: Read the data file line by line;

[0030] Step S104: Call the API to submit the import task. The API input parameters are the import task ID, data type, data label, and data content.

[0031] Furthermore, in step S101 and step S102, the configuration server sends the graph database information and the graph data format conversion script to the memory of the graph data import cluster server.

[0032] Furthermore, in step S104, the API is called asynchronously through a message queue.

[0033] Furthermore, the step S2 includes:

[0034] Step S201: Receive the graph data import task transmitted by the import task initiation module through the data import API.

[0035] Step S202: Send the import task to the message server.

[0036] Step S203: The message server distributes the task to a server with a lower load in the data import cluster for processing.

[0037] Step 204: The data import program parses the import task message to obtain the import task ID, data type, data tag, and data content.

[0038] Step 205: Read the data dynamic processing script from the configuration server according to the data import task ID.

[0039] Step 206: Use the data dynamic processing script to convert the data content into a format to obtain a graph database insert statement.

[0040] Step 207: Read the configured graph database information from the configuration server, execute the insert statement according to the configured graph database information, and write the data into the graph database.

[0041] Furthermore, step S3 includes:

[0042] Step S301: After completing a batch of data import tasks, the data import cluster sends the import execution status information to the import task monitoring module.

[0043] Step S302: The import task monitoring module summarizes the import status data in real time.

[0044] Step S303: The import task monitoring module generates an import task execution status report.

[0045] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned graph database data import method.

[0046] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the above-mentioned graph database data import method is implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The import task execution module provides services through API, making the import task initiation module simple to operate without having to pay attention to the details of import execution;

[0049] 2. The import task execution module distributes the import task to the data import cluster through the message server, realizing distributed import and high import efficiency;

[0050] 3. By configuring dynamic data parsing scripts, support for new data sources can be achieved without modifying the code;

[0051] 4. By collecting and summarizing the import task execution information through the import task monitoring module, real-time query of the task execution status can be realized.

[0052] 5. The present invention first reads data through the import task initiation module, and then converts and imports the data format through the import task execution module. When adding a new data source, there is no need to modify the code of the import task execution module. Data conversion is more flexible and versatile through dynamic script execution. The import task execution module is the core part of the system and can be deployed into a stable distributed service system, which has good industrial application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] FIG1 is a flow chart of a method for importing graph database data using a graph database data import system according to an embodiment of the present invention;

[0055] FIG2 is a schematic diagram of a flow chart of a graph database data import initiation module according to an embodiment of the present invention;

[0056] FIG3 is a schematic diagram of a flow chart of a graph database data import execution module according to an embodiment of the present invention;

[0057] FIG4 is a schematic diagram of a process for configuring and executing a dynamic data parsing script according to an embodiment of the present invention;

[0058] FIG5 is a schematic diagram of a process of importing graph database data into a monitoring module according to an embodiment of the present invention;

[0059] FIG6 is a flow chart of checking the import progress of a graph database import monitoring module according to an embodiment of the present invention;

[0060] 7 is a schematic diagram of a flow chart of querying import execution detailed information by a graph database import monitoring module according to an embodiment of the present invention;

[0061] FIG8 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0063] As shown in FIG1 , the present invention provides a graph database data import system, including:

[0064] Import the task initiation module, import the task execution module, import graph data into the cluster, and import the task monitoring module.

[0065] The system of the present invention supports efficient import of large-scale graph data, and the implementation method is simple. It supports data files in different formats and the import progress can be checked at any time.

[0066] It should be noted that the graph data import execution module mentioned in the present invention is responsible for utilizing the graph data import cluster to efficiently complete the graph data import task submitted by the client.

[0067] It should be noted that the graph data import initiation module mentioned in the present invention mainly completes the task of reading the imported data and sends the read data to the graph data import execution module through the API for processing.

[0068] It should be noted that the import task monitoring module mentioned in the present invention is mainly responsible for collecting and summarizing task execution information during the import process and generating an import status report for viewing.

[0069] FIG1 is a flow chart of a method for importing graph database data using a graph database data import system according to an embodiment of the present invention. As shown in FIG1 , the method includes the following steps:

[0070] Step S1: Initiate the import task module, configure the graph database information, configure the graph data format conversion script, read the data file line by line, and call the API to submit the import task;

[0071] In step S2, the import task execution module receives the data import task through the API, and distributes the task to the graph data import cluster through the message server. The servers in the graph data import cluster use the data processing dynamic script to parse the distributed task data according to the graph data format conversion script to obtain the graph database insert statement, and then execute the insert statement according to the graph database information to write the data into the graph database.

[0072] In step S3, the import task monitoring module collects import execution status data, summarizes the import status data in real time, and generates an import task execution status report.

[0073] As shown in FIG2 , step S1 specifically includes the following sub-steps:

[0074] Step S101: The import task initiation module configures the graph database information, including the import task ID, graph database IP address, user name, and graph space name, and saves this information to the configuration server;

[0075] In some implementation cases, the configuration server sends the configured graph database information to the server memory of the graph data import cluster to avoid the graph data import cluster having to access the configuration server to obtain database configuration information every time it performs an import, thereby reducing the performance of data import.

[0076] Step S102: Configuring a graph data format conversion script, including the import task ID and the data format conversion script content, and saving it to the configuration server;

[0077] In some implementation cases, the configuration server sends the data conversion script to the server memory of the graph data import cluster to avoid the graph data import cluster having to access the configuration server to obtain the data conversion script content every time it performs data format conversion, thereby reducing data import performance.

[0078] Step S103: Read the data file line by line;

[0079] In some implementation cases, in order to improve performance, batch or multi-threading methods are used to speed up the reading of data files. Furthermore, distributed file reading can be achieved by placing files on multiple servers.

[0080] In some implementation cases, the data source is a non-local file such as a relational database (such as MySQL), a big data storage system (such as Hive), a messaging system (such as Kafka), etc. The data reading API corresponding to the data source can be used to query and traverse the data and obtain data line by line.

[0081] Step S104: Call the API to submit the import task. The API input parameters are the import task ID, data type, data label, and data content.

[0082] In some implementation cases, the data import server also provides a batch import API, which supports importing multiple rows of data at one time.

[0083] In some implementation cases, to avoid performance bottlenecks in data import APIs, the APIs are designed to support asynchronous calls and higher call frequencies through message queues.

[0084] As shown in 3, step S2 specifically includes the following sub-steps:

[0085] Step S201: Receive the graph data import task transmitted by the import task initiation module through the data import API.

[0086] In some implementation cases, in order to prevent the API from being abused and writing false data, it is necessary to set up an API signature mechanism.

[0087] In some implementation cases, the API is designed as a RESTfull-style HTTP interface that can support calls from clients in different languages.

[0088] In some implementation cases, the API is designed to import a local SDK that can be directly called by the task initiation module to reduce the overhead caused by network calls.

[0089] Step S202: Send the graph data import task to the message server.

[0090] Step S203: The message server distributes the task to the servers in the graph data import cluster for processing.

[0091] In some implementation cases, the message distribution mode of the message server is designed to be a simple and efficient hash mode, which is suitable for the case where the configurations of the imported cluster servers are basically the same.

[0092] In some implementation cases, the message distribution mode of the message server is designed to be dynamically adjusted according to the server pressure, which is suitable for situations where the configuration of the imported cluster servers is different or the load pressure is different, and the message distribution is given priority to the servers with lower load.

[0093] Step 204: The server that receives the task parses the import task message through the data import program to obtain the import task ID, data type, data tag, and data content.

[0094] In some implementation cases, the import task ID is generated using a snowflake algorithm or a UidGenerator program to ensure global uniqueness.

[0095] In some implementation cases, the data type parameter indicates whether the data is point data or edge data, and the data label indicates the type of node (such as student, teacher) or the type of edge (such as teaching, course selection).

[0096] Step 205: The server reads the data format conversion script from the configuration server according to the data import task ID.

[0097] In some implementation cases, the data format conversion script is sent to the server memory of the import cluster through the configuration server.

[0098] Step 206: The server uses the data dynamic processing script according to the data format conversion script to convert the data content into a format and obtain a graph database insert statement.

[0099] In some implementations, the execution of a dynamic processing script is shown in Figure 4. The dynamic script is a Groovy function that takes as input the data type (vertex or edge), the data label (vertex type or edge type), and the data content. The function returns an insert statement for the graph database. The Groovy function internally converts the data content to the appropriate format for each data type and label, and then concatenates the data into the graph database insert statement.

[0100] Step 207: The server reads the graph database configuration from the configuration server, executes the insert statement according to the configuration, and writes the data into the graph database.

[0101] In some implementation cases, due to the speed limit of the graph database write interface, a flow control mechanism is added when writing to the database. For requests that exceed the write speed limit, failure is returned and feedback is given to the message system. The message system will subsequently redistribute the import task to the import cluster.

[0102] As shown in FIG5 , step S3 specifically includes the following sub-steps:

[0103] Step S301: Collecting import information: After the graph data import cluster completes a batch of data import tasks, it sends the import execution status information to the import task monitoring module.

[0104] Step S302: Summarize the imported data in real time.

[0105] Step S303: Generate an import task execution status report.

[0106] In some implementations, as shown in Figure 6, a distributed cache (such as Redis) can be used to count successful and failed imports, enabling progress summary with minimal performance overhead. The cache key can be designed as follows: success_a represents the number of successful imports for task a, and fail_a represents the number of failed imports for task a. The cache value is the number of successful or failed imports. Each time a record is successfully or unsuccessfully imported, the corresponding cache value is incremented by one.

[0107] In some implementation cases, as shown in Figure 7, users not only want to obtain the number of successful or failed task imports, but also want to obtain detailed information about the import of each data, such as import time, data content, failure reason, etc. The big data processing platform can be used to collect and process the local logs generated by the data import cluster server, and import them into the log search engine for users to search and query the import details according to the data content.

[0108] As shown in Figure 8, an embodiment of the present application provides an electronic device, which includes a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, any method of the first aspect described above is implemented.

[0109] The system also includes a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0110] Among them, the memory 101 can be, but is not limited to, a random access memory 101 (Random Access Memory, RAM), a read-only memory 101 (Read Only Memory, ROM), a programmable read-only memory 101 (Programmable Read-Only Memory, PROM), an erasable programmable read-only memory 101 (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable read-only memory 101 (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0111] The processor 102 may be an integrated circuit chip having signal processing capabilities. The processor 102 may be a general-purpose processor 102, including a central processing unit 102 (CPU), a network processor 102 (NP), etc.; it may also be a digital signal processing processor 102 (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0112] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can also be implemented in other ways. The method and system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the methods and systems, methods, and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a portion of code, and the module, program segment, or a portion of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0113] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0114] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the method of any one of the first aspects described above when executed by the processor 102. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory 101 (ROM, Read-Only Memory), a random access memory 101 (RAM, Random Access Memory), a magnetic disk or an optical disk.

[0115] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.

[0116] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A graph database data import system, characterized in that: include: The import task initiation module is used to configure the graph database information, configure the graph data format conversion script, read the data file line by line, and call the API to submit the import task; The import task execution module is used to receive the import task through the API and distribute the task to the data import cluster through the message server; The graph data import cluster includes multiple servers, which are used to parse the distributed task data using the data processing dynamic script according to the graph data format conversion script to obtain the graph database insert statement, and then execute the insert statement according to the graph database information to write the data into the graph database; The import task monitoring module is used to collect import execution data, summarize the import data in real time, and generate import task execution reports.

2. A method for importing data into a graph database, characterized in that: The method comprises: Step S1, through the import task initiation module, configure the graph database information, configure the graph data format conversion script, read the data file line by line, and call the API to submit the import task; Step S2: The import task execution module receives the import task through the API, and distributes the task to the graph data import cluster through the message server. The servers in the graph data import cluster respectively use the data processing dynamic script to parse the distributed task data according to the graph data format conversion script to obtain the graph database insert statement, and then execute the insert statement according to the graph database information to write the data into the graph database. Step S3, the import task monitoring module collects import execution status data, summarizes the import status data in real time, and generates an import task execution status report.

3. The method for importing graph database data according to claim 2, characterized in that: The step S1 specifically includes the following sub-steps: Step S101: The import task initiation module configures the graph database information, including the import task ID, graph database IP address, user name, graph space name, and saves it to the cloud configuration server; Step S102: configure the graph data format conversion script, including the import task ID and the data format conversion script content, and save it to the configuration server in the cloud; Step S103: Read the data file line by line; Step S104: Call the API to submit the import task. The API input parameters are the import task ID, data type, data label, and data content.

4. The method for importing graph database data according to claim 3, characterized in that: In step S101 and step S102, the configuration server sends the graph database information and the graph data format conversion script to the memory of the graph data import cluster server.

5. The method for importing graph database data according to claim 3, characterized in that: In step S104, the API is called in an asynchronous manner through a message queue.

6. The method for importing graph database data according to claim 2, characterized in that: The step S2 specifically includes the following sub-steps: Step S201: receiving the import task transmitted by the import task initiation module through the data import API; Step S202: Send the import task to the message server; Step S203: the message server distributes the import task to the servers in the data import cluster for processing; Step 204: The server receiving the task parses the import task message through the data import program to obtain the import task ID, data type, data tag and data content; Step 205: The server reads the data format conversion script according to the import task ID; Step 206: The server uses the data dynamic processing script according to the data format conversion script to convert the data content format and obtain a graph database insert statement; Step 207: The server reads the configured graph database information and executes an insert statement based on the graph database information to write the data into the graph database.

7. The method for importing graph database data according to claim 6, characterized in that: The message distribution mode of the message server is a mode that is dynamically adjusted according to the server pressure.

8. The method for importing graph database data according to claim 2, characterized in that: The step S3 specifically includes the following sub-steps: Step S301: after completing a batch of data import tasks, the graph data import cluster sends the import execution status information to the import task monitoring module; Step S302: the import task monitoring module summarizes the import status data in real time; Step S303: The import task monitoring module generates an import task execution status report.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the graph database data import method described in any one of claims 2-8 above.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for importing graph database data as described in any one of claims 2 to 8 is implemented.

Images