A method, apparatus and system for importing data

Abstract

The application discloses a data import method, device and system, and relates to the technical field of big data data access. A specific embodiment of the method comprises the following steps: a corresponding import task can be constructed for a plurality of to-be-imported files; a plurality of preset service processes are utilized to concurrently acquire and execute the import task, so that to-be-imported data in the to-be-imported files is imported into a target graph database in operation; and the detected abnormal data in the import is re-imported into the target graph database. Through the technical means of the target graph database in operation, concurrent execution of the import task, abnormal data monitoring and re-importing, the flexibility and reliability of the imported data are improved, the efficiency of the imported data is improved, and the complexity of the imported data is reduced.

Description

Technical Field

[0001] This invention relates to the field of big data access technology, and in particular to a method, apparatus and system for importing data. Background Technology

[0002] As internet applications become increasingly complex, the demands on data processing also increase. For example, it is necessary to import data from multiple data sources into a designated database (such as a graph database) in order to provide corresponding services based on the data in the designated database.

[0003] Existing methods for importing data include: importing data in a cold loading mode when the target graph database (such as NEO4J graph database) stops providing services, or importing data line by line for each data item that needs to be imported. As can be seen, existing methods for importing data have problems such as poor flexibility, low efficiency, and high complexity. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, and system for importing data, capable of constructing corresponding import tasks for multiple files to be imported; concurrently acquiring and executing import tasks using multiple preset service processes to import the data to be imported from the files to be imported into a running target graph database; and re-importing detected abnormal import data into the target graph database. By employing techniques such as concurrent execution of import tasks, abnormal data monitoring, and re-importing on a running target graph database, the flexibility and reliability of data import are improved, the efficiency of data import is increased, and the complexity of data import is reduced.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for importing data is provided, characterized by comprising: determining a plurality of files to be imported corresponding to the data to be imported; wherein the files to be imported are node files associated with a target graph database, or relationship files between nodes; constructing a corresponding import task for each file to be imported; concurrently acquiring and executing the import tasks using a plurality of preset service processes to import the data to be imported from the files to be imported corresponding to the import tasks into the running target graph database; and detecting abnormal import data in the executed import tasks, and re-importing the detected abnormal import data into the target graph database.

[0006] Optionally, determining the multiple files to be imported corresponding to the data to be imported includes: obtaining one or more data files containing the data to be imported from the data source; the data file is any one of node file or relationship file between nodes; for each data file, performing the following: determining whether the data file meets the preset splitting conditions; if so, splitting the data file into multiple files to be imported of a set size; otherwise, directly using the data file as the file to be imported.

[0007] Optionally, splitting the data file into multiple files of a set size to be imported includes: reading the data file splitting strategy from a preset configuration file; and splitting the data file into multiple files to be imported based on the splitting strategy.

[0008] Optionally, the step of concurrently acquiring import tasks using multiple preset service processes includes: multiple idle service processes searching for unexecuted import tasks, and if a task is found, the idle service process pulling the file to be imported corresponding to the unexecuted import task.

[0009] Optionally, after the idle service process retrieves the file to be imported corresponding to the unexecuted import task, the method further includes: obtaining one or more attribute fields of the file to be imported from the preset configuration file; the execution of the import task includes: writing one or more attribute fields and the file identifier of the file to be imported into a preset import module containing import instructions, so that the import module imports the attribute fields and the attribute values ​​corresponding to the attribute fields in the file to be imported into the target graph database based on one or more attribute fields and the file identifier of the file to be imported.

[0010] Optionally, the data import method further includes: obtaining import cycle configuration information corresponding to the execution of the unexecuted import task from the preset configuration file; and the import module performing the step of importing the attribute field and the attribute values ​​corresponding to the attribute field in the file to be imported into the target graph database based on the import cycle configuration information.

[0011] Optionally, after detecting abnormal import data in the executed import task, the method further includes: if the number of abnormal import data exceeds a set fault tolerance threshold, performing the step of re-importing the detected abnormal import data into the target graph database; if the number of abnormal import data does not exceed the set fault tolerance threshold, the service process sends a message indicating that the import has been completed.

[0012] Optionally, when the imported abnormal data is detected, each piece of imported abnormal data is recorded; the step of re-importing the detected imported abnormal data into the target graph database includes: for each piece of imported abnormal data, executing: writing one or more attribute fields corresponding to the imported abnormal data into a preset splicing module including splicing instructions, so that the splicing module re-imports the attribute fields and the attribute values ​​corresponding to the attribute fields in the imported abnormal data into the target graph database based on one or more attribute fields.

[0013] Optionally, when the number of imported abnormal data is detected to exceed a set fault tolerance threshold, the method further includes: sending a warning message corresponding to the abnormal data so that the data processing party can process the abnormal data.

[0014] Optionally, importing the data to be imported from the file to be imported corresponding to the import task into the running target graph database includes: importing one or more node files and the relationship files between multiple nodes into the running target graph database, so that the target graph database constructs the graph relationship between the attribute values ​​in the file to be imported based on one or more node files and the relationship files between multiple nodes.

[0015] To achieve the above objectives, according to a second aspect of the present invention, an apparatus for importing data is provided, characterized in that it includes: a file acquisition module, a service module, and an import module; wherein,

[0016] The file acquisition module is used to determine multiple files to be imported corresponding to the data to be imported; wherein, the files to be imported are node files associated with the target graph database, or relationship files between nodes;

[0017] The service module is used to build a corresponding import task for each of the files to be imported;

[0018] The import module includes multiple preset service processes. These multiple service processes concurrently acquire and execute import tasks to import the data to be imported from the files corresponding to the import tasks into the running target graph database. The module also detects abnormal import data in the executed import tasks and re-imports the detected abnormal import data into the target graph database.

[0019] Optionally, the data import device is used to determine multiple files to be imported corresponding to the data to be imported, including: obtaining one or more data files containing the data to be imported from a data source; the data file is any one of node files or relationship files between nodes; for each data file, performing the following: determining whether the data file meets a preset splitting condition; if so, splitting the data file into multiple files to be imported of a set size; otherwise, directly using the data file as the file to be imported.

[0020] Optionally, the data import device is used to split the data file into multiple files of a predetermined size to be imported, including: reading the data file splitting strategy from a preset configuration file; and splitting the data file into multiple files to be imported based on the splitting strategy.

[0021] Optionally, the data import device is used to concurrently acquire import tasks using multiple preset service processes, including: multiple idle service processes searching for unexecuted import tasks, and if a task is found, the idle service process pulls the file to be imported corresponding to the unexecuted import task.

[0022] Optionally, the data import device, after the idle service process retrieves the file to be imported corresponding to the unexecuted import task, further includes: obtaining one or more attribute fields of the file to be imported from the preset configuration file; the execution of the import task includes: writing one or more attribute fields and the file identifier of the file to be imported into a preset import module including import instructions, so that the import module imports the attribute fields and the attribute values ​​corresponding to the attribute fields in the file to be imported into the target graph database based on one or more attribute fields and the file identifier of the file to be imported.

[0023] Optionally, the data import device is further configured to obtain import cycle configuration information corresponding to the execution of the unexecuted import task from the preset configuration file; the import module performs the step of importing the attribute field and the attribute value corresponding to the attribute field in the file to be imported into the target graph database based on the import cycle configuration information.

[0024] Optionally, the data import device, after detecting abnormal import data in the executed import task, further includes: if the number of abnormal import data detected exceeds a set fault tolerance threshold, performing the step of re-importing the detected abnormal import data into the target graph database; if the number of abnormal import data detected does not exceed the set fault tolerance threshold, the service process sends a message indicating that the import has been completed.

[0025] Optionally, the data import device is configured to record each piece of import error data when an import error is detected; the step of re-importing the detected import error data into the target graph database includes: for each piece of import error data, executing: writing one or more attribute fields corresponding to the import error data into a preset splicing module including splicing instructions, so that the splicing module re-imports the attribute fields and the attribute values ​​corresponding to the attribute fields in the import error data into the target graph database based on one or more attribute fields.

[0026] Optionally, the data importing device is further configured to, when detecting that the number of imported abnormal data exceeds a set fault tolerance threshold, send a warning message corresponding to the abnormal data so that the data processing party processes the abnormal data.

[0027] Optionally, the data import device is used to import the data to be imported from the file to be imported corresponding to the import task into the running target graph database, including: importing one or more node files and the relationship files between multiple nodes into the running target graph database, so that the target graph database constructs the graph relationship between the attribute values ​​in the file to be imported based on one or more node files and the relationship files between multiple nodes.

[0028] To achieve the above objectives, according to a third aspect of the present invention, a system for importing data is provided, characterized in that it includes: an apparatus for importing data according to the second aspect, and one or more target graph databases; wherein the apparatus for importing data imports data to be imported from the file to be imported into the running target graph database.

[0029] To achieve the above objectives, according to a fourth aspect of the present invention, an electronic device for importing data is provided, characterized in that it includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described above for importing data.

[0030] To achieve the above objectives, according to a fifth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements any of the methods described above for importing data.

[0031] To achieve the above objectives, according to a sixth aspect of the present invention, the present invention also provides a computer program product, including a computer program, characterized in that, when the computer program is executed by a processor, it implements any of the methods described above for importing data.

[0032] One embodiment of the above invention has the following advantages or beneficial effects: it can construct corresponding import tasks for multiple files to be imported; it utilizes multiple preset service processes to concurrently acquire and execute import tasks, importing the data to be imported from the files to be imported into the running target graph database; and it re-imports the detected abnormal import data into the target graph database. By employing techniques such as concurrently executing import tasks, monitoring abnormal data, and re-importing from the running target graph database, the flexibility and reliability of imported data are improved, the efficiency of imported data is increased, and the complexity of imported data is reduced.

[0033] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0034] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0035] Figure 1 This is a flowchart illustrating a method for importing data according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of a data import process provided in another embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of a data import device provided in one embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the structure of a data import system provided in one embodiment of the present invention;

[0039] Figure 5 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0040] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0041] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description; the acquisition, storage, use, and processing of data in the technical solutions of this application all comply with relevant national laws and regulations.

[0042] like Figure 1 As shown, this embodiment of the invention provides a method for importing data, which may include the following steps:

[0043] Step S101: Determine multiple files to be imported corresponding to the data to be imported; the files to be imported are node files associated with the target graph database, or relationship files between nodes.

[0044] Specifically, different types of databases contain corresponding methods for importing data. Each method processes the data to be imported, which exists in multiple files, such as file1.csv, file2.dat, ..., fileN.xx. This invention does not limit the file format or file identifier of the files to be imported. When importing data into a target graph database (e.g., NEO4J graph database), the files to be imported may include node files associated with the target graph data and relationship files between nodes. It is understood that the node files and relationship files correspond to multiple data values ​​and relationships between multiple data values ​​in the graph database. The data contained in the node files and the data contained in the relationship files need to be imported into the target graph database. It is understood that the import order of the node files or relationship files is arbitrary, thereby utilizing the graph database to generate corresponding graph data based on all imported data and the relationships between the data.

[0045] The file to be imported is generated based on the data contained in the data source. For example, the file to be imported corresponding to data source 1 is file1.csv (e.g., a node file). That is, determining multiple files to be imported corresponding to the data to be imported includes: obtaining one or more data files containing the data to be imported from the data source; the data file is any one of a node file or a file relating nodes.

[0046] Methods to determine if a data file is the file to be imported include:

[0047] The first method: directly use one or more data files (such as node files or relationship files between nodes) as the files to be imported.

[0048] Preferably, the second method is as follows: for each data file (e.g., a node file or a file relating nodes), it is determined whether the data file meets the preset splitting conditions. If so, the data file is split into multiple files of a set size to be imported; otherwise, the data file is directly used as the file to be imported.

[0049] Specifically, when the amount of data to be imported is large, in order to improve the efficiency of data import, the import file containing the data to be imported is split into multiple import files of a set size. Whether a data file meets the preset splitting condition can be determined by the number of data entries in the data file exceeding a set data threshold, for example, a threshold of 200,000; file1.csv contains 20 million data entries, thus meeting the preset splitting condition; for example, file1.csv contains 20 million data entries, and according to the splitting strategy, file1.csv is split into 100 files, each containing 200,000 data entries; that is, splitting the data file into multiple import files of a set size includes: reading the data file splitting strategy from a preset configuration file; and splitting the data file into multiple import files based on the splitting strategy. The configuration information for the splitting strategy can be configured in a preset configuration file, such as setting.txt. For example, the configuration information for the splitting strategy could be: split_rows_size = 200000; meaning that the current file1.csv will be split into multiple files, with each file containing 200,000 rows. It's understandable that the preset configuration file can also configure the directory where the files to be imported are located, for example:

[0050] csv_save_dir='home / ap / nas / abc / import'

[0051] This directory allows for the automatic retrieval of files to be imported, and the import process can be tailored to the data volume required for the application scenario. The processing cycle for these files can be configured in the settings file, such as importing files generated each month or every 10 days. By using a splitting strategy to segment the files, the split files can be imported concurrently through multiple processes, further improving data import efficiency.

[0052] Step S102: Construct a corresponding import task for each of the files to be imported.

[0053] Specifically, an import task is constructed for each file to be imported. The file to be imported can be a directly obtained data file or a data file that has been split and processed. The splitting of the file to be imported and the construction of the corresponding import task can be executed by the Master node. For example, the Master node generates import tasks A, B, C, etc., and waits for multiple Slave nodes (such as service processes) to process them.

[0054] Step S103: Utilize multiple preset service processes to concurrently acquire and execute import tasks, so as to import the data to be imported from the files to be imported corresponding to the import tasks into the running target graph database; and detect the import abnormal data of the executed import tasks, and re-import the detected import abnormal data into the target graph database.

[0055] Specifically, concurrent execution of import tasks involves using multiple pre-defined service processes to execute multiple import tasks concurrently. Before concurrently acquiring and executing import tasks using these pre-defined service processes, the service processes can actively search for and retrieve import tasks. That is, concurrently acquiring import tasks using multiple pre-defined service processes includes: multiple idle service processes searching for unexecuted import tasks; and, if found, the idle service process retrieving the file to be imported corresponding to the unexecuted import task. This step enables resource allocation among multiple concurrent service processes, improving the efficiency of data import and optimizing resource allocation for imported data.

[0056] Furthermore, after the idle service process retrieves the files to be imported corresponding to the unexecuted import tasks, the import operation is performed.

[0057] In one embodiment of the present invention, the method for performing an import task is as follows: after the idle service process retrieves the file to be imported corresponding to the unexecuted import task, the method further includes: obtaining one or more attribute fields of the file to be imported from the preset configuration file; the method for performing the import task includes: writing one or more attribute fields and the file identifier of the file to be imported into a preset import module including import instructions, so that the import module imports the attribute fields and the attribute values ​​corresponding to the attribute fields in the file to be imported into the target graph database based on one or more attribute fields and the file identifier of the file to be imported. The following example uses the NEO4J graph database as the target graph database to illustrate the import module, which includes import commands such as Load csv and Create. The data associated with these commands includes one or more attribute fields, their corresponding attribute values, and the file identifier of the file to be imported (e.g., file1.csv). The attribute fields can be automatically obtained from a preset configuration file. For example, the configuration file might be configured as follows: table_columns = {'abcd': ['a', 'b', 'c'...]}; where 'abcd' represents the table identifier to which the attribute field in file1.csv belongs, and 'a', 'b', 'c', etc., represent multiple attribute fields contained in the table. The import module, based on one or more attribute fields and the file identifier of the file to be imported, imports the attribute fields and their corresponding attribute values ​​from the file to be imported into the target graph database. It is understood that a file to be imported can contain data from multiple tables, which can be achieved by configuring attribute fields for multiple tables in a preset configuration file and adding corresponding code to the import module to process these multiple tables.

[0058] Further, preferably, the import cycle configuration information corresponding to the unexecuted import task is obtained from the preset configuration file; the import module, based on the import cycle configuration information, performs the step of importing the attribute fields and the attribute values ​​corresponding to the attribute fields in the file to be imported into the target graph database. For example, the import cycle configuration information is "Using periodic commit 1000", which means that when the import module is used to perform the import, it commits every 1000 rows of data.

[0059] It's understandable that for a data table to be imported, there might be cases where only some attribute fields' values ​​change within a set time range, such as numeric attribute fields. Therefore, in addition to configuring all attribute fields of a data table in the preset configuration file, you can also configure information about attribute fields that change over time. When importing data, you can choose to import only the field values ​​of the changed attribute fields into the target graph database, thereby updating the data in the target graph database. This further improves the efficiency of data import by reducing the amount of data imported and saving resource consumption.

[0060] Therefore, the code generated by the preset configuration file and the import module can perform import operations on files corresponding to any one or more data sources. Only the configuration information needs to be modified, not the code, improving the versatility and flexibility of imported data processing, and increasing import efficiency. Furthermore, the import module allows data to be imported into a running target graph database (without stopping database operations), overcoming the technical limitation of requiring database operations to be stopped before data import, further improving the flexibility of data import and enhancing the reliability and efficiency of the target graph database in providing data services.

[0061] Furthermore, to ensure the integrity and stability of imported data, embodiments of the present invention provide a fault-tolerance method for import anomalies. Import anomalies can be caused by factors such as server performance fluctuations or unstable network connections leading to import task timeouts. In such cases, abnormal import data is identified. The criteria for determining anomalies can be set through a configuration file; for example, `tt_load_csv_secs=600` represents a timeout threshold of 600 seconds. If the import operation exceeds this timeout threshold and fails to complete, an import anomaly is determined. Abnormal import data refers to data that should have been imported when the import task encountered an anomaly, but failed to be imported successfully due to the anomaly.

[0062] After detecting abnormal import data in the executed import task, the method further includes: if the number of abnormal import data exceeds a set fault tolerance threshold, performing the step of re-importing the detected abnormal import data into the target graph database; if the number of abnormal import data does not exceed the set fault tolerance threshold, the service process sends a message indicating that the import is complete. Specifically, if an anomaly is determined, the number of import data corresponding to the import anomaly is obtained; if the number of abnormal import data exceeds the set fault tolerance threshold, the re-import step is performed; otherwise, the import is considered complete, and the service process sends a message indicating that the import is complete. The set fault tolerance threshold can be configured in a preset configuration file; for example, setting tt_error_rows = 100 represents setting the fault tolerance threshold to 100 rows; if the number of abnormal import data rows exceeds 100, the step of re-importing the detected abnormal import data into the target graph database is performed. This invention does not limit the value or form of the set fault tolerance threshold.

[0063] Furthermore, in order to perform the re-import step, when the import abnormal data is detected, each piece of import abnormal data is recorded; the step of re-importing the detected import abnormal data into the target graph database includes: for each piece of import abnormal data, performing the following: writing one or more attribute fields corresponding to the import abnormal data into a preset splicing module including splicing instructions, so that the splicing module re-imports the attribute fields and the attribute values ​​corresponding to the attribute fields in the import abnormal data into the target graph database based on one or more attribute fields.

[0064] Specifically, each import exception record is documented. This includes the content (attribute values ​​for attribute fields, etc.), quantity, time of occurrence, and the data table to which the exception belongs, all saved in a file or database. This allows subsequent error-handling operations to be performed based on this record. Preferably, if an import file times out, the task (or the process running the task) can be automatically terminated, preventing task blocking, recording information associated with the import file, and skipping that file to continue executing other import files.

[0065] The following example uses the NEO4J graph database as the target graph database to illustrate the preset splicing module, which includes splicing instructions. The splicing instructions can include Merge, On Create, On Match, Return, etc. The data associated with the splicing instructions includes attribute fields and the attribute values ​​corresponding to the attribute fields. The attribute fields (e.g., Property1, Property2, etc.) are obtained from the preset configuration file.

[0066] Further, preferably, the step of detecting that the number of imported abnormal data exceeds a set fault tolerance threshold further includes: sending an early warning message corresponding to the abnormal data, so that the data processing party can process the abnormal data. Specifically, the master node can be used to provide early warning information related to the abnormal data, for example: sending early warning information via SMS, providing early warning information via a webpage, etc.; so that the data processing party can process the abnormal data, for example: analyzing the abnormal data to determine whether the operation of re-importing the abnormal data has been successful, or determining the cause of the abnormality, etc.

[0067] In one embodiment of the present invention, the target graph database is taken as NEO4J graph database. In the prior art, the methods for importing data into the graph database NEO4J include: using tools such as Batch_Inserter, Batch_Import, and Neo4j_import. These methods all require the database to be stopped during the data import process, that is, unable to provide data services. Alternatively, data can be imported one by one into the running graph database using the Create statement. This method has the problem of low efficiency when the data volume is large. In view of this, the embodiment of the present invention can determine multiple files to be imported corresponding to the data to be imported, wherein the files to be imported are node files or relationship files between nodes; construct a corresponding import task for each file to be imported; use multiple preset service processes to concurrently obtain and execute the import tasks to import the data to be imported from the files to be imported corresponding to the import tasks into the running target graph database; and detect abnormal import data in the executed import tasks and re-import the detected abnormal import data into the target graph database.

[0068] That is, importing the data to be imported from the file to be imported corresponding to the import task into the running target graph database includes: importing one or more node files and the relationship files between multiple nodes into the graph database, so that the graph database can construct the graph relationship between the attribute values ​​in the file to be imported based on one or more node files and the relationship files between multiple nodes. The attribute fields of nodes and the relationships between nodes can be configured through preset configuration files. For example, `nodedict = {'node123': {'table': 'NEO4J table name', 'primarykeys': [primary key list],}` configures the data table corresponding to node `node123` and the multiple primary key information contained in the attribute fields of the data table. `relationdict = {relation name: {'reldict': {'table': 'table1', 'relname': 'cooperation',}, 'node1dict': {'nodename': node1', 'pkeys': [primary key list],}, 'node2dict': {'nodename': 'node2', 'pkeys': [primary key list],}` represents the configuration information for setting the relationship between two nodes (e.g., configuring it as cooperation). Furthermore, based on the configuration information in the configuration file, the attribute values ​​contained in one or more node files, as well as the relational attribute values ​​contained in the relationship files between multiple nodes, are imported into a graph database (e.g., NEO4J graph database), so that the graph database can construct the graph relationships between the attribute values ​​in the files to be imported based on one or more node files and the relationship files between multiple nodes.

[0069] As can be seen from steps S101-S103, the embodiments of the present invention improve the efficiency and stability of data import by concurrent task processing, file splitting, and a combination of multiple import commands (such as load_csv, create, etc.). This largely overcomes the problem of frequent timeouts caused by single-task importing large amounts of data. Furthermore, the embodiments of the present invention further perform fault-tolerant operations to improve the integrity and high availability of imported data when import anomalies are detected. In addition, by combining configuration information with the import module and the splicing module, data providers unfamiliar with the execution commands of the target graph database can also use the methods of the embodiments of the present invention to import data, improving the efficiency of data import, reducing the complexity of data import, and reducing the labor and time costs associated with data import.

[0070] like Figure 2 As shown, this embodiment of the invention provides a method for importing data, which may include the following steps:

[0071] Step S201: Determine multiple files to be imported corresponding to the data to be imported, including: obtaining one or more data files containing the data to be imported from the data source, wherein the data file is any one of node file or relationship file between nodes.

[0072] Specifically, the description of obtaining one or more data files containing the data to be imported from the data source is consistent with the description of step S101, and will not be repeated here.

[0073] Step S202: Using the Master node: determine whether the data file meets the preset splitting conditions. If so, split the data file into multiple files of a set size to be imported; otherwise, directly use the data file as the file to be imported.

[0074] Specifically, the names of the Master node and Slave node in this embodiment of the invention are merely examples used to distinguish the services being executed. The Master node and Slave node can run on the same physical machine or on multiple physical devices respectively.

[0075] Step S203: Utilize the service processes in multiple Slave nodes to locate and retrieve the files to be imported from the Master node for any unexecuted import tasks, and then execute the import tasks. Determine if any import errors exist; if so, proceed with the steps to re-import the abnormal data into the target graph database.

[0076] Specifically, when any service process detects the completion of an import task, it sends the result back to the Master node. This causes the Master node to update the task status, marking any unexecuted import tasks, thus enabling multiple Slave nodes (e.g., ...) to... Figure 2 The service processes in Slave nodes (Slave node 1, Slave node 2, ..., Slave node N, etc.) retrieve unexecuted import tasks from the Master node using a preemptive method. Each Slave node consists of multiple idle service processes. These idle service processes search for unexecuted import tasks, and when a task is found, they retrieve the corresponding file to be imported.

[0077] As can be seen, by automatically determining the import task through the Master node and dynamically pulling and executing unfinished tasks through multiple Slave nodes, the efficiency and automation of data import are greatly improved.

[0078] Furthermore, when each Slave node performs the import task, it checks whether there is an error in the import. If so, it performs the step of re-importing the abnormal data into the target graph database. The steps of re-importing the abnormal data into the target graph database are the same as those in step S103, and will not be repeated here.

[0079] Step S204: The data to be imported is imported into the running target graph database, so that the target graph database constructs the graph relationship between the attribute values ​​in the file to be imported based on one or more of the node files and the relationship files between multiple nodes.

[0080] Specifically, the service processes in multiple Slave nodes concurrently execute the import task, thereby importing the data to be imported into the running target graph database, so that the target graph database constructs the graph relationship between the attribute values ​​in the file to be imported based on one or more node files and the relationship files between multiple nodes.

[0081] like Figure 3 As shown, this embodiment of the invention provides a data import device 300, including: a file acquisition module 301, a service module 302, and an import module 303; wherein,

[0082] The file acquisition module 301 is used to determine multiple files to be imported corresponding to the data to be imported; wherein, the files to be imported are node files or relationship files between nodes;

[0083] The service module 302 is used to construct a corresponding import task for each of the files to be imported.

[0084] The import module 303 includes multiple preset service processes. The multiple service processes concurrently acquire and execute import tasks to import the data to be imported from the files to be imported corresponding to the import tasks into the running target graph database. The module also detects abnormal import data in the executed import tasks and re-imports the detected abnormal import data into the target graph database.

[0085] like Figure 4 As shown, an embodiment of the present invention provides a data import system 400, including: a data import device 300, and one or more target graph databases; wherein, the data to be imported from the file to be imported is imported into the running target graph database using the data import device 300.

[0086] This invention also provides an electronic device for importing data, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in any of the above embodiments.

[0087] This invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in any of the above embodiments.

[0088] This invention also provides a computer program product, including a computer program, characterized in that the program, when executed by a processor, implements any of the methods described above.

[0089] Figure 5 An exemplary system architecture 500 is shown, in which a method or apparatus for importing data can be applied according to embodiments of the present invention.

[0090] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0091] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various client applications can be installed on terminal devices 501, 502, and 503, such as e-commerce client applications, web browser applications, search applications, instant messaging tools, and email clients.

[0092] Terminal devices 501, 502, and 503 can be various electronic devices with displays and supporting various client applications, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0093] Server 505 can be a server that provides various services, such as a backend management server that supports client applications used by users through terminal devices 501, 502, and 503. The backend management server can process received requests to import data and return the results of the data import to the terminal devices.

[0094] It should be noted that the data import method provided in this embodiment of the invention is generally executed by server 505; the data import device is generally located in server 505.

[0095] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0096] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing a terminal device of the present invention. Figure 6 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0097] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0098] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0099] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0100] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The modules and / or units described in the embodiments of the present invention can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, a processor can be described as including a file acquisition module, a service module, and an import module. The names of these modules do not necessarily limit the module itself; for example, the file acquisition module can also be described as "a module for determining multiple files corresponding to the data to be imported."

[0103] In another aspect, the present invention also provides a computer program product that implements a method for importing data according to embodiments of the present invention when the program is executed by a processor.

[0104] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: determining a plurality of files to be imported corresponding to data to be imported; wherein the files to be imported are node files associated with a target graph database, or relationship files between nodes; constructing a corresponding import task for each file to be imported; concurrently acquiring and executing the import tasks using a plurality of preset service processes to import the data to be imported from the files corresponding to the import tasks into the running target graph database; and detecting abnormal import data in the executed import tasks, and re-importing the detected abnormal import data into the target graph database.

[0105] The embodiments of the present invention can construct corresponding import tasks for multiple files to be imported; utilize multiple preset service processes to concurrently acquire and execute import tasks, importing the data to be imported from the files into a running target graph database; and re-import detected abnormal import data into the target graph database. By employing techniques such as concurrent execution of import tasks, abnormal data monitoring, and re-import on a running target graph database, the flexibility and reliability of imported data are improved, the efficiency of imported data is increased, and the complexity of imported data is reduced.

[0106] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of importing data, characterized by, The method comprises the following steps: determining a plurality of to-be-imported files corresponding to to-be-imported data; wherein the to-be-imported files are node files associated with a target graph database or relationship files between nodes; the to-be-imported files are generated according to data contained in a data source; constructing a corresponding import task for each of the to-be-imported files; concurrently acquiring and executing the import tasks by using a plurality of preset service processes, so as to import to-be-imported data in the to-be-imported files corresponding to the import tasks into the running target graph database; and detecting import abnormal data of the executed import tasks, and reimporting the detected import abnormal data into the target graph database; the step of concurrently acquiring the import tasks by using the plurality of preset service processes comprises the following steps: a plurality of idle service processes respectively search for unexecuted import tasks, and in the case of finding, the idle service processes pull to-be-imported files corresponding to the unexecuted import tasks, and acquire one or more attribute fields of the to-be-imported files from a preset configuration file; the step of executing the import tasks comprises the following steps: writing the one or more attribute fields and a file identifier of the to-be-imported file into a preset import module including an import instruction, so that the import module imports the attribute fields and attribute values corresponding to the attribute fields in the to-be-imported file into the target graph database based on the one or more attribute fields and the file identifier of the to-be-imported file.

2. The method of claim 1, wherein: the step of determining a plurality of to-be-imported files corresponding to to-be-imported data comprises the following steps: acquiring one or more data files containing the to-be-imported data from a data source, wherein the data files are any one of node files or relationship files between nodes; for each of the data files, performing the following steps: determining whether the data file meets a preset splitting condition, and if so, splitting the data file into a plurality of to-be-imported files of a set size; otherwise, directly taking the data file as a to-be-imported file.

3. The method of claim 2, wherein: the step of splitting the data file into a plurality of to-be-imported files of a set size comprises the following steps: reading a splitting strategy of the data file from a preset configuration file; based on the splitting strategy, splitting the data file into a plurality of to-be-imported files.

4. The method of claim 1, wherein, Further comprising the following steps: acquiring import period configuration information corresponding to the execution of the unexecuted import task from the preset configuration file; the step of importing the attribute fields and the attribute values corresponding to the attribute fields in the to-be-imported file into the target graph database based on the import period configuration information.

5. The method of claim 1, wherein: after the step of detecting import abnormal data of the executed import tasks, further comprising the following steps: in the case that the number of the detected import abnormal data exceeds a set fault tolerance threshold, performing the step of reimporting the detected import abnormal data into the target graph database. In a case where it is detected that the number of the import abnormal data does not exceed the set fault tolerance threshold, the service process feeds back a message indicating that the import is completed.

6. The method of claim 5, wherein, Further comprising: In a case where the import abnormal data is detected, record each piece of import abnormal data where the import abnormality occurs; The re-importing of the detected import abnormal data into the target graph database comprises: For each piece of the import abnormal data, performing: Write one or more attribute fields corresponding to the import abnormal data into a preset splicing module including splicing instructions, so that the splicing module re-imports the attribute fields and attribute values corresponding to the attribute fields in the import abnormal data into the target graph database based on the one or more attribute fields.

7. The method of claim 5, wherein, In a case where it is detected that the number of the import abnormal data exceeds the set fault tolerance threshold, further comprising: Sending early warning information corresponding to the abnormal data to enable a data processing party to handle the abnormal data.

8. The method of claim 1, wherein, The importing of the to-be-imported data in the to-be-imported file corresponding to the import task into the running target graph database comprises: Importing one or more node files and relationship files between multiple nodes into the running target graph database, so that the target graph database constructs a graph relationship between attribute values in the to-be-imported file based on the one or more node files and the relationship files between multiple nodes.

9. An apparatus for importing data, characterized by Comprising: An acquisition file module, a service module, and an import module; wherein, The acquisition file module is configured to determine multiple to-be-imported files corresponding to to-be-imported data; wherein, the to-be-imported file is a node file or a relationship file between nodes associated with a target graph database; the to-be-imported file is generated according to data contained in a data source; The service module is configured to construct a corresponding import task for each to-be-imported file; The import module comprises a plurality of preset service processes, and uses the plurality of service processes to concurrently acquire and execute import tasks to import to-be-imported data in to-be-imported files corresponding to the import tasks into a running target graph database, and detect import abnormal data of the executed import tasks, and re-import the detected import abnormal data into the target graph database; The import module uses a plurality of preset service processes to concurrently acquire import tasks, comprising: Multiple idle service processes respectively search for unexecuted import tasks, and in a case where an unexecuted import task is found, the idle service process pulls a to-be-imported file corresponding to the unexecuted import task; The import module, after the idle service process pulls the to-be-imported file corresponding to the unexecuted import task, further comprises: Acquire one or more attribute fields of the to-be-imported file from the preset configuration file; The execution of the import task comprises: write one or more of the attribute fields and the file identifier of the file to be imported into a preset import module including an import instruction, so that the import module imports the attribute fields and attribute values corresponding to the attribute fields in the file to be imported into the target graph database based on one or more of the attribute fields and the file identifier of the file to be imported.

10. The apparatus of claim 9, wherein, the obtaining file module determines a plurality of files to be imported corresponding to the data to be imported, comprising: obtaining one or more data files containing the data to be imported from a data source, the data file being any one of a node file or a relationship file between nodes; for each of the data files, performing: determining whether the data file meets a preset splitting condition, if yes, splitting the data file into a plurality of files to be imported with a set size; otherwise, directly taking the data file as a file to be imported.

11. The apparatus of claim 9, wherein, the import module further comprises, after detecting the import exception data of the executed import task: in a case where it is detected that the number of the import exception data exceeds a set fault tolerance threshold, performing the step of re-importing the detected import exception data into the target graph database; in a case where it is detected that the number of the import exception data does not exceed the set fault tolerance threshold, the service process feeds back a message indicating that the import is completed.

12. A system for importing data, characterized by comprising: the apparatus of claim 9 having import data, and one or more target graph databases; wherein the apparatus of the import data imports the data to be imported in the file to be imported into a running target graph database.

13. An electronic device, comprising: comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-8.

14. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-8.

15. A computer program product comprising a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-8. The program is executed by the processor to implement the method of any one of claims 1-8.

Images