Method and device for online importing graph data and graph database system

Abstract

The embodiment of the invention provides a method and device for online importing graph data and a graph database system. The method for online importing the graph data comprises the following steps: acquiring the graph data stored in a first storage format from a data source; performing outer sorting on the graph data to obtain sorted graph data; packaging the sorted graph data according to a second storage format specified by a target graph database to obtain a to-be-imported graph data file and a corresponding metadata file; and providing storage position information of the to-be-imported graph data file and the corresponding metadata file to a graph database server, so that the graph database server imports the graph data file into the target graph database, and updating the metadata file corresponding to the target graph database according to the obtained metadata file.

Description

Technical Field

[0001] The embodiments of the present specification generally relate to the field of computer technology, and more particularly to a method, device, and graph database system for online importing of graph data. Background Art

[0002] A graph database is a data management system that uses nodes and edges as basic storage units and is designed with the principle of efficient storage and query of graph data. As a non-relational database (NoSQL), graph databases are very suitable for scenarios that require processing and analysis of complex relationships and connectivity data, such as social network analysis, knowledge graphs, recommendation systems, and financial risk management. The above scenarios often require processing very complex queries. To improve efficiency, graph databases usually store data in an ordered storage engine. Therefore, how to maintain the orderliness of the original data structure when importing data while ensuring import performance has become a problem that needs to be solved. Summary of the invention

[0003] In view of the above, the embodiments of this specification provide a method, device and graph database system for online importing of graph data. Utilizing the method for online importing of graph data, graph data stored in a first storage format is obtained from a data source; the graph data is externally sorted to obtain sorted graph data; the sorted graph data is packaged according to a second storage format specified by a target graph database to obtain a graph data file to be imported and a corresponding metadata file; and the storage location information of the graph data file to be imported and the corresponding metadata file is provided to a graph database server, so that the graph database server imports the graph data file into the target graph database, and updates the metadata file corresponding to the target graph database according to the obtained metadata file. This achieves the maintenance of the orderliness of the original data structure and the guarantee of good import performance when importing data, and also enables high-performance online import of graph data.

[0004] According to one aspect of an embodiment of the present specification, a method for online importing graph data is provided, comprising: obtaining graph data stored in a first storage format from a data source; externally sorting the graph data to obtain sorted graph data; packaging the sorted graph data in accordance with a second storage format specified by a target graph database to obtain a graph data file to be imported and a corresponding metadata file; and providing storage location information of the graph data file to be imported and the corresponding metadata file to a graph database server, so that the graph database server imports the graph data file into the target graph database, and updates the metadata file corresponding to the target graph database according to the obtained metadata file.

[0005] According to another aspect of an embodiment of the present specification, there is provided an apparatus for online importing graph data, comprising: a data acquisition unit, configured to acquire graph data stored in a first storage format from a data source; an external sorting unit, configured to perform external sorting on the graph data to obtain sorted graph data; an adaptation processing unit, configured to package the sorted graph data in accordance with a second storage format specified by a target graph database to obtain a graph data file to be imported and a corresponding metadata file; and a storage location providing unit, configured to provide storage location information of the graph data file to be imported and the corresponding metadata file to a graph database server, so that the graph database server imports the graph data file into the target graph database, and updates the metadata file corresponding to the target graph database according to the obtained metadata file.

[0006] According to another aspect of an embodiment of the present specification, a graph database system is provided, including: a graph database server, configured to send a data import instruction to a data source server in response to receiving a graph data import request, wherein the graph data import request includes data source information, and the data source information includes storage location information corresponding to the target graph data; the data source server is configured to store the graph data to be imported in a first storage format; in response to receiving the data import instruction, obtain the target graph data targeted by the data import instruction; perform external sorting on the target graph data to obtain sorted target graph data; package the sorted target graph data according to a second storage format specified by the target graph database to obtain a graph data file to be imported and a corresponding metadata file; and provide the storage location information of the graph data file to be imported and the corresponding metadata file to the graph database server, so that the graph database server imports the graph data file into the target graph database, and updates the metadata file corresponding to the target graph database according to the obtained metadata file.

[0007] According to another aspect of an embodiment of the present specification, there is provided an apparatus for online importing graph data, comprising: at least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the method for online importing graph data as described above.

[0008] According to another aspect of an embodiment of the present specification, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the method for online importing graph data as described above is implemented.

[0009] According to another aspect of an embodiment of the present specification, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to implement the method for online importing graph data as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A further understanding of the nature and advantages of the content of this specification may be achieved by referring to the following drawings. In the drawings, similar components or features may have the same reference numerals.

[0011] Figure 1 An exemplary architecture of a method, apparatus, and graph database system for online importing graph data according to an embodiment of the present specification is shown.

[0012] Figure 2 A flowchart of an example of a method for online importing graph data according to an embodiment of the present specification is shown.

[0013] Figure 3 A schematic diagram showing an example of a process of externally sorting graph data according to an embodiment of the present specification.

[0014] Figure 4 A flowchart showing an example of a process for generating a node data sub-file and an edge data sub-file according to an embodiment of the present specification is shown.

[0015] Figure 5 A flowchart showing an example of a process of multi-way merging of node data and edge data according to an embodiment of the present specification.

[0016] Figure 6 A block diagram showing an example of a device for online importing graph data according to an embodiment of the present specification.

[0017] Figure 7 A block diagram showing an example of an external sorting unit in an apparatus for online importing graph data according to an embodiment of the present specification.

[0018] Figure 8 A block diagram showing an example of a graph database system according to an embodiment of the present specification.

[0019] Fig. 9 A schematic diagram showing an example of a device for online importing graph data according to an embodiment of the present specification. DETAILED DESCRIPTION

[0020] The subject matter described herein will be discussed below with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the embodiments of this specification. Various processes or components may be omitted, substituted or added to each example as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0021] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0022] In this specification, the term "ordered storage engine" may refer to a type of storage engine that organizes data into an ordered structure for fast retrieval. Since graph databases need to handle a large number of queries with complex relationships, ordered structures are often used to store data. Ordered storage engines may include, but are not limited to, one of the following: B-tree, LSM tree, index heap.

[0023] In this specification, the term "schema" is used to define and describe the structure and constraints of a data model. In some examples, a schema can be used to indicate that the first attribute of a binary string representing node data is an integer (int) and the second attribute is a single-precision floating point (float).

[0024] In this specification, the term "external sort" refers to a sorting algorithm used to process large-scale data, and is usually applicable to situations where the amount of data is too large to be loaded into memory for sorting at one time.

[0025] In this specification, the term "meta data" refers to data that describes, defines, and manages data. In some examples, metadata can be data that describes, defines, and manages a complete graph data. In some examples, the graph database can support a multi-graph model, that is, each graph database instance can contain multiple subgraphs, and each subgraph is logically isolated from each other (that is, each subgraph has independent nodes and edges) and physically isolated. Thus, metadata can be data that describes, defines, and manages a subgraph data. It can be understood that for a graph database containing multiple subgraphs, in addition to the metadata corresponding to each subgraph, it can also have metadata corresponding to the entire graph database.

[0026] In recent years, with the continuous development and innovation of graph database technology, the application of graph database has become more and more extensive, such as social network analysis, knowledge graph, recommendation system and financial risk management. When importing data into a graph database, it is usually necessary to first split the data and then upload the split data blocks to the graph database server in sequence. However, this method of uploading blocks and then inserting them into the ordered storage graph database in sequence will not only bring a large network bandwidth load, but also occupy a large amount of server resources during the import process, resulting in poor data import performance of the graph database. Therefore, how to maintain the orderliness of the original data structure when importing data and ensure the import performance has become a problem that needs to be solved.

[0027] In view of this, the embodiments of this specification propose a method, device and graph database system for online importing of graph data. Utilizing the method for online importing of graph data, the graph data to be imported and the corresponding schema are obtained according to the data source information and the target subgraph identifier included in the received graph data import request; the graph data to be imported is externally sorted according to the graph data to be imported and the corresponding schema to obtain the sorted graph data; the sorted graph data is then adapted according to the storage format of the target subgraph indicated by the target subgraph identifier in the graph database to obtain the underlying database file and the corresponding metadata file that match the storage format; and the underlying database file is imported into the target subgraph, and the metadata file corresponding to the target subgraph is updated according to the obtained metadata file. This achieves the maintenance of the orderliness of the original data structure and the guarantee of good import performance when importing data, and also enables high-performance online import of graph data.

[0028] The method, device and graph database system for online importing graph data according to the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1An exemplary architecture 100 of a method, apparatus, and graph database system for online importing of graph data according to an embodiment of the present specification is shown.

[0030] exist Figure 1 In the example, the network 110 is used to interconnect the terminal device 120 , the graph database server 130 , the data source server 140 and the intermediate server 150 .

[0031] The network 110 may be any type of network capable of interconnecting network entities. The network 110 may be a single network or a combination of various networks. In terms of coverage, the network 110 may be a local area network (LAN), a wide area network (WAN), etc. In terms of carrier media, the network 110 may be a wired network, a wireless network, etc. In terms of data exchange technology, the network 110 may be a circuit switching network, a packet switching network, etc.

[0032] The terminal device 120 may be any type of electronic computing device that can connect to the network 110, access a server or website on the network 110, process data or signals, etc. For example, the terminal device 120 may be a desktop computer, a laptop computer, a tablet computer, a smart phone, etc. Figure 1 Only one terminal device is shown in FIG. 1 , but it should be understood that a different number of terminal devices may be connected to the network 110 .

[0033] In one embodiment, the terminal device 120 can be used by a user. The terminal device 120 may include an application client (e.g., application client 121) that can provide various services to the user. In some cases, the application client 121 can interact with the graph database server 130. For example, the application client 121 can transmit a message input by the user to the graph database server 130, and receive a response associated with the above message from the graph database server 130 (e.g., a message indicating that the data import is successful). In this document, "message" may refer to any input information, such as storage location information and target subgraph identifiers that indicate the graph data to be imported and the corresponding schema, respectively.

[0034] The data source server 140 can be connected to the terminal device 120, the graph database server 130 and the intermediate server 150 through the network 110. The data source server 140 can store the graph data and the corresponding mode in a first storage format. In some implementations, the intermediate server 150 can perform at least some of the steps in the above-mentioned method for online importing graph data. In some implementations, the data source server 140 can perform at least some of the steps in the above-mentioned method for online importing graph data. In some implementations, the above-mentioned method for online importing graph data can be executed by a process other than the process for providing online graph database services in the graph database server 130. Thus, the graph data stored in the first storage format can be packaged into the graph data file to be imported and the storage location information of the corresponding metadata file stored in the second storage format specified by the graph database server 130, and provided to the graph database server 130, so as to realize the online import of graph data in the graph database server 130.

[0035] It should be understood that Figure 1 All network entities shown in the figure are exemplary. Any other network entities may be involved in the architecture 100 according to specific application requirements.

[0036] Figure 2 A flowchart of a method 200 for online importing graph data according to an embodiment of the present specification is shown.

[0037] like Figure 2 As shown, at 210, graph data stored in a first storage format is obtained from a data source.

[0038] In this embodiment, the graph data may be stored in a data source in a first storage format. In some examples, the data source may be Figure 1 The data source server shown in 140. In some examples, the client can upload the graph data to be imported and receive feedback storage location information. For example, the data source server can store the uploaded graph data in a first storage format, and the data source server can feedback the storage location information corresponding to the above graph data to the client. For another example, it can be uploaded to other servers in the same local area network as the graph database server, and the server that receives the above graph data can feedback the storage location information corresponding to the above graph data to the client. In some examples, the graph data can be divided into data files and schema files and stored separately. In these examples, the feedback storage location information may include storage location information of data files and storage location information of schema files.

[0039] In some examples, the acquisition of graph data stored in the first storage format may be performed in response to receiving a graph data import request. The graph data import request may include data source information. The data source information may include storage location information corresponding to the graph data to be imported. The storage location information may be used to indicate a resource storage path. In some examples, the storage location information may be represented by a URI (Uniform Resource Identifier).

[0040] In some examples, the graph data may include nodes, edges, and attributes corresponding to the nodes and edges, respectively. In some examples, the graph data may also include index data. In some examples, the index data may be established based on the attributes. In some examples, the index data may be embodied as a table to be imported into the graph data.

[0041] In some examples, the target graph database may include multiple subgraphs that are isolated from each other. In these examples, the graph data import request may also include a subgraph identifier of a target subgraph in the target graph database into which the graph data is to be imported.

[0042] At 220 , external sorting is performed on the graph data to obtain sorted graph data.

[0043] In this embodiment, each node data, edge data and corresponding node attribute data and edge attribute data in the graph data to be imported can be read. Then, external sorting is performed on each piece of data read to obtain sorted graph data. In some examples, the corresponding data file can be read according to the schema file to obtain each node data, edge data and corresponding node attribute data and edge attribute data in the graph data. In some examples, external sorting can be performed with the help of the ordered storage engine of the graph database, so that the sorted graph data can be obtained.

[0044] Figure 3 A schematic diagram showing an example of a process 300 for externally sorting graph data according to an embodiment of the present specification.

[0045] like Figure 3 As shown, the graph data to be imported can be divided into multiple node data sub-files (e.g., node data sub-file 1 to node data sub-file n) and multiple edge data sub-files (e.g., edge data sub-file 1 to edge data sub-file m) according to the capacity of the memory. In some examples, each node data sub-file and each edge data sub-file can be embodied as a data block (block) that does not exceed a specified capacity size. It is understood that the specified capacity size does not exceed the capacity of the memory.

[0046] Figure 4A flowchart showing an example of a process for generating a node data sub-file and an edge data sub-file according to an embodiment of the present specification is shown.

[0047] like Figure 4 As shown, at 410, the node data and edge data in the graph data are divided into data blocks of a specified size according to the capacity of the memory.

[0048] In this embodiment, the graph data may be divided into multiple node data and multiple edge data. The multiple node data and multiple edge data are then divided into data blocks in units of a specified size. It is understood that each data block may contain multiple node data or multiple edge data, and the size of each data block is not greater than the capacity of the memory.

[0049] At 420 , each node data or edge data in the obtained data block is packaged into a corresponding data record to generate a corresponding node data sub-file or edge data sub-file.

[0050] In this embodiment, according to the instructions of the schema file, for each obtained data block, each node data in the data block and the corresponding node attribute data can be packed into a node data record, thereby forming a node data sub-file corresponding to the node data block. Similarly, each edge data in the data block and the corresponding edge attribute data can be packed into an edge data record, thereby forming an edge data sub-file corresponding to the edge data block.

[0051] In the above manner, the attribute data in the graph data can be packaged with the corresponding node data or edge data to form a record, and then the node data sub-file and the edge data sub-file of the specified size can be specified, thereby providing technology for subsequent internal sorting.

[0052] Back to Figure 3 For each node data subfile, each node data record in the node data subfile can be arranged in order in the memory, so that an internally ordered node data subfile can be obtained. In some examples, the order can be arranged according to the unique identifier of the node (such as VID or Vertex ID). Similarly, for each edge data subfile, each edge data record in the edge data subfile can be arranged in order in the memory, so that an internally ordered edge data subfile can be obtained. In some examples, the order can be arranged according to the unique identifier of the edge. In some examples, each internally sorted node data subfile and edge data subfile can be persisted to the hard disk.

[0053] Furthermore, the obtained internally ordered node data subfiles can be multi-way merged in various ways, and the node data records in the node data subfiles can be sorted to obtain sorted node data. Similarly, the obtained internally ordered edge data subfiles can be multi-way merged, and the edge data records in the edge data subfiles can be sorted to obtain sorted edge data. In some examples, the sorting can be performed according to the unique identifiers of the points and edges respectively.

[0054] Through the above method, the generated node data sub-files and edge data sub-files can be first sorted internally, and then the sorted node data and sorted edge data can be further obtained based on the internally ordered node data sub-files and edge data sub-files, thereby realizing two-stage efficient sorting and enriching the external sorting method.

[0055] Figure 5 A flowchart of an example of a process 500 for performing multi-way merging of node data and edge data according to an embodiment of the present specification is shown.

[0056] At 510, the obtained internally ordered node data sub-files and internally ordered edge data sub-files are multi-way merged to obtain node data and edge data stored in a log structure merge tree structure.

[0057] In this embodiment, a log-structured merge tree (LSM tree) can be used to perform multi-way merging of the internally ordered node data sub-files and edge data sub-files to obtain multiple sorted string tables (SSTable), each of which contains multiple ordered node data records or edge data records. SSTable files are persistent data files on disk, and the data in SSTable files are usually arranged in the order of specified keys. In some examples, the node data records in the sorted string table can be arranged in the order of the unique identifiers of the nodes, and the edge data records in the sorted string table can be arranged in the order of the unique identifiers of the edges. In some examples, there may be SSTable files of different levels (e.g., Level 0, Level 1, Level 2...Level n). SSTable files of different levels may contain data of different time periods and / or the results of different merge operations. In an LSM tree using multi-level SSTable files, the SSTable closer to the top (e.g., Level 0) is newer, and the SSTable closer to the bottom (e.g., Level n) is older.

[0058] In some examples, when a log-structured merge tree is used for multi-way merging, an ordered storage structure may be used in the memory to temporarily store newly written data (e.g., a node data subfile or an edge data subfile). The ordered storage structure may be, for example, a sorted tree (a red-black tree, a balanced binary tree, etc.), a skip list, an ordered array, etc. When the data stored in the ordered storage structure in the memory reaches a predetermined size, the data is written to a storage medium such as a disk or flash memory, and a sorted string table (e.g., a sorted string table containing node data records or a sorted string table containing edge data records) is formed. Thus, at least part of the node data and edge data may be persisted in the disk in the form of a sorted string table.

[0059] At 520 , all node data and edge data in the log structure merge tree are sequentially read to obtain sorted node data and sorted edge data.

[0060] In this embodiment, all node data can be read in the order of the log structure merge tree to obtain the sorted node data. Similarly, all edge data can be read in the order of the log structure merge tree to obtain the sorted edge data.

[0061] Through the above method, the ordered storage of node data and edge data is achieved by constructing a log-structured merge tree, and the sorted node data and the sorted edge data are obtained according to the reading order of the log-structured merge tree, thereby using the data structure of high-performance data storage to achieve efficient external sorting of node data and edge data.

[0062] In some implementations, the aforementioned method of performing external sorting on node data and edge data may be referred to to perform external sorting on index data to obtain sorted index data, thereby achieving efficient sorting of index data.

[0063] Back to Figure 2 At 230, the sorted graph data is packaged according to the second storage format specified by the target graph database to obtain the graph data file to be imported and the corresponding metadata file.

[0064] In some examples, the sorted graph data can be packaged into a second storage format specified by the target graph database, thereby obtaining the graph data file to be imported and the corresponding metadata file. In some examples, the second storage format can be the storage format of the target subgraph specified in the target database. In some examples, the storage format can be used to indicate the order of non-empty attributes and empty attributes. In some examples, the storage format can be used to indicate whether the incoming edges and outgoing edges are stored together with the point data. In some examples, the storage format can also be used to indicate the way to split the value (value) when the data volume exceeds a threshold, so as to avoid a significant performance degradation due to excessive data volume at a certain node in the tree structure.

[0065] It is understandable that the corresponding metadata file can also be stored in the second storage format. Thus, the obtained graph data file to be imported and the corresponding metadata file can be used as the underlying database file adapted to the target database.

[0066] In some examples, the obtained graph data file to be imported and the corresponding metadata file can be stored in a transit storage device. In these examples, the transit storage device and the graph database server are located in the same local area network. For example, the transit storage device can be Figure 1 The intermediate server 150 shown in . In these examples, the storage location information of the graph data file and the corresponding metadata file can indicate the storage location on the transit storage device. In some examples, the transit storage device can perform the above steps 210-230. In these examples, the obtained graph data file to be imported and the corresponding metadata file can be directly stored locally. Thus, by storing the obtained graph data file to be imported and the corresponding metadata file in a device located in the same local area network as the graph database, high-speed data transmission can be achieved, and the performance consumption of the data import process is effectively reduced, so that the graph database server can efficiently import the above graph data files into the target graph database.

[0067] In this embodiment, the graph database server can import the graph data file to be imported and the corresponding metadata file as the underlying database file directly into the target database. In some examples, the graph data file to be imported and the corresponding metadata file can be imported into the target subgraph indicated by the subgraph identifier. In some examples, the target subgraph can be a new subgraph created with the above-mentioned underlying database file. Accordingly, the obtained metadata file can be determined as the metadata file corresponding to the above-mentioned target subgraph. In some examples, the original data may exist in the target subgraph, and the original data can be replaced by the above-mentioned underlying database file. Accordingly, the corresponding original metadata file can be replaced by the obtained metadata file. Thereby, the full import of the subgraph data is realized.

[0068] Optionally, the original data of the target subgraph may be incrementally updated using the underlying database file. Accordingly, the original metadata file corresponding to the target subgraph may be updated using the obtained metadata file, thereby realizing incremental import of subgraph data.

[0069] use Figure 1-Figure 5A method for online importing of graph data disclosed in . Using the method for online importing of graph data, graph data stored in a first storage format can be obtained from a data source; the graph data can then be externally sorted to obtain sorted graph data; the sorted graph data can be packaged according to a second storage format specified by a target graph database to obtain a graph data file to be imported and a corresponding metadata file; and the storage location information of the graph data file to be imported and the corresponding metadata file can be provided to a graph database server, so that the graph database server imports the graph data file into the target graph database, and updates the metadata file corresponding to the target graph database according to the obtained metadata file. In this way, the orderliness of the original data structure is maintained during data import, and the performance consumption during import is effectively reduced, so that high-performance online import of graph data can be achieved. In addition, by splitting the entire data import process into two relatively independent parts, the external sorting stage and the overall data import stage, only the resources of the graph database server need to be occupied in the overall data import part, and the phased execution also reduces the difficulty of monitoring and maintenance of the data import process.

[0070] Figure 6 A block diagram of an example of an apparatus 600 for online importing of graph data according to an embodiment of the present specification is shown. Figure 2-Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0071] like Figure 6 As shown, the apparatus 600 for online importing graph data may include a data acquisition unit 610 , an external sorting unit 620 , an adaptation processing unit 630 , and a storage location providing unit 640 .

[0072] The data acquisition unit 610 is configured to acquire graph data stored in a first storage format from a data source.

[0073] In one example, the graph data also includes index data.

[0074] The external sorting unit 620 is configured to perform external sorting on the graph data to obtain sorted graph data.

[0075] Figure 7 A block diagram showing an example of an external sorting unit 700 in an apparatus for online importing graph data according to an embodiment of the present specification.

[0076] like Figure 7 As shown, the external sorting unit 700 may include a data partitioning module 710 , an internal sorting module 720 and a merging module 730 .

[0077] The data partitioning module 710 is configured to partition the graph data according to the capacity of the memory to generate a plurality of node data sub-files and edge data sub-files.

[0078] In one example, the data partitioning module 710 is further configured to: partition the node data and edge data in the graph data into data blocks of specified sizes according to the capacity of the memory; and pack each node data or edge data in the obtained data blocks into corresponding data records to generate corresponding node data sub-files or edge data sub-files.

[0079] The internal sorting module 720 is configured to sort the data records in the obtained node data sub-file and edge data sub-file in the memory respectively to obtain an internally ordered node data sub-file and an internally ordered edge data sub-file.

[0080] The merging module 730 is configured to perform multi-way merging on the obtained internally ordered node data sub-files and internally ordered edge data sub-files respectively to obtain sorted node data and sorted edge data.

[0081] In one example, the merging module 730 is further configured to: perform multi-way merging on the obtained internally ordered node data sub-files and the internally ordered edge data sub-files to obtain node data and edge data stored in a log-structured merge tree structure, wherein at least part of the node data and edge data are persisted in the disk in the form of a sorted string table; and sequentially read all node data and edge data in the log-structured merge tree respectively to obtain sorted node data and sorted edge data.

[0082] Back to Figure 6 The adaptation processing unit 630 is configured to package the sorted graph data according to the second storage format specified by the target graph database to obtain the graph data file to be imported and the corresponding metadata file.

[0083] In one example, the target graph database includes a plurality of subgraphs isolated from each other. The metadata file corresponding to the target graph database is a metadata file corresponding to a newly created subgraph imported by the graph data file.

[0084] The storage location providing unit 640 is configured to provide the storage location information of the graph data file to be imported and the corresponding metadata file to the graph database server, so that the graph database server imports the graph data file into the target graph database and updates the metadata file corresponding to the target graph database according to the obtained metadata file.

[0085] In one example, the apparatus 600 for online importing graph data may further include: a data transfer unit 650 configured to store the graph data file to be imported and the corresponding metadata file in a transfer storage device, wherein the transfer storage device and the graph database server are located in the same local area network.

[0086] The operations of the data acquisition unit 610, the external sorting unit 620, the adaptation processing unit 630, the storage location providing unit 640 and the data transfer unit 650 can refer to the above Figure 2-Figure 5 Describes the relevant operations.

[0087] Figure 8 A block diagram of an example of a graph database system 800 according to an embodiment of the present specification is shown.

[0088] like Figure 8 As shown, the graph database system 800 may include a graph database server 810 and a data source server 820. In some cases, the graph database server 800 may store only one graph. In some cases, the graph database server 810 may store multiple subgraphs that are isolated from each other, such as subgraphs. Figure 1 and metadata 1. Figure 2 and metadata 2, and so on. Each subgraph and the corresponding metadata are logically and physically isolated from other subgraphs and the corresponding metadata. In some examples, there may also be shared metadata between the subgraphs. In these examples, data import only modifies the subgraph involved and the metadata corresponding to the subgraph, and does not modify the shared metadata.

[0089] In response to receiving the graph data import request, the graph database server 810 may send a data import instruction to the data source server 820, wherein the graph data import request includes data source information, and the data source information includes storage location information corresponding to the target graph data. For a detailed description of the graph data import request, please refer to Figure 2Related description of 210 in the embodiment. The data source server 820 can store the graph data to be imported in a first storage format. In some examples, the graph data can be stored in the form of a data file and a Schema file respectively. In response to receiving the data import instruction, the data source server 820 can obtain the target graph data for which the data import instruction is directed. Afterwards, the data source server 820 can perform external sorting on the target graph data to obtain the sorted target graph data. Furthermore, the data source server 820 can package the sorted target graph data in accordance with the second storage format specified by the target graph database (e.g., the graph database provided by the graph database server 810) to obtain the graph data file to be imported and the corresponding metadata file. Then, the data source server 820 can provide the location information of the obtained graph data file and the corresponding metadata file to the graph database server 810. Thus, the graph database server 810 can import the graph data file into the target graph database, and update the metadata file corresponding to the target graph database according to the obtained metadata file. In some examples, the graph database server 810 can import the graph data file into a specified subgraph in the target graph database, and update the metadata file corresponding to the specified subgraph accordingly.

[0090] It should be noted that the above specific operations can refer to the aforementioned Figure 2-Figure 5 The corresponding description in the embodiments will not be repeated here.

[0091] Reference above Figures 1 to 8 , describes a method and device for online importing graph data and an embodiment of a graph database system according to an embodiment of this operating manual.

[0092] The device for online importing of graph data in the embodiment of this specification can be implemented by hardware, software, or a combination of hardware and software. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the device in which it is located reading the corresponding computer program instructions in the memory into the memory and running them. In the embodiment of this specification, the device for online importing of graph data can be implemented, for example, using an electronic device.

[0093] Fig. 9 A schematic diagram showing an example of an apparatus 900 for online importing graph data according to an embodiment of the present specification.

[0094] like Fig. 9As shown, the apparatus 900 for online importing graph data may include at least one processor 910, a memory (e.g., a non-volatile memory) 920, a memory 930, and a communication interface 940, and the at least one processor 910, the memory 920, the memory 930, and the communication interface 940 are connected together via a bus 950. At least one processor 910 executes at least one computer-readable instruction stored or encoded in the memory (i.e., the above-mentioned element implemented in the form of software).

[0095] In one embodiment, computer executable instructions are stored in a memory, which when executed cause at least one processor 910 to: obtain graph data stored in a first storage format from a data source; perform external sorting on the graph data to obtain sorted graph data; package the sorted graph data in accordance with a second storage format specified by a target graph database to obtain a graph data file to be imported and a corresponding metadata file; and provide storage location information of the graph data file to be imported and the corresponding metadata file to a graph database server, so that the graph database server imports the graph data file into the target graph database, and updates the metadata file corresponding to the target graph database according to the obtained metadata file.

[0096] It should be understood that the computer executable instructions stored in the memory, when executed, cause at least one processor 910 to perform the above combined operations in various embodiments of the present specification. Figure 1-Figure 5 Describes the various operations and functions.

[0097] According to one embodiment, a program product such as a computer-readable medium is provided. The computer-readable medium may have instructions (i.e., the above-mentioned elements implemented in the form of software), which, when executed by a computer, causes the computer to perform the above-mentioned combination of various embodiments of this specification. Figure 1-Figure 5 Describes the various operations and functions.

[0098] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0099] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above-mentioned embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute a part of the present invention.

[0100] The computer program code required for the operation of each part of this specification can be written in any one or more programming languages, including object-oriented programming languages, such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB, NET and Python, conventional procedural programming languages ​​such as C language, Visual Basic 2003, Perl, COBOL2002, PHP and ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be run on the user's computer, or run on the user's computer as an independent software package, or run partly on the user's computer and another part on the remote computer, or all on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service, such as software as a service (SaaS).

[0101] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0102] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0103] Not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be determined according to needs. The device structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0104] The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply "preferred" or "advantageous" over other embodiments. The detailed description includes specific details for the purpose of providing an understanding of the described techniques. However, the techniques may be implemented without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0105] The optional implementation modes of the embodiments of the present specification are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present specification are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present specification, various simple modifications can be made to the technical solutions of the embodiments of the present specification, and these simple modifications all belong to the protection scope of the embodiments of the present specification.

[0106] The above description of the contents of this specification is provided to enable any person of ordinary skill in the art to implement or use the contents of this specification. Various modifications to the contents of this specification will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the contents of this specification. Therefore, the contents of this specification are not limited to the examples and designs described herein, but are consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for online importing graph data, comprising: Acquire graph data stored in a first storage format from a data source; externally sorting the graph data to obtain sorted graph data; The sorted graph data is packaged according to a second storage format specified by the target graph database to obtain a graph data file to be imported and a corresponding metadata file, and the metadata file is stored according to the second storage format to form an underlying database file compatible with the target database; as well as The graph data file is imported into the target graph database, and the metadata file corresponding to the target graph database is updated according to the obtained metadata file.

2. The method of claim 1, wherein: The step of importing the graph data file into the target graph database and updating the metadata file corresponding to the target graph database according to the obtained metadata file includes: The storage location information of the graph data file to be imported and the corresponding metadata file is provided to the graph database server, so that the graph database server imports the graph data file into the target graph database, and updates the metadata file corresponding to the target graph database according to the obtained metadata file.

3. The method of claim 2, wherein: Before providing the storage location information of the graph data file and the corresponding metadata file to the graph database server, the method further includes: The graph data file to be imported and the corresponding metadata file are stored in a transit storage device, wherein the transit storage device and the graph database server are located in the same local area network, and the storage location information indicates the storage location on the transit storage device.

4. The method of claim 1, wherein: The external sorting of the graph data to obtain sorted graph data includes: Dividing the graph data according to the capacity of the memory to generate a plurality of node data sub-files and edge data sub-files; sorting the data records in the obtained node data sub-file and edge data sub-file in the memory respectively to obtain an internally ordered node data sub-file and an internally ordered edge data sub-file; and The obtained internally ordered node data sub-files and internally ordered edge data sub-files are respectively subjected to multi-way merging to obtain sorted node data and sorted edge data.

5. The method of claim 4, wherein: The step of dividing the graph data according to the capacity of the memory to generate a plurality of node data sub-files and edge data sub-files includes: Dividing the node data and edge data in the graph data into data blocks of specified sizes according to the capacity of the memory; and Each node data or edge data in the obtained data block is packaged into a corresponding data record to generate a corresponding node data sub-file or edge data sub-file.

6. The method of claim 4, wherein: The step of performing multi-way merging on the obtained internally ordered node data sub-files and the internally ordered edge data sub-files to obtain sorted node data and sorted edge data comprises: Performing multi-way merging on the obtained internally ordered node data sub-files and internally ordered edge data sub-files to obtain node data and edge data stored in a log structure merge tree structure, wherein at least part of the node data and edge data are persisted in a disk in the form of a sorted string table; and All node data and edge data in the log structure merge tree are read sequentially to obtain sorted node data and sorted edge data.

7. The method according to any one of claims 1 to 6, wherein: The target graph database includes a plurality of subgraphs isolated from each other. The metadata file corresponding to the target graph database is a metadata file corresponding to a newly created subgraph imported by the graph data file. The graph data also includes index data.

8. A device for online importing graph data, comprising: A data acquisition unit, configured to acquire graph data stored in a first storage format from a data source; an external sorting unit, configured to perform external sorting on the graph data to obtain sorted graph data; an adaptation processing unit, configured to package the sorted graph data according to a second storage format specified by a target graph database to obtain a graph data file to be imported and a corresponding metadata file, wherein the metadata file is stored according to the second storage format to form an underlying database file adapted to the target database; as well as The importing unit is configured to import the graph data file into the target graph database, and update the metadata file corresponding to the target graph database according to the obtained metadata file.

9. A graph database system, comprising: A data source server is configured to store graph data to be imported in a first storage format; In response to receiving a data import instruction, acquiring target graph data targeted by the data import instruction; Performing external sorting on the target graph data to obtain sorted target graph data; packaging the sorted target graph data according to a second storage format specified by the target graph database to obtain a graph data file to be imported and a corresponding metadata file, wherein the metadata file is stored according to the second storage format to form an underlying database file compatible with the target database; The graph data file is imported into the target graph database, and the metadata file corresponding to the target graph database is updated according to the obtained metadata file.

10. A device for online importing graph data, comprising: At least one processor, a memory coupled to the at least one processor, and a computer program stored on the memory, wherein the at least one processor executes the computer program to implement the method for online importing graph data as described in any one of claims 1 to 7.

Images