A data processing method, apparatus, device, and storage medium for a graph database

Abstract

The present disclosure provides a method, apparatus, device, and storage medium for processing data of a graph database. The method includes: after receiving a write request for first target graph data, determining a B-tree corresponding to each node in the first target graph data, storing graph data of at least one edge type corresponding to the node in a leaf node of the corresponding B-tree, and storing index information of the leaf node in an internal node corresponding to the leaf node in the B-tree. Determining a key-value pair corresponding to the node in the key-value storage system, storing an index key of the node in the key of the corresponding key-value pair, and storing graph data or index information stored on the node in the value of the corresponding key-value pair. The present disclosure organizes graph data into a B-tree storage structure. Since the storage structure of the B-tree supports efficient and low-latency data reading and writing, the graph database implemented based on the B-tree also supports efficient and low-latency data reading and writing, especially for nodes with a large number of edges, which greatly improves the data reading and writing efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular, to a method, apparatus, device, and storage medium for processing data of a graph database. Background Art

[0002] With the rapid development of big data industries such as social networking, e-commerce, finance, retail, and the Internet of Things, a huge and complex data relationship network has been woven in the real world. The relationships between the data that the big data industries need to process grow geometrically with the data volume. Traditional databases are difficult to handle relationship operations, and there is an urgent need for a database that supports complex data relationship operations. Therefore, graph databases have emerged.

[0003] However, the current graph data processing method of graph databases is not friendly to the data reading and writing of a small number of nodes with a large number of edges. For the above types of graph data, there is a problem of low data reading and writing efficiency. Summary of the Invention

[0004] To solve the above technical problems, an embodiment of the present disclosure provides a method for processing data of a graph database.

[0005] In a first aspect, the present disclosure provides a method for processing data of a graph database, the method comprising:

[0006] Receiving a write request for first target graph data; the first target graph data includes nodes and edges, and the edge type of the edges is used to characterize the relationship type between the nodes associated by the edges;

[0007] Determining a balanced B-tree corresponding to each node in the first target graph data, and storing graph data of at least one edge type corresponding to each node in a leaf node of the corresponding B-tree, and storing index information of the leaf node in an internal node corresponding to the leaf node in the B-tree; the index information of the B-tree is stored in the root node of the B-tree, and the index information of the B-tree is determined based on the node identifier of the node corresponding to the B-tree and the at least one edge type corresponding to the B-tree;

[0008] In a key-value storage system, determining a key-value pair corresponding to each node in the B-tree, and persistently storing the index key in the index information of each node in the key of the corresponding key-value pair, and persistently storing the graph data or index information stored on each node in the value of the corresponding key-value pair; the key-value storage system is used to persistently store graph data.

[0009] In an optional implementation manner, before determining the B-tree corresponding to each node in the first target graph data, further comprising:

[0010] Generate the log information corresponding to the write request; wherein, the log information is used to record the operation information of the write request;

[0011] Store the log information corresponding to the write request into the key - value storage system;

[0012] When it is determined that the B - tree corresponding to the log information meets the preset first replay condition and / or the current system idle state meets the preset second replay condition, obtain the log information from the key - value storage system;

[0013] Correspondingly, determining the B - tree corresponding to each node in the first target graph data includes:

[0014] Based on the operation information of the write request recorded in the log information, determine the B - tree corresponding to each node in the first target graph data corresponding to the operation information.

[0015] In an alternative embodiment, storing the log information corresponding to the write request into the key - value storage system includes:

[0016] Store the log information corresponding to the write request in the log stream of the B - tree corresponding to the first target graph data in the key - value storage system in the form of key - value pairs; wherein, the key - value storage system stores log streams respectively having a mapping relationship with each B - tree.

[0017] In an alternative embodiment, after generating the log information corresponding to the write request, further includes:

[0018] Record the index information of the B - tree corresponding to the first target graph data in a preset dirty linked list in the cache; the preset dirty linked list is used to maintain the index information of the B - trees with un - replayed log information;

[0019] When obtaining the index information of the B - tree corresponding to the first target graph data from the preset dirty linked list, execute the step of determining that the B - tree corresponding to the log information meets the preset first replay condition and / or the current system idle state meets the preset second replay condition.

[0020] In an alternative embodiment, multiple data shards corresponding to the entire graph data stored in the key - value storage system are respectively stored in the caches of different devices; the multiple data shards are obtained by partitioning the entire graph data based on a preset partitioning strategy, and the preset partitioning strategy is determined based on the application - scenario characteristics of the entire graph data.

[0021] In an alternative embodiment, the storage capacity of the leaf nodes of the B-tree is preconfigured based on the read-write feature information of the graph data stored on the leaf nodes, and the read-write feature information is used to characterize the requirements of the graph data for data read-write amplification.

[0022] In an alternative embodiment, the method further includes:

[0023] Receiving a read request for second target graph data through the graph query engine layer, and passing the read request to the graph storage engine layer;

[0024] Using the graph storage engine layer, determining the B-tree corresponding to the read request based on the node identifiers of the nodes and the edge types in the second target graph data, and executing the read request based on the B-tree to obtain a read result, and returning the read result to the graph query engine layer.

[0025] In a second aspect, an embodiment of the present disclosure provides a data processing device for a graph database, and the device includes:

[0026] A first receiving module, configured to receive a write request for first target graph data; the first target graph data includes nodes and edges, and the edge type of the edge is used to characterize the relationship between the nodes associated with the edge;

[0027] A first storage module, configured to determine a balanced B-tree corresponding to each node in the first target graph data, and store the graph data of at least one edge type corresponding to each node in the leaf nodes of the corresponding B-tree, and store the index information of the leaf nodes in the internal nodes corresponding to the leaf nodes in the B-tree; the index information of the B-tree is stored in the root node of the B-tree, and the index information of the B-tree is determined based on the node identifier of the node corresponding to the B-tree and the at least one edge type corresponding to the B-tree;

[0028] A second storage module, configured to determine a key-value pair corresponding to each node in the B-tree in the key-value storage system, and persistently store the index key in the index information of each node in the key of the corresponding key-value pair, and persistently store the graph data or index information stored on each node in the value of the corresponding key-value pair; the key-value storage system is used to persistently store graph data.

[0029] In a third aspect, the present disclosure provides a computer-readable storage medium, in which instructions are stored, and when the instructions are run on a terminal device, the terminal device is enabled to implement the above method.

[0030] Fourthly, the present disclosure provides a data processing device for a graph database, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above method is implemented.

[0031] Fifthly, the present disclosure provides a computer program product, which includes computer programs / instructions. When the computer programs / instructions are executed by a processor, the above method is implemented.

[0032] The technical solutions provided by the embodiments of the present disclosure have at least the following advantages compared with the prior art:

[0033] The embodiments of the present disclosure provide a data processing method for a graph database. After receiving a write request for first target graph data, first determine a balanced B-tree corresponding to each node in the first target graph data, and store graph data of at least one edge type corresponding to each node in leaf nodes of the corresponding B-tree, and store index information of the leaf nodes in internal nodes corresponding to the leaf nodes in the B-tree; the index information of the B-tree is stored in the root node of the B-tree, and the index information of the B-tree is determined based on the node identifier of the node corresponding to the B-tree and the at least one edge type corresponding to the B-tree. Furthermore, in the key-value storage system, determine key-value pairs corresponding to each node in the B-tree, and persistently store the index keys in the index information of each node in the keys of the corresponding key-value pairs, and persistently store the graph data or index information stored on each node in the values of the corresponding key-value pairs; the key-value storage system is used to persistently store graph data. The embodiments of the present disclosure organize graph data into a storage structure of a B-tree. Since the storage structure of the B-tree supports efficient and low-latency data reading and writing, the graph database implemented based on the B-tree can also support efficient and low-latency data reading and writing. Especially for nodes with a large number of edges, the data reading and writing efficiency can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0035] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0036] Figure 1 It is a flowchart of a graph data storage method provided by the embodiments of the present disclosure;

[0037] Figure 2 Structural schematic diagram of a B-tree provided by an embodiment of the present disclosure;

[0038] Figure 3 Architectural schematic diagram of a single-machine memory engine provided by an embodiment of the present disclosure;

[0039] Figure 4 Structural schematic diagram of a graph data storage device provided by an embodiment of the present disclosure;

[0040] Figure 5 Structural schematic diagram of a graph data storage device provided by an embodiment of the present disclosure. Detailed implementation manners

[0041] In order to more clearly understand the above objects, features and advantages of the present disclosure, the solution of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0042] Many specific details are set forth in the following description in order to fully understand the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all the embodiments.

[0043] In the big data industry, the relationships between data grow geometrically with the increase in the amount of data. Therefore, the big data industry requires a database that can handle complex data relationship operations. Compared with traditional databases, graph databases are better at handling data relationship operations.

[0044] A graph database is a database that uses a graph structure for semantic queries and associates data items in storage with a set of nodes and edges respectively. A graph database is a non-relational database that can solve the limitations of traditional relational databases. A graph database is used for storing data securely without loss and for related queries, updates, controls, and organizing data.

[0045] Currently, graph databases are usually implemented based on a storage structure that stores a single edge in a single key-value pair or aggregates all the adjacent edges of a vertex. For nodes with a large number of edges, there is a problem of low data read and write efficiency. To this end, the embodiments of the present disclosure organize graph data into a B-tree storage structure. Since the B-tree storage structure supports efficient and low-latency data read and write, a graph database implemented based on a B-tree can also support efficient and low-latency data read and write. Especially for nodes with a large number of edges, the data read and write efficiency can be greatly improved.

[0046] B - tree, also known as balanced multi - way search tree, includes a root node root, leaf nodes leaf, and internal nodes internal. Among them, the root node refers to the node without a parent node, the leaf node refers to the node without child nodes, and the internal node refers to other nodes in the B - tree except the root node and leaf nodes, which have both a parent node and child nodes.

[0047] In practical applications, an m - order B - tree (also called an m - ary tree) can have the following characteristics:

[0048] The maximum value among the number of child nodes of all nodes in the B - tree is called the order of the B - tree, denoted as M; each node in the B - tree has at most M sub - trees, that is, if the order of the B - tree is defined as M, then the number of child nodes of any node in this B - tree cannot exceed M; if the root node of the B - tree is not a terminal node, it has at least two sub - trees; except for the root node and leaf nodes, all nodes have at least m / 2 sub - trees, and so on.

[0049] To improve the data reading and writing efficiency of the graph database, the embodiments of the present disclosure provide a method for processing graph database data. After receiving a write request for the first target graph data, first determine the balanced B - tree corresponding to each node in the first target graph data, and store the graph data of at least one edge type corresponding to each node in the leaf nodes of the corresponding B - tree, and store the index information of the leaf nodes in the internal nodes corresponding to the leaf nodes in the B - tree; the index information of the B - tree is stored in the root node of the B - tree, and the index information of the B - tree is determined based on the node identifier of the node corresponding to the B - tree and the at least one edge type corresponding to the B - tree. Furthermore, in the key - value storage system, determine the key - value pair corresponding to each node in the B - tree, and persistently store the index key in the index information of each node in the key of the corresponding key - value pair, and persistently store the graph data or index information stored on each node in the value of the corresponding key - value pair; the key - value storage system is used to persistently store graph data. The embodiments of the present disclosure organize graph data into the storage structure of a B - tree. Since the storage structure of the B - tree supports efficient and low - latency data reading and writing, the graph database implemented based on the B - tree can also support efficient and low - latency data reading and writing. Especially for nodes with a large number of edges, it can greatly improve the data reading and writing efficiency.

[0050] Based on this, the embodiments of the present disclosure provide a method for processing graph database data, refer to Figure 1 , which is a flowchart of a method for processing graph database data provided by the embodiments of the present disclosure. The method specifically includes:

[0051] S101: Receive a write request for the first target graph data.

[0052] Among them, the first target graph data includes nodes and edges, and the edge type of the edge is used to characterize the relationship type between the nodes associated with the edge.

[0053] The embodiments of the present disclosure are applied to a graph database and are used to implement functions such as reading and writing graph data.

[0054] In the embodiments of the present disclosure, the first target graph data can be any graph data to be written. Among them, the first target graph data includes nodes and edges. Each node is used to represent an entity, such as a user in social network data. Each edge is used to represent the relationship between the corresponding entities, such as the friendship relationship and the follow relationship between users in social network data. The edge type is used to characterize the relationship type between the nodes associated with the edge. For example, the friendship relationship belongs to the friendship relationship type, and the follow relationship belongs to the follow relationship type, etc.

[0055] S102: Determine the balanced B-tree corresponding to each node in the first target graph data, store the graph data of at least one edge type corresponding to each node in the leaf node of the corresponding B-tree, and store the index information of the leaf node in the internal node corresponding to the leaf node in the B-tree.

[0056] Among them, the index information of the B-tree is stored in the root node of the B-tree, and the index information of the B-tree is determined based on the node identifier of the node corresponding to the B-tree and the at least one edge type corresponding to the B-tree.

[0057] In the embodiments of the present disclosure, after receiving a write request for the first target graph data, first determine whether there is a B-tree corresponding to each node in the first target graph data. For the nodes in the first target graph data that have a B-tree, store the graph data of at least one edge type corresponding to the node in the leaf node of the corresponding B-tree, and store the index information of the leaf node in the internal node corresponding to each leaf node in the B-tree.

[0058] For the nodes in the first target graph data that do not have a B-tree, construct a B-tree for the node, and determine the index information for the B-tree based on the node identifier of the node and the edge type of the graph data to be stored in the B-tree. For example, use the hash algorithm to calculate the hash value based on the node identifier and the edge type as the index information of the B-tree. The embodiments of the present disclosure do not specifically limit the method for determining the index information based on the node identifier of the node corresponding to the B-tree and the at least one edge type corresponding to the B-tree.

[0059] Store graph data using the storage structure of a B-tree. Specifically, each B-tree corresponds to a node in the first target graph data that has at least one edge, and the B-tree is used to store graph data of at least one edge type corresponding to the node. Among them, the edge type refers to the relationship type to which the edges between nodes in the graph data belong. For example, the edge type can include the friendship relationship type, the follow relationship type, etc. between users in social network data.

[0060] In the embodiments of the present disclosure, for nodes with more connected edges, such as the nodes corresponding to big V users in social network data, use the storage structure of the B-tree to store them. Compared with other storage structures, the storage structure of the B-tree can greatly improve the read and write efficiency of the above types of node data, and provide users with an efficient and low-latency data access service.

[0061] As Figure 2 shown, it is a schematic structural diagram of a B-tree provided by the embodiments of the present disclosure. Among them, the root node 201 of the B-tree is used to store the index information of the B-tree and the index information of the internal nodes and leaf nodes connected to the root node 201. The index information includes index keys such as pageKey1 - pageKey n. Based on the index information of the B-tree, the root node 201 of the B-tree can be indexed. By obtaining the index keys of each node stored in the root node, the read and write operations for the graph data stored on each leaf node in the B-tree can be realized. The index information of the internal node 202 of the B-tree is pageKey1 stored in the root node. Based on pageKey1, the internal node 202 can be indexed. The internal node 202 is used to store the index information pageKey m of the corresponding leaf node 203. Based on pageKey m, the leaf node 203 of the B-tree can be indexed. The leaf node 203 is used to store graph data of at least one edge type corresponding to the first node in the graph data, such as edge1 - edgei.

[0062] In the embodiments of the present disclosure, taking the first B-tree as an example, the first B-tree can be any B-tree storing graph data. The index information stored in the root node of the first B-tree can be index information determined based on the node identifier of the first node and at least one edge type. For example, the index information can be a string obtained by performing a hash operation on the node identifier and the edge type, etc.

[0063] In an optional implementation manner, the storage capacity of the leaf node of the B-tree can be pre-configured based on the read and write feature information of the graph data stored on the leaf node. Among them, the storage capacity of the leaf node refers to the maximum length of graph data that the leaf node can store. As described above Figure 2 shown, the storage capacity of the leaf node 203 is 10. In addition, the read and write feature information is used to characterize the requirements of the graph data stored on the leaf node for data read and write amplification.

[0064] Since the larger the storage capacity of the leaf nodes, the greater the write amplification of the B-tree, and the smaller the storage capacity of the leaf nodes, the greater the read amplification of the B-tree. To seek a balance between the write and read amplifications of the B-tree, the embodiments of the present disclosure provide a custom setting function for the storage capacity of the leaf nodes of the B-tree, and users can configure the storage capacity of the leaf nodes of the B-tree based on the graph data read and write requirements.

[0065] S103: In the key-value storage system, determine the key-value pairs corresponding to each node in the B-tree, and persistently store the index keys in the index information of each node in the keys of the corresponding key-value pairs, and persistently store the graph data or index information stored on each node in the values of the corresponding key-value pairs; the key-value storage system is used to persistently store graph data.

[0066] In practical applications, the disk storage of a graph database is usually implemented based on a key-value storage KV store system, and the key-value storage system is used to persistently store graph data. Therefore, the embodiments of the present disclosure realize the disk storage of graph data in the graph database by establishing a mapping relationship between each node in the B-tree and each key-value pair in the key-value storage system.

[0067] In the embodiments of the present disclosure, a node in the B-tree corresponds to a key-value pair in the key-value storage system. Since a B-tree is composed of multiple nodes, a B-tree corresponds to multiple key-value pairs in the key-value storage system. Specifically, the key Key in each pair of key-value pairs is used to store the index key of the corresponding node in the B-tree, where the index key belongs to the index information of the node. The value Value in each pair of key-value pairs is used to store the graph data or index information stored on the corresponding node in the B-tree.

[0068] In practical applications, during the process of disk-storing the first target graph data, first determine whether there are key-value pairs corresponding to each node in each B-tree corresponding to the first target graph data in the key-value storage system. For the nodes whose key-value pairs exist in the key-value storage system, the graph data stored in the corresponding B-tree by the nodes can be used for updating. For the nodes whose key-value pairs do not exist in the key-value storage system, the index information of the nodes can be persistently stored using the keys of the key-value pairs, and the graph data or index information stored on the nodes can be persistently stored using the values of the same key-value pairs.

[0069] In an alternative embodiment, for the leaf nodes of the B-tree, the Key in the key-value pair in the key-value storage system that has a mapping relationship with the leaf node is used to store the index key of the leaf node. Based on this index key, the leaf node can be indexed to read the graph data stored in the leaf node. The Value in this key-value pair is used to store the graph data stored in the leaf node.

[0070] In another alternative embodiment, for an internal node in a B-tree, the Key in the key-value pair in the key-value storage system that has a mapping relationship with the internal node is used to store the index key of the internal node. Based on this index key, the internal node can be indexed, so as to read the index information stored in the internal node. The Value in the key-value pair is used to store the index information stored in the internal node.

[0071] In yet another alternative embodiment, for the root node in a B-tree, the Key in the key-value pair in the key-value storage system that has a mapping relationship with the root node is used to store the index key of the root node. Based on this index key, the root node can be indexed, so as to read the index information stored in the root node. The Value in the key-value pair is used to store the index information stored in the root node.

[0072] In the data processing method of the graph database provided by the embodiments of the present disclosure, by establishing a mapping relationship between each node in the B-tree and the key-value pair in the key-value storage system stored on the disk, and using the key-value pair to store the index key of the node and the graph data stored in the node, the disk storage of the graph database is realized.

[0073] In addition, in the data processing method of the graph database provided by the embodiments of the present disclosure, after receiving a write request for first target graph data, determine the B-tree corresponding to each node in the first target graph data, and store the graph data of at least one edge type corresponding to each node in the leaf node of the corresponding B-tree, store the index information of the leaf node in the internal node corresponding to the leaf node in the B-tree, and store the index information of the B-tree in the root node of the B-tree. The index information of the B-tree is determined based on the node identifier of the node corresponding to the B-tree and at least one edge type corresponding to the B-tree. Furthermore, in the key-value storage system, determine the key-value pair corresponding to each node in the B-tree, and persistently store the index key in the index information of each node in the key of the corresponding key-value pair, and persistently store the graph data or index information stored on each node in the value of the corresponding key-value pair; the key-value storage system is used to persistently store graph data. The embodiments of the present disclosure organize the graph data into the storage structure of a B-tree. Since the storage structure of the B-tree supports efficient and low-latency data reading and writing, the graph database implemented based on the B-tree can also support efficient and low-latency data reading and writing. Especially for nodes with a large number of edges, the data reading and writing efficiency can be greatly improved.

[0074] Based on the above embodiments, after receiving a write request for graph data, a log message can be generated for the write request. The log message is used to record the operation information of the write request, and the log message is persistently stored in the key-value storage system. For example, the operation information of writing a certain graph data to a certain leaf node can be recorded in the log message.

[0075] In the embodiments of the present disclosure, after the log information is generated, the log information can be stored in a key-value storage system to implement disk storage of the log information.

[0076] In an optional implementation, in the key-value storage system, each B-tree has its corresponding log stream. Compared with each B-tree corresponding to the same log stream, the embodiments of the present disclosure can improve the efficiency of data reading and writing. Specifically, in the embodiments of the present disclosure, each generated log information is stored in the form of a key-value pair in the key-value storage system. Specifically, the generated log information is stored in the log stream of the corresponding B-tree.

[0077] Among them, the key Key in the key-value pair is used to store the index key of the log information. Among them, the index key of the log information is composed of the identifier of the log stream to which the log information belongs and the sequence identifier of the log information in the log stream. The value Value in the key-value pair is used to store the log information.

[0078] In practical applications, each B-tree has information such as the most recent replay time and the number of unreplayed logs. Among them, the most recent replay time is used to record the time when the corresponding B-tree last replayed the log information to the key-value storage system; the number of unreplayed logs is used to record the number of log information that has not been replayed to the key-value storage system for the corresponding B-tree. Usually, the log information replayed to the key-value storage system will be deleted. Therefore, the number of unreplayed logs refers to the number of log information of the corresponding B-tree currently in the key-value storage system. In addition, the system idle state is also an optional setting condition for determining log information replay.

[0079] In an optional implementation, when it is detected that the interval time between the most recent replay time of any B-tree and the current time point meets a preset interval value, or when it is detected that the number of unreplayed logs corresponding to any B-tree meets a preset number value, or when it is detected that the system idle state meets a preset idle condition, the log information of the B-tree can be obtained from the key-value storage system. Then, based on the operation information recorded in the log information, the historical write request is replayed to the B-tree to complete the data update of the B-tree. After completing the data update of the B-tree, the corresponding graph data in the key-value storage system (i.e., disk storage) is further updated based on the B-tree.

[0080] In an alternative implementation, after receiving a write request for first target graph data, log information corresponding to the write request is first generated to record the operation information of the write request. Then, the log information corresponding to the write request is stored in a key-value storage system. When it is determined that the B-tree corresponding to the log information meets a preset first playback condition and / or the current system idle state meets a preset second playback condition, the log information is retrieved from the key-value storage system. The first playback condition may include that the interval between the most recent playback time of the B-tree corresponding to the log information and the current time is greater than a preset interval threshold, or the number of unplayed logs is not less than a preset quantity threshold, etc. The second playback condition may include that the current system idle threads are greater than a preset ratio, for example, 2 / 3 of the threads in the current system are in an idle state, etc.

[0081] Furthermore, based on the operation information of the write request recorded in the log information, the B-tree corresponding to each node in the first target graph data corresponding to the operation information is determined, and the graph data corresponding to each node is stored in the leaf node of the corresponding B-tree. The index information of the leaf node is stored in the internal node corresponding to the leaf node in the corresponding B-tree to complete the data update of the B-tree. After completing the data update of the B-tree, the corresponding graph data in the key-value storage system (i.e., disk storage) is further updated based on the B-tree to complete the synchronous update of the graph data in the disk.

[0082] Based on the above embodiments, the graph data processing method of the graph database in the embodiments of the present disclosure can also support a high-performance multi-level least recently used (LRU) cache, and can support multiple modes of synchronizing data with disk storage. For example, data can be synchronized to disk storage each time a write is performed, or memory data can be synchronized to disk storage periodically or when the access traffic is low.

[0083] For this reason, the embodiments of the present disclosure provide a single-machine memory engine, as Figure 3 shown, which is a schematic architecture diagram of a single-machine memory engine provided by the embodiments of the present disclosure. Among them, the single-machine memory engine 301 is connected to the key-value storage system 302 for disk storage to implement the cache function of the graph database. Specifically, each device docked with the key-value storage system can be configured with the above single-machine memory engine to cache some or all of the graph data (i.e., some or all of the B-trees) from the key-value storage system.

[0084] In an alternative implementation, the entire set of graph data stored in the key-value storage system is divided into different data shards, and the multiple data shards corresponding to the entire set of graph data are respectively stored in the caches of different devices, as Figure 3 shown in the single-machine memory engine. Among them, the division method of the multiple data shards is determined based on a preset partitioning strategy. Specifically, the preset partitioning strategy is determined based on the application scenario characteristics of the entire set of graph data.

[0085] Assume that the entire set of graph data stored in the key-value storage system belongs to the knowledge graph scenario. Then the preset partitioning strategy can be a hash partitioning strategy based on edge types. Specifically, partition according to the edge type hash, and divide the graph data of the same edge type into the same data shard, so as to store it in the cache of the same device, thereby reducing the number of queries per fan-out in the graph data query and reducing the query response latency. Assume that the entire set of graph data stored in the key-value storage system belongs to the social scenario. Then the preset partitioning strategy can be an offline social hash algorithm, etc.

[0086] In addition, the above Figure 3 In the single-machine storage engine shown, there is a preset dirty list 303, which can also be called a dirtylist. The dirty list is used to maintain the index information of the B-tree with unplayed log information. That is to say, the dirty list records the index information of the B-tree that is out of sync between the memory and the disk data, such as index information A, B, and N, etc., so as to be used to update the data in the key-value storage system when the preset conditions are met.

[0087] In an alternative implementation, after receiving a write request for the first target graph data, generate log information corresponding to the write request, such as Figure 3 the "log stream of a specific B-tree" shown, and store the log information in the key-value storage system. In addition, since the data of the B-tree corresponding to the log information is inconsistent between the memory and the disk, therefore, the index information of the root node of the B-tree corresponding to the first target graph data can also be recorded in the preset dirty list in the cache, so as to be used to update the data in the key-value storage system later.

[0088] In practical applications, after obtaining the index information of the B-tree corresponding to the first target graph data from the preset dirty list, it is also possible to determine whether the B-tree meets the preset first replay condition and / or whether the current system idle state meets the preset second replay condition. If it is determined that the B-tree meets the preset first replay condition or the current system idle state meets the preset second replay condition, then based on the index information of the B-tree, obtain the log information of the B-tree from the key-value storage system, and use it to update the graph data stored in the key-value storage system.

[0089] In practical applications, when it is determined that the preset conditions are met currently, for example, when the data update cycle is reached or the current access traffic is low, the index information of the B-tree can be obtained from the preset dirty list in the cache, and the log information of the B-tree can be obtained from the key-value storage system based on the index information, and then the graph data in the key-value storage system can be updated based on the log information, that is, the data is written to disk. In addition, after completing the current round of data update, the corresponding log information can also be deleted.

[0090] In addition, embodiments of the present disclosure can implement the query function and storage function of the graph database using separate independent components respectively, that is, decouple the query function and storage function in the graph database, so that when there is a need to expand the graph database (such as a large change in access traffic), only the query function or the storage function can be separately expanded, avoiding waste of resources.

[0091] In an optional implementation, the graph database in embodiments of the present disclosure may include a graph storage engine layer and a graph query engine layer. The graph query engine layer is used to receive a graph data reading request from a user to implement the graph data query function.

[0092] In embodiments of the present disclosure, after receiving a reading request for second target graph data through the graph query engine layer, the reading request is passed to the graph storage engine layer, and then the graph storage engine layer is used to determine a B-tree corresponding to the reading request based on the node identifiers of each node in the second target graph data and the edge type, and execute the reading request based on the B-tree to obtain a reading result, and return the reading result to the graph query engine layer to complete the processing of the graph data reading request.

[0093] Based on the above method embodiments, the present disclosure also provides a graph database data processing device. Refer to Figure 4 , which is a schematic structural diagram of a graph database data processing device provided by embodiments of the present disclosure. The device includes:

[0094] A first receiving module 401, configured to receive a write request for first target graph data; the first target graph data includes nodes and edges, and the edge type of the edge is used to characterize the relationship between the nodes associated with the edge;

[0095] A first storage module 402, configured to determine a balanced B-tree corresponding to each node in the first target graph data, store graph data of at least one edge type corresponding to each node in a leaf node of the corresponding B-tree, and store index information of the leaf node in an internal node corresponding to the leaf node in the B-tree; the index information of the B-tree is stored in the root node of the B-tree, and the index information of the B-tree is determined based on the node identifier of the node corresponding to the B-tree and the at least one edge type corresponding to the B-tree;

[0096] A second storage module 403, configured to determine a key-value pair corresponding to each node in the B-tree in a key-value storage system, persistently store an index key in the index information of each node in the key of the corresponding key-value pair, and persistently store graph data or index information stored on each node in the value of the corresponding key-value pair; the key-value storage system is used to persistently store graph data.

[0097] In an optional implementation, the device further includes:

[0098] A first generation module for generating log information corresponding to the write request; wherein the log information is used to record operation information of the write request.

[0099] A third storage module for storing the log information corresponding to the write request into the key - value storage system.

[0100] A first acquisition module for acquiring the log information from the key - value storage system when it is determined that the B - tree corresponding to the log information meets a preset first replay condition and / or the current system idle state meets a preset second replay condition.

[0101] Correspondingly, the first storage module is specifically configured to:

[0102] Based on the operation information of the write request recorded in the log information, determine the B - tree corresponding to each node in the first target graph data corresponding to the operation information.

[0103] In an optional implementation manner, the third storage module is specifically configured to:

[0104] Store the log information corresponding to the write request in the log stream of the B - tree corresponding to the first target graph data in the key - value storage system in the form of key - value pairs; wherein the key - value storage system stores log streams respectively having a mapping relationship with each B - tree.

[0105] In an optional implementation manner, the apparatus further includes:

[0106] A recording module for recording the index information of the B - tree corresponding to the first target graph data in a preset dirty linked list in the cache; the preset dirty linked list is used to maintain the index information of the B - trees with un - replayed log information.

[0107] A first determination module for, when obtaining the index information of the B - tree corresponding to the first target graph data from the preset dirty linked list, performing the step of determining that the B - tree corresponding to the log information meets the preset first replay condition and / or the current system idle state meets the preset second replay condition.

[0108] In an optional implementation manner, multiple data shards corresponding to the entire graph data stored in the key - value storage system are respectively stored in caches of different devices; the multiple data shards are obtained by partitioning the entire graph data based on a preset partitioning strategy, and the preset partitioning strategy is determined based on the application - scenario characteristics of the entire graph data.

[0109] In an alternative embodiment, the storage capacity of the leaf nodes of the B-tree is preconfigured based on the read-write characteristic information of the graph data stored on the leaf nodes, and the read-write characteristic information is used to characterize the requirements of the graph data for data read-write amplification.

[0110] In an alternative embodiment, the apparatus further includes:

[0111] A second receiving module, configured to receive a read request for second target graph data through the graph query engine layer and pass the read request to the graph storage engine layer;

[0112] A second determining module, configured to use the graph storage engine layer to determine the B-tree corresponding to the read request based on the node identifiers and edge types of the nodes in the second target graph data, and execute the read request based on the B-tree to obtain a read result, and return the read result to the graph query engine layer.

[0113] In the graph database data processing apparatus provided by the embodiments of the present disclosure, after receiving a write request for first target graph data, first determine the balanced B-tree corresponding to each node in the first target graph data, and store the graph data of at least one edge type corresponding to each node in the leaf nodes of the corresponding B-tree, and store the index information of the leaf nodes in the internal nodes corresponding to the leaf nodes in the B-tree; the index information of the B-tree is stored in the root node of the B-tree, and the index information of the B-tree is determined based on the node identifier of the node corresponding to the B-tree and the at least one edge type corresponding to the B-tree. Furthermore, in the key-value storage system, determine the key-value pair corresponding to each node in the B-tree, and persistently store the index key in the index information of each node in the key of the corresponding key-value pair, and persistently store the graph data or index information stored on each node in the value of the corresponding key-value pair; the key-value storage system is used to persistently store graph data. The embodiments of the present disclosure organize the graph data into the storage structure of a B-tree. Since the storage structure of the B-tree supports efficient and low-latency data read and write, therefore, the graph database implemented based on the B-tree can also support efficient and low-latency data read and write. Especially for nodes with a large number of edges, the data read and write efficiency can be greatly improved.

[0114] In addition to the above methods and apparatuses, the embodiments of the present disclosure also provide a computer-readable storage medium, in which instructions are stored. When the instructions run on a terminal device, the terminal device implements the graph database data processing method described in the embodiments of the present disclosure.

[0115] An embodiment of the present disclosure also provides a computer program product, which includes computer programs / instructions. When the computer programs / instructions are executed by a processor, the data processing method of the graph database described in the embodiments of the present disclosure is implemented.

[0116] In addition, an embodiment of the present disclosure also provides a data processing device for a graph database. Refer to Figure 5 as shown, it may include:

[0117] A processor 501, a memory 502, an input device 503, and an output device 504. The number of processors 501 in the data processing device for the graph database may be one or more. Figure 5 Taking one processor as an example. In some embodiments of the present disclosure, the processor 501, the memory 502, the input device 503, and the output device 504 may be connected by a bus or other means. Among them, Figure 5 taking the connection by bus as an example.

[0118] The memory 502 can be used to store software programs and modules. The processor 501 runs the software programs and modules stored in the memory 502 to execute various functional applications and data processing of the data processing device for the graph database. The memory 502 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc. In addition, the memory 502 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The input device 503 can be used to receive input digital or character information and generate signal inputs related to user settings and function controls of the data processing device for the graph database.

[0119] Specifically in this embodiment, the processor 501 will load the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 will run the application programs stored in the memory 502 to implement various functions of the above-mentioned data processing device for the graph database.

[0120] It should be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0121] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather will conform to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method for a graph database, characterized in that, The method includes: Receiving a write request for first target graph data; the first target graph data includes nodes and edges, and the edge type of the edge is used to characterize the relationship type between the nodes associated with the edge; Determining a balanced B-tree corresponding to each node in the first target graph data, storing graph data of at least one edge type corresponding to each node in a leaf node of the corresponding B-tree, and storing index information of the leaf node in an internal node corresponding to the leaf node in the B-tree; the index information of the B-tree is stored in the root node of the B-tree, and the index information of the B-tree is determined based on the node identifier of the node corresponding to the B-tree and the at least one edge type corresponding to the B-tree; In a key-value storage system, determining a key-value pair corresponding to each node in the B-tree, and persistently storing the index key in the index information of each node in the key of the corresponding key-value pair, and persistently storing the graph data or index information stored on each node in the value of the corresponding key-value pair; the key-value storage system is used to persistently store graph data.

2. The method according to claim 1, wherein Before determining the B-tree corresponding to each node in the first target graph data, it further includes: Generating log information corresponding to the write request; wherein, the log information is used to record operation information of the write request; Storing the log information corresponding to the write request into the key-value storage system; When it is determined that the B-tree corresponding to the log information meets a preset first replay condition and / or the current system idle state meets a preset second replay condition, obtaining the log information from the key-value storage system; Correspondingly, determining the B-tree corresponding to each node in the first target graph data includes: Based on the operation information of the write request recorded in the log information, determining the B-tree corresponding to each node in the first target graph data corresponding to the operation information.

3. The method according to claim 2, wherein The storing the log information corresponding to the write request into the key-value storage system includes: Storing the log information corresponding to the write request into the log stream of the B-tree corresponding to the first target graph data in the key-value storage system in the form of a key-value pair; wherein, the key-value storage system stores log streams respectively having a mapping relationship with each B-tree.

4. The method according to claim 2, wherein After generating the log information corresponding to the write request, it further includes: Recording the index information of the B-tree corresponding to the first target graph data in a preset dirty linked list in the cache; the preset dirty linked list is used to maintain the index information of the B-tree with un-replayed log information; When obtaining the index information of the B-tree corresponding to the first target graph data from the preset dirty linked list, performing the step of determining that the B-tree corresponding to the log information meets a preset first replay condition and / or the current system idle state meets a preset second replay condition.

5. The method according to claim 1, wherein Multiple data shards corresponding to the entire graph data set stored in the key-value storage system are respectively stored in caches of different devices; the multiple data shards are obtained by partitioning the entire graph data set based on a preset partitioning strategy, and the preset partitioning strategy is determined based on the application scenario characteristics of the entire graph data set.

6. The method according to claim 1, wherein The storage capacity of the leaf nodes of the B-tree is preconfigured based on the read-write feature information of the graph data stored on the leaf nodes, and the read-write feature information is used to characterize the requirements of the graph data for data read-write amplification.

7. The method according to claim 1, wherein The method further includes: Receiving a read request for second target graph data through the graph query engine layer and passing the read request to the graph storage engine layer; Using the graph storage engine layer, determining the B-tree corresponding to the read request based on the node identifiers of the nodes in the second target graph data and the edge types, and executing the read request based on the B-tree to obtain a read result, and returning the read result to the graph query engine layer.

8. A data processing device for a graph database, characterized in that The apparatus includes: A first receiving module, configured to receive a write request for first target graph data; the first target graph data includes nodes and edges, and the edge type of the edge is used to characterize the relationship between the nodes associated with the edge; A first storage module, configured to determine the balanced B-tree corresponding to each node in the first target graph data, and store the graph data of at least one edge type corresponding to each node in the leaf nodes of the corresponding B-tree, and store the index information of the leaf nodes in the internal nodes corresponding to the leaf nodes in the B-tree; the index information of the B-tree is stored in the root node of the B-tree, and the index information of the B-tree is determined based on the node identifier of the node corresponding to the B-tree and the at least one edge type; A second storage module, configured to determine the key-value pairs corresponding to each node in the B-tree in the key-value storage system, and persistently store the index keys in the index information of each node in the keys of the corresponding key-value pairs, and persistently store the graph data or index information stored on each node in the values of the corresponding key-value pairs; the key-value storage system is used to persistently store graph data.

9. A computer-readable storage medium, characterized in that, Instructions are stored in the computer-readable storage medium, and when the instructions run on the terminal device, the terminal device implements the method according to any one of claims 1-7.

10. A data processing device for a graph database, characterized in that, Including: A memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the method according to any one of claims 1-7 is implemented.

Images