Data processing method, device and storage medium of graph database

Abstract

The disclosure provides a data processing method, device and storage medium of a graph database, relating to the fields of knowledge graph technology and artificial intelligence technology. The specific implementation scheme is: dividing a set of entity nodes to be processed to obtain a plurality of small-batch entity node subsets; after completing entity node storage location analysis on each entity node subset, directly querying the entity nodes through the corresponding storage nodes to obtain the next-hop entity nodes of the corresponding entity node subsets; and based on the next-hop entity nodes of each entity node in the corresponding entity node subset, determining a set of next-hop entity nodes, and processing according to the set of next-hop entity nodes of the entity node subset to obtain a set of next-hop entity nodes corresponding to the entity node set. In this way, concurrent processing of a plurality of small-batch entity node subsets can be realized, and network resources can be reasonably utilized.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical field of knowledge graph and other artificial intelligence technical fields, and especially to a data processing method, device and storage medium of a graph database. BACKGROUND

[0002] A knowledge graph is a kind of associated network knowledge expression based on a graph model. The knowledge graph abstracts entities as vertices and abstracts the relationships between entities as edges, models and describes knowledge in a structured form, and visualizes the knowledge. The knowledge graph is the most closely associated and most widely used application direction of a graph database. The knowledge graph takes a graph database as a storage engine, intelligently processes massive information, forms a large-scale knowledge base, and further supports business applications. SUMMARY

[0003] The present disclosure provides a data processing method, device and storage medium for a graph database.

[0004] According to an aspect of the present disclosure, a data processing method of a graph database is provided, which comprises: acquiring a set of entity nodes to be processed in the graph database; dividing the set of entity nodes to obtain a plurality of sub-sets of entity nodes; for each sub-set of entity nodes, determining the storage nodes to which each entity node in the sub-set of entity nodes belongs, wherein the storage nodes to which the entity node belongs include the entity node and the corresponding next-hop entity node; determining a set of next-hop entity nodes corresponding to each sub-set of entity nodes according to the next-hop entity nodes corresponding to each entity node in the sub-set of entity nodes; and determining a set of next-hop entity nodes corresponding to the set of entity nodes according to the set of next-hop entity nodes corresponding to each sub-set of entity nodes.

[0005] According to another aspect of the present disclosure, a data processing device of a graph database is provided, which comprises: a first acquisition module configured to acquire a set of entity nodes to be processed in the graph database; a division module configured to divide the set of entity nodes to obtain a plurality of sub-sets of entity nodes; a first determination module configured to, for each sub-set of entity nodes, determine the storage nodes to which each entity node in the sub-set of entity nodes belongs, wherein the storage nodes to which the entity node belongs include the entity node and the corresponding next-hop entity node; a second determination module configured to determine a set of next-hop entity nodes corresponding to each sub-set of entity nodes according to the next-hop entity nodes corresponding to each entity node in the sub-set of entity nodes; and a third determination module configured to determine a set of next-hop entity nodes corresponding to the set of entity nodes according to the set of next-hop entity nodes corresponding to each sub-set of entity nodes.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data processing method of the graph database of the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the data processing method of the graph database disclosed in an embodiment of the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the graph database data processing method of the present disclosure when executed by a processor.

[0009] One embodiment of the above application has the following advantages or beneficial effects:

[0010] When processing a set of entity nodes in a graph database, the set of entity nodes to be processed is divided into multiple small batches of entity node subsets. After the storage location analysis of each entity node subset is completed, the entity node is directly queried through the corresponding storage node to obtain the next-hop entity node of the corresponding entity node. Based on the next-hop entity nodes corresponding to each entity node in the corresponding entity node subset, a next-hop entity node set is determined. Processing is then performed based on the next-hop entity node set of the entity node subset to obtain the next-hop entity node set corresponding to the entity node set. This allows for concurrent processing of multiple small batches of entity node subsets, effectively utilizing network resources.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 This is a flowchart of a data processing method for a graph database provided according to an embodiment of the present disclosure;

[0014] Figure 2 This is a schematic diagram of a detailed process of determining a next-hop entity node set corresponding to an entity node set based on a next-hop entity node set corresponding to each entity node subset, provided according to another embodiment of the present disclosure;

[0015] Figure 3 This is a schematic diagram of a detailed process of determining a next-hop entity node set corresponding to an entity node subset according to a storage node to which each entity node in the entity node subset belongs, provided according to another embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of a detailed process of deduplicating the next-hop entity node set corresponding to each entity node subset to obtain deduplication results corresponding to each entity node subset is provided according to another embodiment of the present disclosure;

[0017] Figure 5 is a flowchart of a data processing method for a graph database provided according to another embodiment of the present disclosure;

[0018] Figure 6 This is a schematic diagram of the structure of a data processing device for a graph database provided according to one embodiment of the present disclosure;

[0019] Figure 7 is a structural diagram of a data processing device for a graph database provided according to another embodiment of the present disclosure;

[0020] Figure 8 is a structural diagram of a data processing device for a graph database provided according to another embodiment of the present disclosure;

[0021] Figure 9 It is a block diagram of an electronic device used to implement the data processing method of the graph database of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] Graph walking is a fundamental problem in the field of graph retrieval. Its basic meaning is to traverse multiple layers in an abstract logical graph model (i.e., a topological graph). Currently, graph walking in distributed graph databases typically adopts a single-step walking strategy. That is, before each walk, the computing node uniformly analyzes the storage location of the input node's related relationship information, and then sends it to the corresponding storage node for query. The query results are collected and deduplicated, and the deduplicated results serve as the input for the next walk. However, in this method, only one process is executed at any given moment, resulting in insufficient resource utilization and low concurrency.

[0024] To this end, the present application proposes a data processing method for a graph database. When processing a set of entity nodes in a graph database, the method divides the set of entity nodes to be processed to obtain multiple small batches of entity node subsets. After completing the analysis of the storage location of each entity node subset, the entity node is directly queried through the corresponding storage node to obtain the next-hop entity node of the corresponding entity node. Based on the next-hop entity nodes corresponding to each entity node in the corresponding entity node subset, the next-hop entity node set is determined, and processing is performed according to the next-hop entity node set of the entity node subset to obtain the next-hop entity node set corresponding to the entity node set. In this way, multiple small batches of entity node subsets can be processed concurrently, and network resources can be rationally utilized.

[0025] The following describes the data processing method, device and storage medium of the graph database of the embodiment of the present disclosure with reference to the accompanying drawings.

[0026] Figure 1 This is a flow chart of a data processing method for a graph database provided according to an embodiment of the present disclosure. The graph database in this embodiment is a distributed graph database based on a storage and computing separation architecture design, wherein the graph database includes computing nodes and multiple storage nodes, each storage node stores a different part of the knowledge graph, and the different parts of the knowledge graph stored by each storage node are spliced ​​together to obtain a complete knowledge graph, wherein the knowledge graph includes multiple entity nodes and connecting edges between multiple entity nodes. It should be noted that the method of this embodiment is applied to the computing nodes in the graph database.

[0027] like Figure 1 As shown, the data processing method of the graph database may include:

[0028] Step 101: Obtain a set of entity nodes to be processed in a graph database.

[0029] Step 102: Divide the entity node set to obtain multiple entity node subsets.

[0030] In some embodiments, after obtaining a set of entity nodes to be processed in a graph database, the set of entity nodes may be divided into groups with a preset number of entity nodes to obtain multiple entity node subsets.

[0031] Among them, the preset number is the number of nodes pre-set in the computing node. For example, the preset number can be 100,000 or 300,000. In actual applications, the preset number can be set according to actual business needs. This embodiment does not specifically limit the preset number.

[0032] In some embodiments, in a graph walk scenario, for each step of the walk, a set of entity nodes to be processed for the current step of the walk may be obtained.

[0033] The set of entity nodes to be processed in the current walk is the entity node result obtained in the previous walk corresponding to the current walk.

[0034] Step 103 : For each entity node subset, determine the storage node to which each entity node in the entity node subset belongs.

[0035] The storage node to which the corresponding physical node belongs includes the corresponding physical node and the corresponding next-hop physical node.

[0036] Step 104 : Determine a next-hop entity node set corresponding to the entity node subset according to the next-hop entity node corresponding to each entity node in the entity node subset.

[0037] In some embodiments, a possible implementation method for determining the next hop entity node set corresponding to the entity node subset based on the next hop entity node corresponding to each entity node in the entity node subset is: based on the storage node to which each entity node in the entity node subset belongs, the entity nodes belonging to the same storage node are divided into a group to obtain multiple groups of entity nodes; for each group of entity nodes, the deduplicated next hop entity node result returned by the storage node corresponding to the current entity node group for the current entity node group is received; based on the next hop entity node result of each group of entity nodes, the next hop entity node set corresponding to the entity node subset is determined.

[0038] In some exemplary embodiments, when a computing node analyzes the storage locations of entity nodes using a single-threaded approach, the computing node may sequentially traverse multiple entity node subsets in a traversal order. For the current entity node subset currently traversed, the computing node may analyze the storage location of each entity node in the current entity node subset to determine the storage node to which each entity node in the current entity node subset belongs. Correspondingly, after determining the storage node to which each entity node in the current entity node subset belongs, the computing node groups entity nodes belonging to the same storage node into a group based on the storage node to which each entity node in the current entity node subset belongs, thereby obtaining multiple groups of entity nodes. For each group of entity nodes, the computing node sends a query request to the storage node corresponding to the current entity node group. Correspondingly, the storage node queries the part of the knowledge graph it saves according to the query request to obtain the entity node obtained after each entity node in the current entity node group performs one-step walk, and uses the obtained entity node as the next-hop entity node of the corresponding entity node, and obtains the next-hop entity node result of the current entity node group based on the next-hop entity node corresponding to each entity node in the current entity node group.

[0039] It should be understood that, for any physical node, when the storage node to which the physical node belongs determines the next-hop physical node corresponding to the physical node, it can be determined that the physical node has completed one step of roaming.

[0040] Correspondingly, the computing node receives the next-hop entity node result returned by the storage node for the current entity node group, and determines the next-hop entity node set corresponding to the current entity node subset according to the next-hop entity node result of each group of entity nodes.

[0041] In some embodiments, in order to reduce the amount of data that the storage node needs to send while reducing the number of nodes processed by the computing node, after the storage node obtains the next-hop entity node result of the current entity node group, the storage node can deduplicate the next-hop entity node result corresponding to the current entity node group.

[0042] There are many ways to deduplicate nodes, and the ways to deduplicate nodes will be described in subsequent embodiments.

[0043] In other embodiments, when a computing node uses multiple threads to process multiple entity node subsets, the thread corresponding to the corresponding entity node subset determines the storage node to which each entity node in the entity node subset belongs, and then queries the corresponding storage node. Correspondingly, the storage node queries the next-hop entity node corresponding to each entity node in the entity node subset based on the portion of the knowledge graph it stores. Based on the next-hop node corresponding to each entity node, the storage node obtains the set of next-hop entity nodes corresponding to the entity node subset.

[0044] In some embodiments, in order to reduce the amount of data that the storage node needs to send while reducing the number of nodes processed by the computing node, the storage node can also deduplicate the next-hop entity node set corresponding to the entity node subset sent by the node before sending the next-hop entity node set corresponding to the entity node subset to the computing node.

[0045] Step 105 : Determine the next hop entity node set corresponding to the entity node set according to the next hop entity node set corresponding to each entity node subset.

[0046] The data processing method for a graph database of an embodiment of the present disclosure divides the entity node set to be processed into multiple small batch entity node subsets when processing the entity node set in the graph database. After the entity node storage location analysis is completed for each entity node subset, the entity node is directly queried through the corresponding storage node to obtain the next-hop entity node of the corresponding entity node. Based on the next-hop entity nodes corresponding to each entity node in the corresponding entity node subset, the next-hop entity node set is determined. The next-hop entity node set of the entity node subset is processed to obtain the next-hop entity node set corresponding to the entity node set. In this way, multiple small batch entity node subsets can be processed concurrently, and network resources can be reasonably utilized.

[0047] Based on the above description, it can be seen that compared with the existing wandering method, in which the computing node needs to analyze all the entity nodes input in this step of wandering before interacting with the storage node, the computing node in this implementation divides the entity node set of this step of wandering to obtain multiple small batch entity node subsets. After completing the analysis of the entity node storage location of each entity node subset, the entity node is directly queried through the corresponding storage node to obtain the next hop entity node set of the corresponding entity node subset, and the next hop entity node set of the corresponding entity node subset is processed through the computing node. As a result, the computing node can interact with the corresponding storage node at the storage location of some of the analyzed entity nodes, and multiple small batch entity node subsets can be processed concurrently, making rational use of network resources and improving query efficiency.

[0048] In different application scenarios, determining the next-hop entity node set corresponding to the entity node set according to the next-hop entity node set corresponding to each entity node subset can be implemented in various ways, as exemplified below.

[0049] In some embodiments, as Figure 2 As shown, this may include:

[0050] Step 201 : Deduplication is performed on the next-hop entity node set corresponding to each entity node subset to obtain deduplication results corresponding to each entity node subset.

[0051] Step 202 : Merge the deduplication results corresponding to all entity node subsets to obtain a candidate next-hop entity node set corresponding to the entity node set.

[0052] Step 203: De-duplicate the candidate next-hop entity node set again to obtain the next-hop entity node set corresponding to the entity node set.

[0053] In this embodiment, after the deduplication results corresponding to all entity node subsets are merged, in order to avoid the possibility that the same entity node exists in the deduplication results corresponding to different entity node subsets and to avoid repeated wandering of the same entity node, this embodiment deduplicates the merged entity node set again, and uses the entity node set after deduplication as the next-hop entity node set corresponding to the entity node set, that is, uses the entity node set after deduplication as the node query result corresponding to the entity node set.

[0054] In other embodiments, when deduplicating the next-hop entity node set corresponding to the entity node subset based on the traversal order, the node deduplication result corresponding to the last traversed entity node subset can be obtained in combination with the traversal order, and the next-hop entity node set corresponding to the currently traversed entity node subset can be deduplicated based on the node deduplication result until all next-hop entity node sets are deduplicated. Correspondingly, all deduplicated next-hop entity node sets are merged, and the merged result is used as the next-hop entity node set corresponding to the entity node set.

[0055] In other embodiments, one implementation method for determining the next hop entity node set corresponding to the entity node set based on the next hop entity node set corresponding to each entity node subset is: merging the next hop entity node sets corresponding to all node subsets to obtain a merged next hop entity node set; and deduplicating the merged next hop entity node set to obtain the next hop entity node set corresponding to the entity node set.

[0056] In this embodiment, the next-hop entity node sets corresponding to multiple entity node subsets are directly merged and deduplication is performed on the merged entity nodes to obtain the next-hop entity node set corresponding to the entity node set. This achieves deduplication of the entity node set corresponding to the entity node set, avoids performing the next traversal on duplicate entity nodes, reduces network resource waste, and improves the efficiency of traversal results.

[0057] Based on the above embodiment, in order to reduce the amount of data that the storage node needs to send and reduce the number of nodes processed by the computing node, a possible implementation method of determining the next hop entity node set corresponding to the entity node subset according to the next hop entity node corresponding to each entity node in the entity node subset is as follows: Figure 3 As shown, this may include:

[0058] Step 301 : According to the storage node to which each entity node in the entity node subset belongs, the entity nodes belonging to the same storage node are divided into a group to obtain multiple groups of entity nodes.

[0059] Step 302 : For each group of entity nodes, receive the deduplicated next-hop entity node result returned by the storage node corresponding to the current entity node group.

[0060] Step 303 : Determine the next-hop entity node set corresponding to the entity node subset according to the next-hop entity node result of each group of entity nodes.

[0061] In one embodiment of the present disclosure, in order to reduce the memory consumption in the deduplication process, one possible implementation of the above-mentioned deduplication of the next-hop entity node set corresponding to each entity node sub-set to obtain the deduplication result corresponding to each entity node sub-set respectively, as shown in Figure 4 may include:

[0062] Step 401, for each entity node sub-set, performing node traversal on the next-hop entity node set corresponding to the entity node sub-set.

[0063] Step 402, for the target entity node traversed, determining the interval to which the node identifier of the target entity node belongs.

[0064] Wherein, the interval is obtained by segmenting the value range of the node identifier.

[0065] Wherein, the data type of the node identifier in the present embodiment can be integer data, for example, an 8-byte integer data can be used to represent the node identifier, wherein the value range of the node identifier is 1-2 ^64 .

[0066] In some embodiments, the node identifier of the target entity node can be compared with the interval range corresponding to the preset multiple intervals, and according to the comparison result, the interval to which the node identifier of the target node belongs is determined.

[0067] For example, the node identifier of the target entity node is 1000, if the interval range corresponding to interval 1 is 0-2 ^32 , the interval range corresponding to interval 2 is 2 ^32 -2 ^33 , the interval range corresponding to interval 3 is 2 ^33 -3*2 ^32 , the interval range corresponding to interval 4 is 3*2 ^32 -2 ^34 , the interval range corresponding to interval 5 is 2 ^34 -2 ^64 , at this time, by comparing the node identifier with the interval range corresponding to each interval, it can be determined that the node identifier 1000 is in the interval range 0-2 ^32 corresponding to interval 1, at this time, it can be determined that the node identifier of the target node belongs to interval 1.

[0068] Step 403, according to the deduplication mode corresponding to the interval, determining that the target entity node has appeared in the traversed nodes.

[0069] Wherein, in different application scenarios, the implementation of determining that the target entity node has appeared in the traversed nodes according to the deduplication mode corresponding to the interval is different, which is exemplarily explained as follows:

[0070] As an exemplary implementation, multiple bit sets corresponding to the interval are obtained; the bits of the node identifier are segmented to obtain multiple bit segments corresponding to the node identifier; for each bit segment, a target bit set corresponding to the bit segment is obtained from the multiple bit sets, wherein the bit length of the target bit set is greater than or equal to the bit length of the bit segment; bits corresponding to the value of the bit segment are obtained from the target bit set; when the values ​​of the bits corresponding to the multiple bit segments are all first values, it is determined that the target entity node has appeared in the traversed nodes, wherein the first value indicates that the value of the corresponding bit segment has appeared.

[0071] In this embodiment, in the process of deduplicating nodes of the next-hop entity node set corresponding to the entity node subset, multiple bit sets with a bit length smaller than the node identifier are combined to deduplicate the entity nodes in the next-hop entity node set, thereby reducing the memory resources occupied by the bit set during the deduplication process.

[0072] As another exemplary embodiment, a hash table corresponding to the interval is obtained; the upper N bits and lower M bits of the node identifier are obtained, where N is an integer greater than 1 and less than K, K represents the bit length of the node identifier, and M is the difference between K and N; a bit set corresponding to the values ​​of the upper N bits is obtained from the hash table; a target bit corresponding to the value of the lower M bits is obtained from the bit set; when the value on the target bit is a first value, it is determined that the target entity node has appeared in the traversed nodes, where the first value is used to indicate that the value of the lower M bits has appeared.

[0073] In this embodiment, in the process of deduplicating nodes of the next-hop entity node set corresponding to the entity node subset, the entity nodes in the next-hop entity node set are deduplicated by combining the hash table and the bit set, which can reduce the memory resources occupied by the hash table and the bit set, and reduce the memory resources occupied in the deduplication process.

[0074] As another exemplary implementation, a bit set corresponding to the interval is obtained; a target bit corresponding to the node identifier is obtained from the bit set; and when the value of the target bit is a first value, it is determined that the target entity node has appeared in the traversed nodes, wherein the first value indicates that the node identifier has appeared.

[0075] In this embodiment, during the deduplication process of entity nodes, the entity nodes are deduplicated in combination with a bit set smaller than the bit position of the node identifier of the entity node, which can reduce the memory consumption occupied by the bit set during the deduplication process.

[0076] As another exemplary implementation, a hash table corresponding to the interval can be obtained, and a hash calculation can be performed on the node identifier to obtain a hash value corresponding to the node identifier. At least one candidate value corresponding to the hash value can be obtained from the hash table. When there is a candidate value with a first value among at least one candidate value, it is determined that the target entity node has appeared in the traversed nodes, wherein the first value indicates that the node identifier has appeared.

[0077] In this embodiment, during the process of deduplicating entity nodes, the entity nodes are deduplicated in combination with the hash table corresponding to the interval to which the node identifier belongs, thereby reducing the time for searching the hash table and reducing memory consumption.

[0078] Step 404: Delete the target entity node in the next-hop entity node set.

[0079] In some embodiments, a possible implementation method of further deduplicating the candidate next-hop entity node set to obtain the next-hop entity node set obtained by this step of the walk is: performing node traversal on the candidate next-hop entity node set, and for the traversed next-hop entity node, determining the interval to which the node identifier of the next-hop entity node belongs; according to the deduplication method corresponding to the interval, determining that the next-hop entity node has appeared in the traversed nodes, and deleting the next-hop entity node in the candidate next-hop entity node set.

[0080] Among them, the process of determining whether the next-hop entity node has appeared in the traversed nodes according to the deduplication method corresponding to the interval is similar to the above-mentioned process of determining whether the target entity node has appeared in the traversed nodes according to the deduplication method corresponding to the interval, and this embodiment will not repeat it again.

[0081] In order to make the present disclosure clearly understood by those skilled in the art, Figure 5 The method of this embodiment is further described. It should be noted that this embodiment takes the example of computing nodes and storage nodes performing data processing in a single-threaded manner. It should be noted that: Figure 5 A represents the first batch of input data, B represents the second batch of input data, C represents the third batch of input data, and D represents the fourth batch of input data.

[0082] like Figure 5 As shown, this may include:

[0083] In this embodiment, after the computing node obtains the input data of the current walk input, the computing node may divide the input data into multiple batches of input data. Figure 5In this example, the input data is divided into four batches. Compute nodes analyze the batches sequentially. After analyzing the first batch, they send it to the corresponding storage nodes. The storage nodes collect the relationship data from the first batch and send it to the compute nodes.

[0084] It should be noted that the input data here is a set of entity nodes.

[0085] It should be noted that, in the process of the storage node collecting the relational data of the first batch of input data, the computing node analyzes the storage location of the second batch of input data. After analyzing the storage location corresponding to each input data in the second batch of input data, the computing node sends a query request to the storage node corresponding to the storage location to query the relational data corresponding to the input data.

[0086] It should be noted that, for the current batch of input data, when the computing node deduplicates the query results of the previous batch of input data, if the computing node receives the query results of the current batch of input data, since the query results corresponding to different batches of input data may have the same entity node, the computing node can complete the deduplication of the query results of the previous batch of input data, and then deduplicate the query results of the current batch of input data in combination with the deduplication query results of the previous batch of input data. It should be noted that in order to reduce the deduplication weight of the computing node while reducing the amount of data that needs to be transmitted on the network, the above-mentioned storage node can deduplicate the query results of the batch input data after obtaining the query results of the corresponding batch input data.

[0087] The deduplication processing method can be found in the relevant description of the above embodiment and will not be repeated here.

[0088] In this embodiment, the input data of this step is split to obtain multiple batches of input data, and the multiple batches of input data are processed in parallel, thereby improving the concurrency between different execution steps, rationally utilizing network resources, and improving the efficiency of obtaining query results.

[0089] Furthermore, this embodiment employs a two-stage deduplication approach: first, query results obtained from node roaming are deduplicated in the storage nodes, and then deduplicated again in the compute nodes. Deduplication in the storage nodes reduces the amount of data required to be transmitted over the network, and reduces the number of nodes that need to be deduplicated on the compute nodes, thereby alleviating the deduplication burden on the compute nodes.

[0090] In order to implement the above embodiments, the embodiments of the present disclosure also provide a data processing device for a graph database.

[0091] Figure 6 is a structural schematic diagram of a data processing apparatus of a graph database according to one embodiment of the present disclosure.

[0092] As Figure 6 shown, the data processing apparatus 600 of the graph database can include a first acquisition module 601, a division module 602, a first determination module 603, a second determination module 604, and a third determination module 605, wherein:

[0093] The first acquisition module 601 is configured to acquire a set of entity nodes to be processed in the graph database.

[0094] The division module 602 is configured to divide the set of entity nodes to obtain a plurality of sub-sets of entity nodes.

[0095] The first determination module 603 is configured to determine, for each sub-set of entity nodes, a storage node to which each entity node in the sub-set of entity nodes belongs, wherein the storage node to which the entity node belongs includes the entity node and a corresponding next-hop entity node.

[0096] The second determination module 604 is configured to determine, according to the corresponding next-hop entity node of each entity node in the sub-set of entity nodes, a set of next-hop entity nodes corresponding to the sub-set of entity nodes.

[0097] The third determination module 605 is configured to determine, according to the set of next-hop entity nodes corresponding to each sub-set of entity nodes, a set of next-hop entity nodes corresponding to the set of entity nodes.

[0098] It should be noted that the above description of the embodiments of the data processing method of the graph database also applies to this embodiment, and the present embodiment will not be repeated.

[0099] The data processing apparatus of the graph database according to the embodiments of the present disclosure divides the set of entity nodes to be processed in the graph database to obtain a plurality of small batches of sub-sets of entity nodes, and after completing the analysis of the storage location of each entity node in each sub-set of entity nodes, directly queries the entity nodes through the corresponding storage node to obtain the next-hop entity node of the corresponding sub-set of entity nodes, and determines the set of next-hop entity nodes based on the corresponding next-hop entity node of each entity node in the corresponding sub-set of entity nodes, and processes according to the set of next-hop entity nodes of the sub-set of entity nodes to obtain the set of next-hop entity nodes corresponding to the set of entity nodes. Therefore, a plurality of small batches of sub-sets of entity nodes can be processed concurrently, and network resources can be utilized reasonably.

[0100] In one embodiment of the present disclosure, as Figure 7As shown, the data processing device of the graph database may include: a first acquisition module 701, a division module 702, a first determination module 703, a second determination module 704, and a third determination module 705, wherein the above-mentioned second determination module 704 includes: a first deduplication unit 7041, a first merging unit 7042, and a second deduplication unit 7043; the first deduplication unit 7041 may include: a node traversal sub-unit 70411, a first determination sub-unit 70412, a second determination sub-unit 70413, and a deletion sub-unit 70414.

[0101] For detailed descriptions of the first acquisition module 701, the division module 702, the first determination module 703 and the third determination module 705, please refer to Figure 6 The description of the first acquisition module 601, the division module 602, the first determination module 603 and the third determination module 605 in the illustrated embodiment will not be repeated here.

[0102] In one embodiment of the present disclosure, Figure 7 As shown, the second determining module 704 includes:

[0103] The first deduplication unit 7041 is configured to perform deduplication on the next-hop entity node set corresponding to each entity node subset, so as to obtain a deduplication result corresponding to each entity node subset;

[0104] A first merging unit 7042 is configured to merge the deduplication results corresponding to all entity node subsets to obtain a candidate next-hop entity node set of the entity node set;

[0105] The second deduplication unit 7043 is configured to perform deduplication again on the candidate next-hop entity node set to obtain a next-hop entity node set corresponding to the entity node set.

[0106] In one embodiment of the present disclosure, the first determination module 703 is specifically configured to: divide the entity nodes belonging to the same storage node into a group according to the storage node to which each entity node in the entity node subset belongs, so as to obtain multiple groups of entity nodes; for each group of entity nodes, receive the deduplicated next-hop entity node results returned by the storage node corresponding to the current entity node group for the current entity node group; and determine the next-hop entity node set corresponding to the entity node subset according to the next-hop entity node results of each group of entity nodes.

[0107] In one embodiment of the present disclosure, Figure 7 As shown, the first deduplication unit 7041 includes:

[0108] The node traversal subunit 70411 is used to perform node traversal on the next-hop entity node set corresponding to each entity node subset;

[0109] The first determining subunit 70412 is configured to determine, for a traversed target entity node, the interval to which the node identifier of the target entity node belongs, wherein the interval is obtained by segmenting the value range of the node identifier;

[0110] The second determining subunit 70413 is configured to determine, based on a deduplication method corresponding to the interval, whether the target entity node has appeared in the traversed nodes;

[0111] The deletion subunit 70414 is configured to delete the target entity node from the next-hop entity node set.

[0112] In one embodiment of the present disclosure, the second determination subunit 70413 is specifically used to: obtain multiple bit sets corresponding to the interval; segment the bits of the node identifier to obtain multiple bit segments corresponding to the node identifier; for each bit segment, obtain a target bit set corresponding to the bit segment from multiple bit sets, wherein the bit length of the target bit set is greater than or equal to the bit length of the bit segment; obtain bits corresponding to the value of the bit segment from the target bit set; in a case where the values ​​of the bits corresponding to the multiple bit segments are all first values, determine that the target entity node has appeared in the traversed nodes, wherein the first value indicates that the value of the corresponding bit segment has appeared.

[0113] In one embodiment of the present disclosure, the second determination subunit 70413 is specifically used to: obtain a hash table corresponding to the interval; obtain the upper N bits and the lower M bits of the node identifier, wherein N is an integer greater than 1 and less than K, K represents the bit length of the node identifier, and M is the difference between K and N; obtain a bit set corresponding to the values ​​of the upper N bits from the hash table; obtain a target bit corresponding to the value of the lower M bits from the bit set; when the value on the target bit is the first value, determine that the target entity node has appeared in the traversed nodes, wherein the first value is used to indicate that the value of the lower M bits has appeared.

[0114] In one embodiment of the present disclosure, the second determination subunit 70413 is specifically used to: obtain a bit set corresponding to the interval; obtain a target bit corresponding to the node identifier from the bit set; and when the value on the target bit is a first value, determine that the target entity node has appeared in the traversed nodes, wherein the first value indicates that the node identifier has appeared.

[0115] In one embodiment of the present disclosure, Figure 8As shown, the data processing apparatus of the graph database can include: a first acquisition module 801, a division module 802, a first determination module 803, a second determination module 804, and a third determination module 805, wherein the second determination module 804 can include: a second merging unit 8041 and a third deduplication unit 8042.

[0116] In one embodiment of the present disclosure, as Figure 8 shown, the second determination module 804 includes:

[0117] The second merging unit 8041 is configured to perform merging processing on the next-hop entity node set corresponding to each node sub-set to obtain a merged next-hop entity node set.

[0118] The third deduplication unit 8042 is configured to perform deduplication processing on the merged next-hop entity node set to obtain a next-hop entity node set obtained by the current step of walking.

[0119] It should be noted that the above description of the data processing method of the graph database is also applicable to the data processing apparatus of the graph database in this embodiment, which will not be described here.

[0120] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0121] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0122] As Figure 9 shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0123] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0124] The computing unit 901 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the graph database data processing method. For example, in some embodiments, the graph database data processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the graph database data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the graph database data processing method by any other suitable means (e.g., via firmware).

[0125] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0130] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and poor scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may be a cloud server, a server in a distributed system, or a server integrated with blockchain.

[0131] It's important to note that artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). This encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0132] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0133] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data processing method for a graph database, the method comprising: Obtain a set of entity nodes to be processed in the graph database; Dividing the entity node set to obtain multiple entity node subsets; For each of the entity node subsets, determining a storage node to which each entity node in the entity node subset belongs, wherein the storage node to which the entity node belongs includes the entity node and a corresponding next-hop entity node; According to the storage node to which each entity node in the entity node subset belongs, the entity nodes belonging to the same storage node are divided into a group to obtain multiple groups of entity nodes; For each group of entity nodes, receiving a next-hop entity node result after deduplication returned by a storage node corresponding to the current entity node group for the current entity node group; Determine the next hop entity node set corresponding to the entity node subset according to the next hop entity node result of each group of entity nodes; According to the next-hop entity node set corresponding to each entity node subset, the next-hop entity node set corresponding to the entity node set is determined.

2. The method according to claim 1, wherein The determining, based on the next-hop entity node set corresponding to each entity node subset, the next-hop entity node set corresponding to the entity node set includes: Deduplication is performed on the next-hop entity node set corresponding to each entity node subset to obtain deduplication results corresponding to each entity node subset; Merging the deduplication results corresponding to all the entity node subsets to obtain a candidate next-hop entity node set corresponding to the entity node set; Deduplication is performed again on the candidate next-hop entity node set to obtain a next-hop entity node set corresponding to the entity node set.

3. The method according to claim 1, wherein The determining, based on the next-hop entity node set corresponding to each entity node subset, the next-hop entity node set corresponding to the entity node set includes: Merging the next-hop entity node sets corresponding to all the node subsets to obtain a merged next-hop entity node set; Deduplication processing is performed on the merged next-hop entity node set to obtain a next-hop entity node set corresponding to the entity node set.

4. The method according to claim 2, wherein: Deduplication of the next-hop entity node set corresponding to each entity node subset is performed to obtain deduplication results corresponding to each entity node subset, including: For each entity node subset, performing node traversal on a next-hop entity node set corresponding to the entity node subset; For the traversed target entity node, determining the interval to which the node identifier of the target entity node belongs, wherein the interval is obtained by segmenting the value range of the node identifier; Determining, based on a deduplication method corresponding to the interval, whether the target entity node has appeared in the traversed nodes; The target entity node in the next-hop entity node set is deleted.

5. The method according to claim 4, wherein The determining, based on the deduplication method corresponding to the interval, whether the target entity node has appeared in the traversed nodes includes: Obtaining a plurality of bit sets corresponding to the interval; Segmenting the bits of the node identifier to obtain a plurality of bit segments corresponding to the node identifier; For each bit segment, obtaining a target bit set corresponding to the bit segment from the multiple bit sets, wherein a bit length of the target bit set is greater than or equal to a bit length of the bit segment; Obtaining a bit corresponding to the value of the bit segment from the target bit set; In a case where the values ​​of the bits corresponding to the multiple bit segments are all first values, it is determined that the target entity node has appeared in the traversed nodes, wherein the first value indicates that the value of the corresponding bit segment has appeared.

6. The method according to claim 4, wherein: The determining, based on the deduplication method corresponding to the interval, whether the target entity node has appeared in the traversed nodes includes: Obtaining a hash table corresponding to the interval; Obtain the upper N bits and the lower M bits of the node identifier, where N is an integer greater than 1 and less than K, K represents the bit length of the node identifier, and M is the difference between K and N; Obtaining a bit set corresponding to the values ​​of the upper N bits from the hash table; Obtaining a target bit corresponding to the value of the lower M bits from the bit set; When the value of the target bit is a first value, it is determined that the target entity node has appeared in the traversed nodes, wherein the first value is used to indicate that the values ​​of the lower M bits have appeared.

7. The method according to claim 4, wherein: The determining, based on the deduplication method corresponding to the interval, whether the target entity node has appeared in the traversed nodes includes: Obtaining a bit set corresponding to the interval; Obtaining a target bit corresponding to the node identifier from the bit set; When the value of the target bit is the first value, it is determined that the target entity node has appeared in the traversed nodes.

8. A data processing device for a graph database, the device comprising: A first acquisition module is used to acquire a set of entity nodes to be processed in the graph database; a partitioning module, configured to partition the entity node set to obtain a plurality of entity node subsets; a first determining module configured to determine, for each entity node subset, a storage node to which each entity node in the entity node subset belongs, wherein the storage node to which the entity node belongs includes the entity node and a corresponding next-hop entity node; a second determining module configured to group entity nodes belonging to the same storage node into a group based on the storage node to which each entity node in the entity node subset belongs, so as to obtain multiple groups of entity nodes; for each group of entity nodes, receive a deduplicated next-hop entity node result returned by the storage node corresponding to the current entity node group for the current entity node group; and determine a next-hop entity node set corresponding to the entity node subset based on the next-hop entity node result of each group of entity nodes; The third determining module is configured to determine the next-hop entity node set corresponding to the entity node set according to the next-hop entity node set corresponding to each entity node subset.

9. The device according to claim 8, wherein The second determining module includes: A first deduplication unit is configured to perform deduplication on the next-hop entity node set corresponding to each entity node subset, so as to obtain a deduplication result corresponding to each entity node subset; A first merging unit is configured to merge deduplication results corresponding to all entity node subsets to obtain a candidate next-hop entity node set corresponding to the entity node set; The second deduplication unit is configured to perform deduplication again on the candidate next-hop entity node set to obtain a next-hop entity node set corresponding to the entity node set.

10. The device according to claim 8, wherein The second determining module includes: A second merging unit is configured to merge the next-hop entity node sets corresponding to all the node subsets to obtain a merged next-hop entity node set; The third deduplication unit is configured to perform deduplication processing on the merged next-hop entity node set to obtain a next-hop entity node set corresponding to the entity node set.

11. The device according to claim 9, wherein The first deduplication unit includes: A node traversal subunit, configured to perform node traversal on a next-hop entity node set corresponding to each entity node subset; A first determining subunit is configured to determine, for a traversed target entity node, an interval to which a node identifier of the target entity node belongs, wherein the interval is obtained by segmenting a value range of the node identifier; A second determining subunit is configured to determine, based on a deduplication method corresponding to the interval, whether the target entity node has appeared in the traversed nodes; The deleting subunit is configured to delete the target entity node from the next-hop entity node set.

12. The device according to claim 11, wherein The second determining subunit is specifically configured to: Obtaining a plurality of bit sets corresponding to the interval; Segmenting the bits of the node identifier to obtain a plurality of bit segments corresponding to the node identifier; For each bit segment, obtaining a target bit set corresponding to the bit segment from the multiple bit sets, wherein a bit length of the target bit set is greater than or equal to a bit length of the bit segment; Obtaining a bit corresponding to the value of the bit segment from the target bit set; In a case where the values ​​of the bits corresponding to the multiple bit segments are all first values, it is determined that the target entity node has appeared in the traversed nodes, wherein the first value indicates that the value of the corresponding bit segment has appeared.

13. The device according to claim 11, wherein The second determining subunit is specifically configured to: Obtaining a hash table corresponding to the interval; Obtain the upper N bits and the lower M bits of the node identifier, where N is an integer greater than 1 and less than K, K represents the bit length of the node identifier, and M is the difference between K and N; Obtaining a bit set corresponding to the values ​​of the upper N bits from the hash table; Obtaining a target bit corresponding to the value of the lower M bits from the bit set; When the value of the target bit is a first value, it is determined that the target entity node has appeared in the traversed nodes, wherein the first value is used to indicate that the values ​​of the lower M bits have appeared.

14. The device according to claim 11, wherein The second determining subunit is specifically configured to: Obtaining a bit set corresponding to the interval; Obtaining a target bit corresponding to the node identifier from the bit set; When the value of the target bit is the first value, it is determined that the target entity node has appeared in the traversed nodes.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images