A distributed knowledge graph fragmentation method and device

Abstract

Embodiments of the present specification relate to a distributed knowledge graph fragmentation method and device, the method is applied to a distributed system, the distributed system includes a master device and N working devices, the method comprises: any working device clusters nodes in an edge full set assigned to edges in the device subset into a plurality of node clusters, forms a clustering result and sends it to the master device; the master device combines the received N clustering results and sends them to the N working devices; any working device assigns a plurality of node clusters in the received clustering result to a plurality of fragments according to a preset cost evaluation function, forms a fragmentation proposal and sends it to the master device; the master device receives N fragmentation proposals, and determines the first fragmentation result with the minimum cost value as the first target fragmentation result for fragmenting the target knowledge graph.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of knowledge graphs, and in particular, to a distributed knowledge graph sharding method and device. Background Art

[0002] In recent years, knowledge graphs have been widely used in various scenarios, including search recommendations and financial risk management. As these applications expand, the scale of graph data has also increased dramatically, exceeding the memory limitations of a single machine. This has posed challenges for downstream applications such as graph computing and graph deep learning. Currently, the traditional strategy in these fields is to adopt a distributed approach, using a graph sharding system to split large-scale graph data into multiple slices. Each machine then loads one slice and performs computations in a distributed manner.

[0003] However, the above solution faces two challenges when implemented: 1. Graph sharding introduces data redundancy. For example, vertex sharding and edge sharding algorithms introduce vertex and edge redundancy, respectively. 2. Because graph data is distributed across multiple machines, cross-machine communication is inevitably involved, impacting overall performance.

[0004] To address these issues, graph sharding methods are needed that minimize data redundancy and improve data locality within shards, thereby improving the efficiency of downstream graph computing or deep learning applications. Simple random sharding often fails to meet these requirements, so more efficient graph sharding methods are needed. Summary of the Invention

[0005] One or more embodiments of this specification describe a distributed knowledge graph sharding method and device, which aims to reduce the memory usage of a single machine, reduce data redundancy, and improve operational efficiency.

[0006] In a first aspect, a distributed knowledge graph sharding method is provided, which is applied to a distributed system, wherein the distributed system includes a master device and N working devices, and the method includes:

[0007] The master device divides the entire set of edges in the target knowledge graph into N first edge sets and sends them to the N working devices respectively;

[0008] Any working device clusters the nodes of the edges in the first edge set it receives into several node clusters, forms a first clustering result and sends it to the master device;

[0009] The master device merges the received N first clustering results into a second clustering result, and sends the second clustering result to the N working devices;

[0010] Any working device allocates multiple node clusters in the second clustering result to a number of shards according to a preset cost evaluation function, forms a shard proposal, and sends it to the master device, wherein the cost evaluation function is used to evaluate the cost value of the shard result based on the uniformity of the nodes and / or edges allocated between different shards. The shard proposal includes the first shard result and its corresponding cost value;

[0011] The master device receives N sharding proposals and determines the first sharding result with the smallest cost value as the first target sharding result for sharding the target knowledge graph.

[0012] In one possible implementation, the master device divides the entire set of edges in the target knowledge graph into N first edge sets, including:

[0013] The master device divides the entire edge set into N initial edge sets through a first division operation, and sends the initial edge sets to the N working devices respectively;

[0014] Any working device receives the initial edge set, maps each edge in the initial edge set into a first N-dimensional matrix based on its node information, and sends the first N-dimensional matrix to the master device;

[0015] The master device combines the received N first N-dimensional matrices into a second N-dimensional matrix, and divides the entire set of edges into N first edge sets according to the distribution of each edge in the second N-dimensional matrix.

[0016] In a possible implementation, the first division operation includes one of the following divisions: random division, average division, and division based on a modulo N result.

[0017] In a possible implementation, for each edge in the initial edge set, mapping it to a first N-dimensional matrix according to its node information includes:

[0018] For any first target edge in the initial edge set, mapping its two nodes to row indexes and column indexes in the first N-dimensional matrix respectively according to a preset mapping function;

[0019] The first target edge is mapped to a matrix position indicated by the row index and the column index.

[0020] In a possible implementation, the mapping function is a hash function.

[0021] In a possible implementation, the entire set of edges is divided into N first edge sets according to the distribution of each edge in the second N-dimensional matrix, including:

[0022] The edges included in the i-th row and i-th column in the second N-dimensional matrix are formed into an i-th first edge set.

[0023] In a possible implementation, any working device clusters the nodes of the edges in the received first edge set into a plurality of node clusters to form a first clustering result, including:

[0024] For a target node of any edge in the first edge set that has not been clustered, determining the node cluster where the target node is located based on existing node cluster information;

[0025] When the preset conditions are met, the existing node clusters are adjusted to form the first clustering result.

[0026] In a possible implementation, determining the node cluster where the target node is located based on existing node cluster information includes:

[0027] If the number of existing node clusters is less than a preset first threshold, the target node is assigned to a new cluster; otherwise, the target node is assigned to an existing node cluster.

[0028] In a possible implementation, adjusting an existing node cluster further includes:

[0029] If the number of existing edges in any node cluster is greater than a preset second threshold, it is split into two new clusters.

[0030] If the number of existing edges in two node clusters where two nodes connected by an edge are located is less than a preset third threshold, the two nodes are moved to the same cluster.

[0031] In one possible implementation, any working device assigns multiple node clusters in the second clustering result to a number of shards according to a preset cost evaluation function to form a shard proposal, including:

[0032] Randomly assign each node cluster in the second clustering result to a shard to form the current shard result;

[0033] Perform multiple rounds of iterative updates on the current sharding result, wherein any round of iterative updates includes:

[0034] Select a portion of node clusters in the current sharding result, change the shards to which they belong, and form a candidate sharding result;

[0035] Calculating cost values ​​of the current sharding result and the candidate sharding result respectively according to the cost evaluation function, and updating the current sharding result using the candidate sharding result according to the cost values;

[0036] The iteratively updated current shard result and its corresponding cost value are determined as the shard proposal.

[0037] In a possible implementation, the function value of the cost evaluation function is determined according to the number of edges in each shard in the corresponding sharding result and the number of edges of two nodes in different shards.

[0038] In a possible implementation, after determining the first target sharding result, the method further includes:

[0039] The master device notifies the N working devices of the first target sharding result;

[0040] The master device divides the entire edge set into N second edge sets and sends them to the N working devices respectively;

[0041] Any working device receives the second edge set, and for any second target edge in the second edge set, determines the shard where the second target edge is located based on the shard where its node is located indicated by the first target shard result, forms a second shard result, and sends it to the master device;

[0042] The master device merges the N received second sharding results to obtain a second target sharding result for point-slicing the target knowledge graph.

[0043] In one possible implementation, determining the shard where the second target edge is located includes:

[0044] The shard where the second target edge is located is determined according to the number of edges in the shard where the two nodes of the second target edge are located, the degree of each node, and a preset fourth threshold.

[0045] In one possible implementation, forming the second sharding result includes:

[0046] Once any working device is allocated M edges in the second edge set, it synchronizes the current information of each shard with all other working devices once, and updates the value of M as the next synchronization interval.

[0047] In a possible implementation, updating the value of M includes:

[0048] The value of M is updated according to the preset minimum synchronization interval value, the preset fourth threshold value, the number of shards whose number of edges currently accommodated has not exceeded the preset maximum capacity, and the value of N.

[0049] In a second aspect, a distributed knowledge graph sharding method is provided, which is applied to a master device in a distributed system, wherein the distributed system further includes N working devices. The method includes:

[0050] Divide the entire set of edges in the target knowledge graph into N first edge sets, and send them to the N working devices respectively;

[0051] receiving N first clustering results from N worker devices; wherein any first clustering result is formed by a corresponding worker device clustering nodes of edges in a first edge set received by the corresponding worker device into a plurality of node clusters;

[0052] combining the N first clustering results into a second clustering result and sending the second clustering result to the N worker devices;

[0053] receiving N shard proposals from N worker devices; wherein any shard proposal is formed by a corresponding worker device assigning a plurality of node clusters in the second clustering result to a plurality of shards according to a preset cost evaluation function; wherein the cost evaluation function is used to evaluate a cost value of a shard result according to a uniformity of nodes and / or edges assigned among different shards, and the shard proposal contains a first shard result and a corresponding cost value thereof;

[0054] determining a first shard result with a minimum cost value in the N shard proposals as a first target shard result for edge-cutting of the target knowledge graph.

[0055] In a third aspect, a distributed knowledge graph shard method is provided, which is applied to any worker device in N worker devices included in a distributed system, and the distributed system further includes a master device. The method includes:

[0056] receiving a target first edge set from the master device, which is one of N first edge sets divided by the master device from a complete set of edges in a target knowledge graph;

[0057] clustering nodes of edges in the target first edge set into a plurality of node clusters to form a first clustering result and sending the first clustering result to the master device;

[0058] receiving a second clustering result from the master device, which is obtained by the master device combining N first clustering results received by the master device;

[0059] assigning a plurality of node clusters in the second clustering result to a plurality of shards according to a preset cost evaluation function to form a shard proposal, wherein the cost evaluation function is used to evaluate a cost value of a shard result according to a uniformity of nodes and / or edges assigned among different shards, and the shard proposal contains a first shard result and a corresponding cost value thereof;

[0060] sending the shard proposal to the master device, so that the master device determines a first shard result with a minimum cost value in N shard proposals received by the master device as a first target shard result for edge-cutting of the target knowledge graph.

[0061] In a fourth aspect, a distributed system is provided, including a master device and N working devices, wherein:

[0062] The master device is used to divide the entire set of edges in the target knowledge graph into N first edge sets and send them to the N working devices respectively;

[0063] Any working device is used to cluster the nodes of the edges in the first edge set it receives into a plurality of node clusters, form a first clustering result and send it to the master device;

[0064] The master device is further configured to merge the received N first clustering results into a second clustering result, and send the second clustering result to the N working devices;

[0065] Any working device is further configured to, based on a preset cost evaluation function, assign the plurality of node clusters in the second clustering result to a plurality of shards, form a shard proposal, and send the shard proposal to the master device, wherein the cost evaluation function is configured to evaluate the cost of the shard result based on the uniformity of the nodes and / or edges assigned between different shards, and the shard proposal includes the first shard result and its corresponding cost value;

[0066] The main device is also used to receive N sharding proposals and determine the first sharding result with the smallest cost as the first target sharding result for edge slicing the target knowledge graph.

[0067] In a fifth aspect, a knowledge graph sharding device is provided, which is deployed in a master device included in a distributed system, wherein the distributed system further includes N working devices, and the device includes:

[0068] a partitioning unit configured to partition the entire set of edges in the target knowledge graph into N first edge sets, and send the first edge sets to the N working devices respectively;

[0069] A first receiving unit is configured to receive N first clustering results from N working devices, wherein any first clustering result is formed by clustering the nodes of the edges in the first edge set received by the corresponding working device into a plurality of node clusters;

[0070] a cluster merging unit, configured to merge the N first clustering results into a second clustering result, and send the second clustering result to the N working devices;

[0071] A second receiving unit is configured to receive N shard proposals from N working devices; each shard proposal is formed by the corresponding working device assigning multiple node clusters in the second clustering result to a plurality of shards according to a preset cost evaluation function; wherein the cost evaluation function is used to evaluate the cost value of the shard result based on the uniformity of the nodes and / or edges assigned between different shards, and the shard proposal includes the first shard result and its corresponding cost value;

[0072] The determination unit is configured to determine the first sharding result with the smallest cost among the N sharding proposals as the first target sharding result for edge segmentation of the target knowledge graph.

[0073] In a sixth aspect, a knowledge graph sharding device is provided, which is deployed in any one of N working devices included in a distributed system, the distributed system further including a master device, and the device includes:

[0074] A first receiving unit is configured to receive a target first edge set from the master device, which is one of N first edge sets obtained by the master device dividing the entire set of edges in the target knowledge graph;

[0075] a clustering unit configured to cluster the nodes of the edges in the target first edge set into a plurality of node clusters, form a first clustering result, and send the first clustering result to the master device;

[0076] a second receiving unit configured to receive a second clustering result from the master device, where the second clustering result is obtained by the master device merging the N first clustering results received;

[0077] a sharding unit configured to assign the plurality of node clusters in the second clustering result to a plurality of shards according to a preset cost evaluation function to form a sharding proposal, wherein the cost evaluation function is used to evaluate the cost value of the sharding result based on the uniformity of the nodes and / or edges assigned between different shards, and the sharding proposal includes the first sharding result and its corresponding cost value;

[0078] The sending unit is configured to send the shard proposal to the main device, so that the main device determines the first shard result with the smallest cost value among the N received shard proposals as the first target shard result for edge segmentation of the target knowledge graph.

[0079] In a seventh aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0080] In an eighth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0081] The embodiments of this specification propose a distributed knowledge graph sharding method and apparatus. This method supports sharding large-scale graph data, and the output can be split in either point or edge sharding mode to accommodate different scenarios, such as graph computing and graph deep learning. Furthermore, the method employs a streaming sharding strategy, requiring only a few passes through the full graph data, eliminating the need to store the graph data in memory, significantly reducing memory usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0083] Figure 1 A schematic diagram of an implementation scenario of mining risk objects based on a knowledge graph according to one embodiment is shown;

[0084] Figure 2 A schematic diagram illustrating point segmentation and edge segmentation of a knowledge graph according to one embodiment;

[0085] Figure 3 A flowchart illustrating a distributed knowledge graph sharding method according to one embodiment is shown;

[0086] Figure 4 A flowchart illustrating a method for partitioning the entire set of edges in a target knowledge graph into N first edge sets according to one embodiment is shown;

[0087] Figure 5 A schematic block diagram of a distributed system according to one embodiment is shown;

[0088] Figure 6 A schematic block diagram of a knowledge graph sharding apparatus deployed in a master device of a distributed system according to one embodiment is shown;

[0089] Figure 7 A schematic block diagram of a knowledge graph sharding apparatus deployed in any working device of a distributed system according to one embodiment is shown. DETAILED DESCRIPTION

[0090] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0091] Figure 1 A schematic diagram of an implementation scenario of mining risk objects based on a knowledge graph according to an embodiment is shown. Figure 1 In the example, the distributed system consists of a master device and N worker devices, which are used to shard the input target knowledge graph and output edge-sharding results. In scenarios where point-sharding results are required, the edge-sharding results are input into the distributed system, which then converts them into point-sharding results.

[0092] Knowledge graph sharding is to divide the nodes and edges in the original graph into multiple slices according to certain rules, which is convenient for distributed processing of large-scale graph data. It can be divided into point sharding and edge sharding, corresponding to Figure 2 The left and right methods are shown. Vertex sharding can cause nodes in the graph to be assigned to multiple shards simultaneously, resulting in node redundancy. Furthermore, to obtain information about nodes in multiple shards, a cross-shard data query is required. Edge sharding can cause edges in the graph to be assigned to multiple shards simultaneously, resulting in edge redundancy. Furthermore, to obtain information about edges in multiple shards, a cross-shard data query is required.

[0093] There are many graph sharding schemes in related technologies, typical ones include METIS, DistrubitedNE, CLUGP, etc. Among them, METIS belongs to the edge sharding mode, supports large-scale distributed graph sharding, and is the most widely used graph sharding solution. However, it has the following problems: 1. The edge sharding mode is less effective for graph data with hot spots, because in this mode, the one-hop neighbors of the point are all stored on the same shard, which can easily cause data imbalance for data with hot spots. 2. This solution requires loading the full amount of graph data into memory for calculation, which has high requirements for memory resources. A hot spot refers to a node in the graph whose degree exceeds a certain threshold, and the degree of a node refers to the number of neighboring nodes of the node.

[0094] DistrubitedNE uses a point-sharding model. Compared to edge-sharding, it can better process heat map data and has better edge balance, which is beneficial for downstream graph computation tasks. However, it has the following issues: 1. The point-sharding model divides a point's one-hop neighbors into different shards, making it more inefficient than edge-sharding for certain tasks, such as neighbor sampling. 2. This solution requires loading the entire graph data into memory for computation, placing high demands on memory resources.

[0095] CLUGP uses a streaming approach to sharding, requiring only a few passes over the graph data, eliminating the need to load the entire graph into memory. It can also produce results for both vertex and edge sharding. However, it has the following issues: 1. It can only run on a single machine, making it difficult to process large-scale graphs. 2. This solution uses Nash iteration, requiring multiple passes, resulting in low efficiency.

[0096] To address the aforementioned issues, one or more embodiments of this specification describe a distributed knowledge graph sharding method and apparatus. This method supports sharding large-scale graph data, and the output can be split in either point or edge sharding mode to accommodate different scenarios, such as graph computing and graph deep learning. Furthermore, the embodiments of this specification employ a streaming sharding strategy, requiring only a few passes through the full graph data, eliminating the need to store the graph data in memory, significantly reducing memory usage.

[0097] The following describes the specific implementation steps of the above-mentioned distributed knowledge graph sharding method in combination with specific embodiments. Figure 3 The flowchart of the distributed knowledge graph sharding method according to one embodiment is shown. The execution subject of the method can be any platform, server, or device cluster with computing and processing capabilities. It should be noted that the distributed system includes a master device and N working devices. Figure 3 Only the communication process between the master device and any working device is shown. The communication process between the master device and other working devices can refer to Figure 3 The communication process in is launched.

[0098] In step S301, the master device divides the entire set of edges in the target knowledge graph into N first edge sets and sends them to the N working devices respectively.

[0099] In one embodiment, the method for the master device to divide the entire set of edges in the target knowledge graph into N first edge sets is as follows: Figure 4 It should be noted that the distributed system includes a master device and N working devices. Figure 4 Only the communication process between the master device and any working device is shown. The communication process between the master device and other working devices can refer to Figure 4 The communication process in is launched.

[0100] In step S401, the master device divides the entire edge set into N initial edge sets through a first division operation and sends the initial edge sets to the N working devices respectively. The first division operation includes one of the following divisions: random division, average division, and division based on the result of taking modulo N.

[0101] In step S402, any working device receives an initial edge set, maps each edge in the initial edge set to a first N-dimensional matrix according to its node information, and sends the first N-dimensional matrix to the master device.

[0102] Specifically, for any first target edge in the initial edge set, according to a preset mapping function, its two nodes are respectively mapped to the row index and column index in the first N-dimensional matrix, and then the first target edge is mapped to the matrix position indicated by the row index and column index. Exemplarily, for any first target edge e, the two nodes of e are u and v respectively, and the preset mapping function is f(), then e is mapped to the (f(u), f(v)) position of the first N-dimensional matrix. It can be understood that the value of the mapping function does not exceed N.

[0103] In one embodiment, the mapping function may be a hash function.

[0104] In step S403 , the master device merges the received N first N-dimensional matrices into a second N-dimensional matrix, and divides the entire set of edges into N first edge sets according to the distribution of each edge in the second N-dimensional matrix.

[0105] In one embodiment, the edges included in the i-th row and i-th column in the second N-dimensional matrix are grouped into an i-th first edge set.

[0106] Step S403 allows nodes and edges with similar positions in the target knowledge graph to be grouped into the same first edge set. That is, the one-hop neighbor nodes of any node in any first edge set are also in the first edge set. A node's one-hop neighbor nodes are nodes that are directly connected to the node by an edge.

[0107] In other embodiments, other methods of dividing the entire set of edges can be used with reference to the above method, as long as they can make the nodes and edges with similar positions in the target knowledge graph be concentrated into the same first edge set. For example, you can start from one end of the target knowledge graph and use the breadth-first traversal method to traverse the target knowledge graph. Every time E / N edges are traversed, they are divided into a first edge set, and then the traversal is started again from one end of the remaining target knowledge graph, and the above steps are repeated until the traversal is completed. Wherein, E is the number of edges contained in the target knowledge graph.

[0108] Then return to Figure 3 In step S302, any working device clusters the nodes of the edges in the first edge set it receives into several node clusters, forms a first clustering result and sends it to the master device.

[0109] In one embodiment, for a target node of any edge in the first edge set that has not yet been clustered, the node cluster to which the target node belongs is determined based on information about existing node clusters. Specifically, if the number of existing node clusters is less than a preset first threshold, the target node is assigned to a new cluster; otherwise, the target node is assigned to an existing node cluster.

[0110] When the preset condition is met, the existing node clusters are adjusted to form the first clustering result. Specifically, if the number of existing edges in any node cluster is greater than a preset second threshold, the node cluster is split into two new clusters. If the number of existing edges in the two node clusters where the two nodes connected by an edge are located are both less than a preset third threshold, the two nodes are moved to the same cluster.

[0111] It should be noted that since the above operation does not involve simultaneous operation on multiple edges in the first edge set, the operation on any edge in the first edge set can be performed in a streaming manner, that is, only one edge in the first edge set saved on the memory of the working device can be read into the memory of the working device at a time, the working device is caused to perform the above operation, and then the next edge in the first edge set is read into the memory. It is not necessary to read all edges in the first edge set from the memory into the memory at a time for operation. In this way, the memory overhead can be greatly saved.

[0112] In step S303, the master device combines the received N first clustering results into a second clustering result, and sends the second clustering result to the N working devices.

[0113] In step S304, any working device assigns multiple node clusters in the second clustering result to a plurality of shards according to a preset cost evaluation function, forms a shard proposal and sends it to the master device, the shard proposal containing a first shard result and its corresponding cost value.

[0114] In one embodiment, each node cluster in the second clustering result is randomly assigned to a shard to form a current shard result.

[0115] The current shard result is updated in multiple rounds of iteration, wherein any round of iteration includes: selecting a part of node clusters in the current shard result, changing the shards to which they belong to form a candidate shard result; calculating the cost values of the current shard result and the candidate shard result respectively according to the cost evaluation function, and updating the current shard result with the candidate shard result according to the cost values.

[0116] The current shard result after iteration and update is taken as the first shard result, and the first shard result and its corresponding cost value are determined as the shard proposal.

[0117] The end of the multiple rounds of iteration and update can be that a maximum number of iterations is set in advance, and the iteration is stopped when the number of iterations reaches the maximum number of iterations, or other indicators related to the content of iteration can be set to control the end of iteration.

[0118] The selection of a portion of node clusters in the current sharding result can be random selection or other selection methods, such as modulo selection, round-robin selection, etc.

[0119] According to the cost value, using the candidate sharding result to update the current sharding result specifically includes: if the cost value of the candidate sharding result is less than the cost value of the current sharding result, using the candidate sharding result to replace the current sharding result and become the new current sharding result.

[0120] The cost evaluation function is used to evaluate the cost of the resulting sharding based on the uniformity of the nodes and / or edges distributed across the shards. The cost evaluation function's value is determined by the number of edges in each shard in the resulting sharding, as well as the number of edges between two nodes in different shards.

[0121] In a specific embodiment, the cost evaluation function cost can be defined as shown in formula (1):

[0122] cost = αcost a +(1-α)cost b (1)

[0123] Among them, cost a It is used to measure the degree of balance of the distribution of nodes and edges between shards. The more evenly the nodes and edges are distributed, the higher the cost. a The smaller the value of cost b It is used to measure the number of edges that are cut off in the sharding result, that is, the number of edges where two nodes are located in different shards. The fewer the number of cut edges, the higher the cost. b The smaller the value of α is, the smaller the cost is. α is a real number between 0 and 1, which is used to adjust the cost a and cost b The weight of the cost.

[0124] In a more specific embodiment, cost a It can be defined as shown in formula (2), cost b It can be defined as shown in formula (3):

[0125]

[0126]

[0127] Where k is the total number of shards, p i represents the i-th fragment, P = {p i} represents the set of all shards, |p i | represents the number of edges contained in shard i, V represents the set of all nodes, |e(pi ,V\p i )| represents the number of severed edges in the i-th shard.

[0128] In some other embodiments, other costs may also be defined. a and cost b , as long as it can measure the balance of the distribution of nodes and edges between each shard, and the number of edges that are cut off in the sharding result.

[0129] In step S305, the master device receives N sharding proposals and determines the first sharding result with the smallest cost as the first target sharding result for sharding the target knowledge graph. The first target sharding result is an edge segmentation result, and any shard in the first target sharding result stores the nodes assigned to the shard and all edges connected to each node in the target knowledge graph.

[0130] In some embodiments, after obtaining the edge segmentation results for the target knowledge graph in step S305, the method further includes steps S306-S309 of converting the edge segmentation results into point segmentation results. Steps S306-S309 are used to determine the shard to which each edge e in the target knowledge graph belongs based on the shards to which its two nodes u and v belong in the first target sharding result.

[0131] In step S306, the master device notifies the N working devices of the first target sharding result.

[0132] In step S307, the master device divides the entire edge set into N second edge sets and sends them to the N working devices respectively. The division method includes one of the following divisions: random division, average division, and division based on the result of taking modulo N.

[0133] In step S308, any working device receives the second edge set, and for any second target edge in the second edge set, determines the shard where the second target edge is located based on the shard where its node is located indicated by the first target shard result, forms a second shard result and sends it to the main device.

[0134] In one embodiment, determining the shard where the second target edge is located includes: determining the shard where the two nodes of the second target edge are located according to the number of edges in the shard where the two nodes of the second target edge are located, the degree of each node, and a preset fourth threshold L max , determine the shard where the second target edge is located.

[0135] Specifically, if the number of edges allocated in one of the shards where the two nodes of the second target edge are located exceeds the fourth threshold L max, the second target edge is allocated to another shard; if the number of allocated edges in both shards exceeds the fourth threshold L max , then assign the second target edge to all shards whose first assigned edge number does not exceed L max of.

[0136] If the number of edges allocated to the shards of the two nodes does not exceed L max , and the two nodes are in the same shard, the second target edge is assigned to that shard; if the two nodes are in different shards, the second target edge is assigned to the shard of the node with the smaller degree.

[0137] In one embodiment, forming the second sharding result includes: once any working device is assigned M edges from the second edge set, synchronizing the current information of each shard with all other working devices, and updating the value of M as the next synchronization interval. Updating the value of M includes updating the value of M based on a preset minimum synchronization interval, a preset fourth threshold, the number of shards whose number of edges currently accommodated does not exceed a preset maximum capacity, and the value of N.

[0138] In a more specific embodiment, the value of M is updated using formula (4):

[0139]

[0140] Among them, γ is the preset minimum synchronization interval value, P * Indicates that the number of edges has not exceeded L max The set of shards, |p| represents the number of edges in shard p.

[0141] During the synchronization interval, each shard can process its assigned edges independently, and no new shard with an edge count exceeding the threshold will appear. (L max - max p ∈P* (|p|)) indicates that the max The number of edges that can be assigned to the shard closest to the threshold is calculated by dividing this value by N to get the maximum number of edges that each worker can assign to this shard. In this way, as long as synchronization is performed every M edges, even if all workers assign their own edges to this shard, the number of edges in this shard will not exceed the threshold.

[0142] In other embodiments, other methods for updating M may be defined with reference to the above method, as long as they can control the number of edges in each shard not to exceed the preset fourth threshold.

[0143] It should be noted that since the above operations do not involve simultaneous operations on multiple edges in the second edge set, the above operations on any second target edge in the second edge set can be performed in a streaming manner. In other words, only one second target edge can be read from the second edge set stored in the memory of the working device into the memory of the working device at a time, and the working device can be instructed to perform the above operations, and then the next second target edge can be read from the second edge set into the memory. This eliminates the need to read all the edges in the second edge set from the memory into the memory at once for the operations. This can significantly save memory overhead.

[0144] In step S309, the master device merges the N received second sharding results to obtain a second target sharding result for point-slicing the target knowledge graph. The second target sharding result is a point-slicing result, and any shard in the second target sharding result stores the edges assigned to the shard and the nodes connected by each edge in the target knowledge graph.

[0145] Based on the above concept, the embodiment of this specification also provides a distributed knowledge graph sharding method, which is applied to a main device in a distributed system, and the distributed system also includes N working devices. The method includes the following steps: dividing the entire set of edges in the target knowledge graph into N first edge sets, and sending them to the N working devices respectively; receiving N first clustering results from the N working devices; any of the first clustering results is formed by the corresponding working device clustering the nodes of the edges in the first edge set it receives into several node clusters; merging the N first clustering results into a second clustering result, and the second clustering The result is sent to the N working devices; N shard proposals are received from the N working devices; any shard proposal is formed by the corresponding working device allocating multiple node clusters in the second clustering result to several shards according to a preset cost evaluation function; wherein the cost evaluation function is used to evaluate the cost value of the shard result according to the uniformity of the nodes and / or edges allocated between different shards, and the shard proposal includes the first shard result and its corresponding cost value; the first shard result with the smallest cost value among the N shard proposals is determined as the first target shard result for edge segmentation of the target knowledge graph.

[0146] An embodiment of the present specification also provides a distributed knowledge graph sharding method, which is applied to any one of N working devices included in a distributed system, wherein the distributed system further includes a master device, and the method includes the following steps: receiving a target first edge set from the master device, which is one of the N first edge sets obtained by the master device dividing the entire set of edges in the target knowledge graph; clustering the nodes of the edges in the target first edge set into several node clusters to form a first clustering result and sending it to the master device; receiving a second clustering result from the master device, wherein the second clustering result is obtained by merging the N first clustering results received by the master device; according to a preset cost evaluation function, allocating multiple node clusters in the second clustering result to several shards to form sharding proposals, wherein the cost evaluation function is used to evaluate the cost value of the sharding result based on the uniformity of the nodes and / or edges allocated between different shards, and the sharding proposal includes the first sharding result and its corresponding cost value; sending the sharding proposal to the master device, so that the master device determines the first sharding result with the smallest cost value among the N received sharding proposals as the first target sharding result for edge sharding of the target knowledge graph.

[0147] Based on the above concept, the embodiment of this specification also provides a distributed system 500, including a master device and N working devices, such as Figure 5 As shown, wherein: the master device is used to divide the entire set of edges in the target knowledge graph into N first edge sets, and send them to the N working devices respectively; any working device is used to cluster the nodes of the edges in the first edge set it receives into several node clusters, form a first clustering result and send it to the master device; the master device is also used to merge the N received first clustering results into a second clustering result, and send the second clustering result to the N working devices; any working device is also used to allocate multiple node clusters in the second clustering result to several shards according to a preset cost evaluation function, form a shard proposal and send it to the master device, wherein the cost evaluation function is used to evaluate the cost value of the shard result according to the uniformity of the nodes and / or edges allocated between different shards, and the shard proposal includes the first shard result and its corresponding cost value; the master device is also used to receive N shard proposals, and determine the first shard result with the smallest cost value as the first target shard result for edge segmentation of the target knowledge graph.

[0148] According to another embodiment, a knowledge graph sharding device is provided, which is deployed in a master device included in a distributed system, and the distributed system also includes N working devices. Figure 6 A schematic block diagram of the knowledge graph sharding device is shown in FIG. Figure 6 As shown, the apparatus 600 includes:

[0149] The dividing unit 601 is configured to divide the entire set of edges in the target knowledge graph into N first edge sets and send them to the N working devices respectively; the first receiving unit 602 is configured to receive N first clustering results from the N working devices; any of the first clustering results is formed by the corresponding working device clustering the nodes of the edges in the first edge set it receives into several node clusters; the cluster merging unit 603 is configured to merge the N first clustering results into a second clustering result, and send the second clustering result to the N working devices; the second receiving unit 604 is configured to receive N first clustering results from the N working devices; The working device receives N shard proposals; any one of the shard proposals is formed by the corresponding working device allocating multiple node clusters in the second clustering result to several shards according to a preset cost evaluation function; wherein the cost evaluation function is used to evaluate the cost value of the shard result according to the uniformity of the nodes and / or edges allocated between different shards, and the shard proposal includes a first shard result and its corresponding cost value; the determination unit 605 is configured to determine the first shard result with the smallest cost value among the N shard proposals as the first target shard result for edge segmentation of the target knowledge graph.

[0150] According to another embodiment, a knowledge graph sharding device is provided, which is deployed in any one of N working devices included in a distributed system, wherein the distributed system further includes a master device. Figure 7 A schematic block diagram of the knowledge graph sharding device is shown in FIG. Figure 7 As shown, the apparatus 700 includes:

[0151] The first receiving unit 701 is configured to receive a target first edge set from the master device, which is one of the N first edge sets obtained by the master device dividing the entire set of edges in the target knowledge graph; the clustering unit 702 is configured to cluster the nodes of the edges in the target first edge set into a plurality of node clusters, form a first clustering result and send it to the master device; the second receiving unit 703 is configured to receive a second clustering result from the master device, which is obtained by the master device merging the N first clustering results it receives; the sharding unit 704 is configured to generate a plurality of node clusters according to the pre-defined set of nodes. Suppose a cost evaluation function to allocate multiple node clusters in the second clustering result to several shards to form a shard proposal, wherein the cost evaluation function is used to evaluate the cost value of the shard result according to the uniformity of the nodes and / or edges allocated between different shards, and the shard proposal includes a first shard result and its corresponding cost value; the sending unit 705 is configured to send the shard proposal to the main device, so that the main device determines the first shard result with the smallest cost value among the N received shard proposals as the first target shard result for edge splitting of the target knowledge graph.

[0152] According to another embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in any one of the above embodiments.

[0153] According to yet another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the above embodiments is implemented.

[0154] As mentioned above, METIS and DistributedNE only support edge sharding and point sharding modes, respectively. However, the technical solutions of the embodiments of this specification can generate both edge and point sharding results simultaneously, thus having wider adaptability. Furthermore, METIS and DistributedNE do not support streaming sharding, requiring the full graph data to be loaded into memory, resulting in huge memory requirements. This solution, on the other hand, only requires several passes through the graph data and stores some intermediate results, eliminating the need to store the full graph data, significantly reducing memory usage.

[0155] The CLUGP algorithm has a similar architecture to this solution, but it can only run on a single machine and lacks distributed support, making it difficult to process large-scale graph data. The technical solutions in the embodiments of this specification provide a distributed solution. Furthermore, CLUGP uses Nash iteration to calculate the optimal sharding, requiring multiple iterations through various possible sharding results. This results in a large search space and low efficiency. This solution, however, transforms the problem into an optimization problem, achieving higher efficiency.

[0156] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0157] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program instructing the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0158] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed knowledge graph sharding method, applied to a distributed system, wherein the distributed system includes a master device and N working devices, the method comprising: The master device divides the entire set of edges in the target knowledge graph into N first edge sets and sends them to the N working devices respectively; Any working device clusters the nodes of the edges in the first edge set it receives into several node clusters, forms a first clustering result and sends it to the master device; The master device merges the received N first clustering results into a second clustering result, and sends the second clustering result to the N working devices; Any working device allocates multiple node clusters in the second clustering result to a number of shards according to a preset cost evaluation function, forms a shard proposal, and sends it to the master device, wherein the cost evaluation function is used to evaluate the cost value of the shard result based on the uniformity of the nodes and / or edges allocated between different shards. The shard proposal includes the first shard result and its corresponding cost value; The master device receives N sharding proposals and determines the first sharding result with the smallest cost value as the first target sharding result for sharding the target knowledge graph.

2. The method according to claim 1, wherein The master device divides the entire set of edges in the target knowledge graph into N first edge sets, including: The master device divides the entire edge set into N initial edge sets through a first division operation, and sends the initial edge sets to the N working devices respectively; Any working device receives the initial edge set, maps each edge in the initial edge set into a first N-dimensional matrix based on its node information, and sends the first N-dimensional matrix to the master device; The master device combines the received N first N-dimensional matrices into a second N-dimensional matrix, and divides the entire set of edges into N first edge sets according to the distribution of each edge in the second N-dimensional matrix.

3. The method according to claim 2, wherein: The first division operation includes one of the following divisions: random division, average division, and division based on a modulo N result.

4. The method according to claim 2, wherein: For each edge in the initial edge set, according to its node information, it is mapped into a first N-dimensional matrix, including: For any first target edge in the initial edge set, mapping its two nodes to row indexes and column indexes in the first N-dimensional matrix respectively according to a preset mapping function; The first target edge is mapped to a matrix position indicated by the row index and the column index.

5. The method according to claim 4, wherein The mapping function is a hash function.

6. The method according to claim 2, wherein: According to the distribution of each edge in the second N-dimensional matrix, the entire set of edges is divided into N first edge sets, including: The edges included in the i-th row and i-th column in the second N-dimensional matrix are formed into an i-th first edge set.

7. The method according to claim 1, wherein Any working device clusters the nodes of the edges in the first edge set it receives into several node clusters to form a first clustering result, including: For a target node of any edge in the first edge set that has not been clustered, determining the node cluster where the target node is located based on existing node cluster information; When the preset conditions are met, the existing node clusters are adjusted to form the first clustering result.

8. The method according to claim 7, wherein determining the node cluster where the target node is located based on existing node cluster information comprises: If the number of existing node clusters is less than a preset first threshold, the target node is assigned to a new cluster; otherwise, the target node is assigned to an existing node cluster.

9. The method according to claim 7, further comprising: adjusting an existing node cluster; If the number of existing edges in any node cluster is greater than the preset second threshold, it will be split into two new clusters; If the number of existing edges in two node clusters where two nodes connected by an edge are located is less than a preset third threshold, the two nodes are moved to the same cluster.

10. The method according to claim 1, wherein Any working device allocates multiple node clusters in the second clustering result to several shards according to a preset cost evaluation function to form a shard proposal, including: Randomly assign each node cluster in the second clustering result to a shard to form the current shard result; Perform multiple rounds of iterative updates on the current sharding result, wherein any round of iterative updates includes: Select a portion of node clusters in the current sharding result, change the shards to which they belong, and form a candidate sharding result; Calculating cost values ​​of the current sharding result and the candidate sharding result respectively according to the cost evaluation function, and updating the current sharding result using the candidate sharding result according to the cost values; The iteratively updated current shard result and its corresponding cost value are determined as the shard proposal.

11. The method according to claim 1 or 10, wherein: The function value of the cost evaluation function is determined according to the number of edges in each shard in the corresponding sharding result and the number of edges of two nodes in different shards.

12. The method according to claim 1, wherein After determining the first target fragmentation result, the method further includes: The master device notifies the N working devices of the first target sharding result; The master device divides the entire edge set into N second edge sets and sends them to the N working devices respectively; Any working device receives the second edge set, and for any second target edge in the second edge set, determines the shard where the second target edge is located based on the shard where its node is located indicated by the first target shard result, forms a second shard result, and sends it to the master device; The master device merges the N received second sharding results to obtain a second target sharding result for point-slicing the target knowledge graph.

13. The method according to claim 12, wherein: Determining the shard where the second target edge is located includes: The shard where the second target edge is located is determined according to the number of edges in the shard where the two nodes of the second target edge are located, the degree of each node, and a preset fourth threshold.

14. The method according to claim 12, wherein: The second sharding result is formed, including: Once any working device is allocated M edges in the second edge set, it synchronizes the current information of each shard with all other working devices once, and updates the value of M as the next synchronization interval.

15. The method according to claim 14, wherein Update the value of M, including: The value of M is updated according to the preset minimum synchronization interval value, the preset fourth threshold value, the number of shards whose number of edges currently accommodated has not exceeded the preset maximum capacity, and the value of N.

16. A distributed knowledge graph sharding method, applied to a master device in a distributed system, wherein the distributed system further includes N working devices, the method comprising: Divide the entire set of edges in the target knowledge graph into N first edge sets, and send them to the N working devices respectively; Receive N first clustering results from N working devices; wherein any first clustering result is formed by clustering the edge nodes in the first edge set received by the corresponding working device into a plurality of node clusters; merging the N first clustering results into a second clustering result, and sending the second clustering result to the N working devices; Receive N shard proposals from N workers; Any shard proposal is formed by the corresponding working device assigning multiple node clusters in the second clustering result to a number of shards according to a preset cost evaluation function; wherein the cost evaluation function is used to evaluate the cost value of the shard result based on the uniformity of the nodes and / or edges assigned between different shards, and the shard proposal includes the first shard result and its corresponding cost value; The first shard result with the smallest cost among the N shard proposals is determined as the first target shard result for edge segmentation of the target knowledge graph.

17. A distributed knowledge graph sharding method, applied to any one of N working devices included in a distributed system, the distributed system also including a master device, the method comprising: Receive a target first edge set from the master device, which is one of N first edge sets obtained by the master device dividing the entire set of edges in the target knowledge graph; Clustering the edge nodes in the target first edge set into a plurality of node clusters, forming a first clustering result and sending it to the master device; receiving a second clustering result from the master device, where the second clustering result is obtained by merging the N first clustering results received by the master device; Allocate the multiple node clusters in the second clustering result to a plurality of shards according to a preset cost evaluation function to form a shard proposal, wherein the cost evaluation function is used to evaluate the cost value of the shard result based on the uniformity of the nodes and / or edges allocated between different shards, and the shard proposal includes the first shard result and its corresponding cost value; The shard proposal is sent to the master device, so that the master device determines the first shard result with the smallest cost value among the N received shard proposals as the first target shard result for edge segmentation of the target knowledge graph.

18. A distributed system comprising a master device and N working devices, wherein: The master device is used to divide the entire set of edges in the target knowledge graph into N first edge sets and send them to the N working devices respectively; Any working device is used to cluster the nodes of the edges in the first edge set it receives into a plurality of node clusters, form a first clustering result and send it to the master device; The master device is further configured to merge the received N first clustering results into a second clustering result, and send the second clustering result to the N working devices; Any working device is further configured to, based on a preset cost evaluation function, assign the plurality of node clusters in the second clustering result to a plurality of shards, form a shard proposal, and send the shard proposal to the master device, wherein the cost evaluation function is configured to evaluate the cost of the shard result based on the uniformity of the nodes and / or edges assigned between different shards, and the shard proposal includes the first shard result and its corresponding cost value; The main device is also used to receive N sharding proposals and determine the first sharding result with the smallest cost as the first target sharding result for edge slicing the target knowledge graph.

19. A knowledge graph sharding device, deployed in a master device included in a distributed system, the distributed system also including N working devices, the device comprising: a partitioning unit configured to partition the entire set of edges in the target knowledge graph into N first edge sets, and send the first edge sets to the N working devices respectively; A first receiving unit is configured to receive N first clustering results from N working devices, wherein any first clustering result is formed by clustering the nodes of the edges in the first edge set received by the corresponding working device into a plurality of node clusters; a cluster merging unit, configured to merge the N first clustering results into a second clustering result, and send the second clustering result to the N working devices; a second receiving unit configured to receive N shard proposals from N working devices; Any shard proposal is formed by the corresponding working device assigning multiple node clusters in the second clustering result to a number of shards according to a preset cost evaluation function; wherein the cost evaluation function is used to evaluate the cost value of the shard result based on the uniformity of the nodes and / or edges assigned between different shards, and the shard proposal includes the first shard result and its corresponding cost value; The determination unit is configured to determine the first sharding result with the smallest cost among the N sharding proposals as the first target sharding result for edge segmentation of the target knowledge graph.

20. A knowledge graph sharding device, deployed in any one of N working devices included in a distributed system, the distributed system also including a master device, the device comprising: A first receiving unit is configured to receive a target first edge set from the master device, which is one of N first edge sets obtained by the master device dividing the entire set of edges in the target knowledge graph; a clustering unit configured to cluster the edge nodes in the target first edge set into a plurality of node clusters, form a first clustering result, and send the first clustering result to the master device; a second receiving unit configured to receive a second clustering result from the master device, where the second clustering result is obtained by the master device merging the N first clustering results received; a sharding unit configured to assign the plurality of node clusters in the second clustering result to a plurality of shards according to a preset cost evaluation function to form a sharding proposal, wherein the cost evaluation function is used to evaluate the cost value of the sharding result based on the uniformity of the nodes and / or edges assigned between different shards, and the sharding proposal includes the first sharding result and its corresponding cost value; The sending unit is configured to send the shard proposal to the main device, so that the main device determines the first shard result with the smallest cost value among the N received shard proposals as the first target shard result for edge segmentation of the target knowledge graph.

21. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 17.

22. A computing device comprising a memory and a processor, wherein: The memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 17 is implemented.

Images