Graph data partitioning method and device

Abstract

The present invention provides a graph data partitioning method and device, which can be applied to the field of graph data processing technology. The method includes: determining at least one boundary vertex from the subgraph data of a computing node; determining the first expected benefit of migrating the boundary vertex to at least one adjacent computing node based on the benefit weight of the connecting edge between the boundary vertex and at least one neighbor vertex of the boundary vertex, wherein the benefit weight is determined according to the communication overhead between the adjacent computing nodes and the computing nodes, and the first expected benefit represents the saved communication overhead value; determining the boundary vertex to be migrated and the target adjacent computing node based on the first expected benefit of migrating at least one boundary vertex to at least one adjacent computing node; when it is determined that the boundary vertex to be migrated meets the migration condition, migrating the boundary vertex to be migrated to the target adjacent computing node to re-divide the subgraph data of the computing node.

Description

Technical Field

[0001] The present invention relates to the field of graph data processing technology, and in particular to a graph data partitioning method, device, equipment, medium and program product. Background Art

[0002] In the real world, graph data is usually large and growing rapidly, with potentially hundreds of billions of vertices and trillions of edges. For large-scale graph data, graph partitioning techniques are usually used to partition a graph into multiple computing nodes for parallel processing. However, the graph partitioning methods in related technologies may have problems such as not taking into account the complex connection relationships between vertices, resulting in defects in the partitioning and low graph partitioning quality. Alternatively, due to the continuous updating of graph data, new vertices may appear in the subgraphs allocated to computing nodes, resulting in a decrease in graph partitioning quality.

[0003] In the process of realizing the concept of the present invention, it is found that there is a technical problem in the related art that the quality of graph partitioning is low due to the inherent defects of the graph partitioning method or the continuous updating of graph data. Summary of the invention

[0004] In view of the above problems, the present invention provides a graph data partitioning method, apparatus, device, medium and program product.

[0005] According to one aspect of the present invention, a graph data partitioning method is provided, comprising: determining at least one boundary vertex from subgraph data of a computing node, wherein the boundary vertex has at least one neighbor vertex, there is a connecting edge between the boundary vertex and the neighbor vertex, at least one target neighbor vertex of at least one neighbor vertex exists in the respective subgraph data of at least one adjacent computing node, the boundary vertex or the neighbor vertex represents a data object, and the connecting edge represents a communication relationship between the data objects; based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex, determining a first expected benefit of migrating the boundary vertex to at least one adjacent computing node, wherein the benefit weight is determined according to the communication overhead between the adjacent computing nodes and the computing nodes, and the first expected benefit represents the saved communication overhead value; based on the first expected benefit of migrating at least one boundary vertex to at least one adjacent computing node, determining the boundary vertex to be migrated and the target adjacent computing node; when it is determined that the boundary vertex to be migrated meets the migration condition, migrating the boundary vertex to be migrated to the target adjacent computing node to re-partition the subgraph data of the computing node.

[0006] According to an embodiment of the present invention, the graph data partitioning method also includes: packaging the first vertex identifier of the boundary vertex to be migrated and the target first expected benefit of migrating the boundary vertex to be migrated to the target adjacent computing node into a migration notification message; and sending the migration notification message to the target adjacent computing node.

[0007] According to an embodiment of the present invention, the migration condition includes a first sub-migration condition, at least one adjacent computing node includes an adjacent computing node to be migrated; the subgraph data of the adjacent computing node to be migrated includes the boundary vertices to be migrated of the adjacent computing node to be migrated; the boundary vertices to be migrated of the adjacent computing node to be migrated are the boundary vertices of the prepared migrated computing node determined from at least one boundary vertex of the adjacent computing node to be migrated; the above method also includes: receiving a target migration notification message sent by the adjacent computing node to be migrated, wherein the target migration notification message includes a second vertex identifier of the boundary vertex to be migrated of the adjacent computing node to be migrated and a second expected benefit of migrating the boundary vertex to be migrated of the adjacent computing node to be migrated into the computing node; when it is determined based on the second vertex identifier that there is a connecting edge between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the adjacent computing node to be migrated, the target first expected benefit and the second expected benefit are compared to obtain a comparison result; when it is determined that the comparison result indicates that the target first expected benefit is greater than the second expected benefit, it is determined that the boundary vertex to be migrated of the computing node meets the first sub-migration condition.

[0008] According to an embodiment of the present invention, the migration condition also includes a second sub-migration condition, and the above method also includes: in response to receiving a migration consent message sent by the target adjacent computing node, determining that the boundary vertex to be migrated of the computing node meets the second sub-migration condition, wherein the migration consent message is sent by the target adjacent computing node when it is determined based on the migration notification message that there is a connecting edge between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the target adjacent computing node, and the target first expected benefit is greater than the third expected benefit of the boundary vertex to be migrated of the target adjacent computing node.

[0009] According to an embodiment of the present invention, based on the benefit weights of the connecting edges between a boundary vertex and at least one neighbor vertex of the boundary vertex, a first expected benefit of migrating a boundary vertex to each of at least one adjacent computing nodes is determined, including: determining an initial communication cost value of the boundary vertex based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex; simulating the migration of the boundary vertex to at least one adjacent computing node to obtain simulated boundary vertices corresponding to at least one adjacent computing node; determining a simulated weighted benefit of the connecting edges between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex based on the first node identifier of the computing node and the second node identifier of at least one adjacent computing node; for each simulated boundary vertex, determining a simulated communication cost value of migrating the boundary vertex to the adjacent computing node based on the simulated weighted benefit of the connecting edges between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex; determining the first expected benefit of migrating the boundary vertex to each of the at least one adjacent computing nodes based on the initial communication cost and the simulated communication cost value of migrating the boundary vertex to each of the at least one adjacent computing nodes.

[0010] According to an embodiment of the present invention, a computing node and at least one adjacent computing node are connected in sequence according to a preset topological structure; the benefit weight is determined in the following manner: using a central node, based on a preset topological structure, the computing node and at least one adjacent computing node are mapped to a coordinate system to obtain the coordinates of the computing node and the coordinates of at least one adjacent computing node, wherein the central node is communicatively connected with the computing node and at least one adjacent computing node; the coordinates of the computing node and the coordinates of at least one adjacent computing node are sent to the computing node, so that the computing node determines the communication cost value between the computing node and at least one adjacent computing node based on the coordinates of the computing node and the coordinates of at least one adjacent computing node, and determines the benefit weight of the connecting edge between a boundary vertex and at least one neighbor vertex of the boundary vertex based on the communication cost value between the computing node and at least one adjacent computing node.

[0011] According to an embodiment of the present invention, before determining at least one boundary vertex from the subgraph data of a computing node, it includes: in response to the computing node receiving a dynamic graph update message, based on the vertices to be updated and the connecting edges to be updated of the vertices to be updated included in the dynamic graph update message, updating the original subgraph data in the computing node to obtain the subgraph data.

[0012] According to an embodiment of the present invention, based on the first expected benefit of migrating at least one boundary vertex to at least one adjacent computing node, the boundary vertices to be migrated and the target adjacent computing node are determined, including: sorting the first expected benefits of migrating the boundary vertices to at least one adjacent computing node to obtain a sorting result; using the adjacent computing node with the highest first expected benefit represented by the sorting result as a candidate adjacent computing node for the boundary vertex; based on the first migration benefit of at least one boundary vertex migrating to the candidate adjacent computing node of at least one boundary vertex, the boundary vertex to be migrated is determined, wherein the boundary vertex to be migrated is the boundary vertex with the highest first expected benefit; and using the candidate adjacent computing node of the boundary vertex to be migrated as the target adjacent computing node.

[0013] According to an embodiment of the present invention, migrating the boundary vertices to be migrated to the target adjacent computing node includes: packaging the boundary vertices to be migrated and the connecting edges of the boundary vertices to be migrated in the subgraph data of the computing node into a migration data packet; sending the migration data packet to the target adjacent computing node; deleting the boundary vertices to be migrated and the connecting edges of the boundary vertices to be migrated in the subgraph data of the computing node.

[0014] Another aspect of the present invention provides a graph data partitioning device, including: a first determination module, used to determine at least one boundary vertex from the subgraph data of a computing node, wherein the boundary vertex has at least one neighbor vertex, there is a connecting edge between the boundary vertex and the neighbor vertex, at least one target neighbor vertex of at least one neighbor vertex exists in the respective subgraph data of at least one adjacent computing node, the boundary vertex or the neighbor vertex represents a data object, and the connecting edge represents the communication relationship between the data objects; a second determination module, used to determine the first expected benefit of migrating the boundary vertex to at least one adjacent computing node based on the benefit weight of the connecting edge between the boundary vertex and at least one neighbor vertex of the boundary vertex, wherein the benefit weight is determined according to the communication overhead between the adjacent computing nodes and the computing nodes, and the first expected benefit represents the saved communication overhead value; a third determination module, used to determine the boundary vertex to be migrated and the target adjacent computing node based on the first expected benefit of migrating at least one boundary vertex to at least one adjacent computing node; a migration module, used to migrate the boundary vertex to be migrated to the target adjacent computing node when it is determined that the boundary vertex to be migrated meets the migration condition, so as to re-divide the subgraph data of the computing node.

[0015] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0016] Another aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0017] Another aspect of the present invention further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0018] In the graph data partitioning method of the present invention, since there are connecting edges between boundary vertices and neighbor vertices, the boundary vertices and neighbor vertices in the subgraph data represent data objects, and the connecting edges represent the communication relationship between data objects, that is, there will be communication between the boundary vertices and neighbor vertices with connecting edges. By determining the boundary vertices in the computing node that have connecting edges with neighbor vertices in the adjacent computing nodes, it is possible to determine the vertices in the computing node that have additional communication overhead, and based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex, determine the first expected benefit of migrating the boundary vertex to at least one adjacent computing node, so as to determine the communication overhead value that can be saved by migrating the boundary vertex to the adjacent computing node before the actual migration, and then determine the appropriate boundary vertex to be migrated and the target adjacent computing node based on the saving of communication overhead, and when it is determined that the boundary vertex to be migrated meets the migration conditions, migrate the boundary vertex to be migrated to the target adjacent computing node to achieve the sub-graph data redivision, thereby at least partially solving the technical problem of low graph division quality existing in the related art, and achieving the technical effect of sub-graph data redivision taking into account the complex connection relationship between vertices, saving overall communication overhead, and improving graph division quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0020] Figure 1 A flowchart of a graph data partitioning method according to an embodiment of the present invention is shown.

[0021] Figure 2 A schematic diagram of a graph data partitioning method according to an embodiment of the present invention is shown.

[0022] Figure 3 A flowchart of a graph data partitioning method according to another embodiment of the present invention is shown.

[0023] Figure 4 A structural block diagram of a graph data partitioning device according to an embodiment of the present invention is shown.

[0024] Figure 5A block diagram of an electronic device suitable for implementing a graph data partitioning method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0028] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0029] It should be noted that the graph data partitioning method and device of the present invention can be used in the field of graph data processing technology, and can also be used in any field other than graph data processing technology, such as the field of artificial intelligence technology, the field of computer technology, etc. There is no limitation on the application field of the graph data partitioning method and device of the present invention.

[0030] Graph computing refers to computing on graph data. Graph computing can be widely used in social networks, recommendation systems, network security, text retrieval, biomedicine and other fields. For example, web pages can be converted into vertices in graph data, and the links between web pages can be converted into edges in graph data.

[0031] In the real world, the graph data we face is usually large and growing rapidly, with hundreds of billions of nodes and trillions of edges. Faced with large-scale graph data, we need graph partitioning technology to divide a graph into multiple computing nodes for parallel processing. Graphs in the real world are dynamically changing, and the dynamic changes of graphs usually reduce the quality of graph partitioning. At this time, we need to use efficient graph partitioning methods to optimize the dynamically changing graph partitions in real time.

[0032] Graph partitioning methods in related technologies generally include a point-based graph partitioning method, which generally divides vertices into different partitions to achieve load balancing. However, due to the complex connection relationships between vertices, and the connection relationships can represent the need for communication between vertices, there may be more vertices with connecting edges but located at different computing nodes after graph partitioning, which makes the communication overhead between computing nodes too large and leads to the problem of low graph partitioning quality.

[0033] Or due to the continuous updating of graph data, new vertices are added to the subgraph data in each computing node, and the new vertices may have complex connection relationships with vertices in other computing nodes, resulting in high communication overhead between computing nodes and low graph partitioning quality.

[0034] An embodiment of the present invention provides a graph data partitioning method, which determines at least one boundary vertex from the subgraph data of a computing node, wherein the boundary vertex has at least one neighbor vertex, there is a connecting edge between the boundary vertex and the neighbor vertex, at least one target neighbor vertex of at least one neighbor vertex exists in the respective subgraph data of at least one adjacent computing node, the boundary vertex or the neighbor vertex represents a data object, and the connecting edge represents the communication relationship between the data objects; based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex, a first expected benefit of migrating the boundary vertex to at least one adjacent computing node is determined, wherein the benefit weight is determined according to the communication overhead between the adjacent computing nodes and the computing nodes, and the first expected benefit represents the saved communication overhead value; based on the first expected benefit of migrating at least one boundary vertex to at least one adjacent computing node, a boundary vertex to be migrated and a target adjacent computing node are determined; when it is determined that the boundary vertex to be migrated meets the migration condition, the boundary vertex to be migrated is migrated to the target adjacent computing node to re-divide the subgraph data of the computing node.

[0035] Figure 1 A flowchart of a graph data partitioning method according to an embodiment of the present invention is shown.

[0036] like Figure 1 As shown, the method includes operations S110 to S140.

[0037] In operation S110, at least one boundary vertex is determined from the subgraph data of the computing node, wherein the boundary vertex has at least one neighbor vertex, there is a connecting edge between the boundary vertex and the neighbor vertex, at least one target neighbor vertex of at least one neighbor vertex exists in the subgraph data of at least one adjacent computing node, the boundary vertex or the neighbor vertex represents a data object, and the connecting edge represents the communication relationship between the data objects.

[0038] In operation S120, based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex, a first expected benefit of migrating the boundary vertex to at least one adjacent computing node is determined, wherein the benefit weight is determined based on the communication overhead between the adjacent computing nodes and the computing nodes, and the first expected benefit represents the value of the saved communication overhead.

[0039] In operation S130 , based on a first expected benefit of migrating at least one boundary vertex to at least one adjacent computing node, a boundary vertex to be migrated and a target adjacent computing node are determined.

[0040] In operation S140 , when it is determined that the boundary vertex to be migrated meets the migration condition, the boundary vertex to be migrated is migrated to the target adjacent computing node to re-divide the subgraph data of the computing node.

[0041] According to an embodiment of the present invention, the execution subject of the graph data partitioning method is not limited, and may be a distributed system including a central node and multiple computing nodes. The multiple computing nodes may be obtained from a graph accelerator, that is, the distributed system may include a graph accelerator and a central node.

[0042] According to an embodiment of the present invention, the type of graph accelerator is not limited, and it may be an in-memory graph accelerator or a non-in-memory graph accelerator. For an in-memory graph accelerator, the multiple computing nodes included in the in-memory graph accelerator may be in-memory computing nodes, and the distributed system is an in-memory computing system.

[0043] According to an embodiment of the present invention, both the central node and the computing node may be computer devices such as servers, etc., for executing computing tasks generated by data objects.

[0044] According to an embodiment of the present invention, there is no limitation on data objects and connection edges, and the data objects and connection edges may be different in different scenarios. For example, when the sub-graph data represents a social network, the data object may be a user, and the connection edge may be a communication relationship between users such as a like relationship, a chat relationship, etc.; when the sub-graph data represents a web page jump relationship graph, the data object may be a web page, and the connection edge may be a jump relationship between web pages.

[0045] According to an embodiment of the present invention, for each computing node, the adjacent computing node is the computing node where the target neighbor vertex is located, and the computing node and the adjacent computing node are different computing nodes.

[0046] According to an embodiment of the present invention, subgraph data can be stored in each computing node. The subgraph data can be obtained by dividing the original graph data by the central node, or can be obtained by updating the original subgraph data stored in the computing node. There is no limitation on the division method, and it can be a point-based division method, etc., such as the classic graph partitioning algorithm METIS, XtraPuLP, streaming graph partitioning algorithm, etc.

[0047] According to an embodiment of the present invention, subgraph data may include at least one vertex and at least one edge connecting each vertex, wherein the connecting edge indicates a connection relationship between vertices, and through the connection relationship, neighbor vertices having connecting edges with vertices in the subgraph data can be determined.

[0048] According to an embodiment of the present invention, the subgraph data may also include adjacent computing nodes where neighbor vertices are located. When determining a boundary vertex, the computing node may determine a vertex having at least one neighbor vertex from at least one vertex included in the subgraph data, and determine whether the computing nodes where at least one neighbor vertex is located are adjacent computing nodes different from the computing node. If it is determined to be an adjacent computing node, the vertex is determined to be a boundary vertex.

[0049] According to an embodiment of the present invention, the first expected benefit may represent the communication overhead value that can be saved when migrating the boundary vertex to the adjacent computing node.

[0050] According to an embodiment of the present invention, a computing node can realize a simulated migration of a boundary vertex to at least one adjacent computing node through the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex and at least one neighbor vertex of the boundary vertex, and determine the first expected benefit of the simulated migration. For example, the initial communication cost value of the boundary vertex can be determined through the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex, and then the boundary vertex is simulated migrated to the adjacent computing node based on the connecting edges between the boundary vertex and at least one neighbor vertex to determine the simulated communication cost value after the simulated migration of the boundary vertex to the adjacent computing node, and the first expected benefit of the migration is obtained through the initial communication cost value and the simulated communication cost value.

[0051] According to an embodiment of the present invention, the boundary vertex with the highest first expected benefit may be used as the boundary vertex to be migrated, and the adjacent computing node corresponding to the highest first expected benefit may be used as the target adjacent computing node.

[0052] According to an embodiment of the present invention, the migration condition is used to avoid migration interference between the boundary vertices to be migrated of different computing nodes, and the migration condition may include: a first sub-migration condition determined based on the boundary vertices to be migrated that are expected to migrate into the computing node. It may also include the first sub-migration condition and a second sub-migration condition determined by the boundary vertices to be migrated of the target adjacent computing node, the boundary vertices to be migrated may be the boundary vertices to be migrated of the adjacent computing node to be migrated, the adjacent computing node to be migrated is the adjacent computing node that is expected to migrate its own boundary vertices to be migrated into the computing node, and the way in which the adjacent computing node to be migrated determines its own boundary vertices to be migrated is the same as operation S110 to operation S130.

[0053] According to an embodiment of the present invention, when it is determined that the boundary vertex to be migrated does not meet the migration condition, the state of the boundary vertex to be migrated is marked as a suspended migration state to suspend the migration of the boundary vertex to be migrated to the target adjacent computing node.

[0054] According to an embodiment of the present invention, since there are connecting edges between boundary vertices and neighbor vertices, the boundary vertices and neighbor vertices in the subgraph data represent data objects, and the connecting edges represent the communication relationship between data objects, that is, there will be communication between the boundary vertices and neighbor vertices with connecting edges. By determining the boundary vertices in the computing node that have connecting edges with neighbor vertices existing in the adjacent computing nodes, it is possible to determine the vertices with additional communication overhead in the computing node, and based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex, determine the first expected benefit of migrating the boundary vertex to at least one adjacent computing node, so as to achieve the communication overhead value that can be saved by migrating the boundary vertex to the adjacent computing node before the actual migration, thereby determining the appropriate boundary vertex to be migrated and the target adjacent computing node based on the saving of communication overhead, and when it is determined that the boundary vertex to be migrated meets the migration condition, migrate the boundary vertex to be migrated to the target adjacent computing node to achieve the re-division of the subgraph data, thereby at least partially solving the technical problem of low graph division quality existing in the related art, and achieving the technical effect of re-dividing the subgraph data under the consideration of the complex connection relationship between vertices, saving the overall communication overhead, and improving the quality of graph division.

[0055] According to an embodiment of the present invention, the graph data partitioning method may further include the following operations.

[0056] The first vertex identifier of the boundary vertex to be migrated and the target first expected benefit of migrating the boundary vertex to be migrated to the target adjacent computing node are packaged into a migration notification message; and the migration notification message is sent to the target adjacent computing node.

[0057] According to an embodiment of the present invention, the target adjacent computing node can be notified by packaging the vertex identifier of the boundary vertex to be migrated, i.e., the first vertex identifier, and the target first migration benefit of the boundary vertex to be migrated into a message and sending it to the target adjacent computing node, so that the target adjacent computing node can determine whether there is migration interference between the boundary vertex to be migrated and the boundary vertex to be migrated of the target adjacent computing node. If so, the migration benefits of the two are compared to ensure that the node to be migrated with a large migration benefit can be migrated, thereby ensuring the overall migration benefit of multiple computing nodes.

[0058] According to an embodiment of the present invention, the migration condition includes a first sub-migration condition; at least one adjacent computing node includes an adjacent computing node to be migrated; the subgraph data of the adjacent computing node to be migrated includes the boundary vertices to be migrated of the adjacent computing node to be migrated; the boundary vertices to be migrated of the adjacent computing node to be migrated are the boundary vertices of the prepared migration computing node determined from at least one boundary vertex of the adjacent computing node to be migrated; the graph data partitioning method may also include the following operations.

[0059] Receive a target migration notification message sent by an adjacent computing node to be migrated, wherein the target migration notification message includes a second vertex identifier of a boundary vertex to be migrated of the adjacent computing node to be migrated and a second expected benefit of migrating the boundary vertex to be migrated of the adjacent computing node to be migrated into the computing node; when it is determined based on the second vertex identifier that there is a connecting edge between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the adjacent computing node to be migrated, compare the target first expected benefit and the second expected benefit to obtain a comparison result; when it is determined that the comparison result indicates that the target first expected benefit is greater than the second expected benefit, determine that the boundary vertex to be migrated of the computing node satisfies the first sub-migration condition.

[0060] According to an embodiment of the present invention, the second expected benefit is the maximum expected benefit of migrating out the boundary vertices to be migrated of the adjacent computing node to be migrated, and the determination method thereof is the same as the determination method of the target first expected benefit of the computing node.

[0061] According to an embodiment of the present invention, the computing node may match the second vertex identifier with the third identifier of at least one neighbor vertex of the boundary vertex to be migrated. If the matching result indicates the existence of a third identifier that is identical to the second vertex identifier, it is determined that there is a connecting edge between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the adjacent computing node to be migrated, that is, it is determined that there is migration interference between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the adjacent computing node to be migrated.

[0062] According to an embodiment of the present invention, when it is determined that there is migration interference, the computing node can compare the migration benefits of the two, that is, compare the target first expected benefit with the second expected benefit. When it is determined that the target first expected benefit is greater than the second expected benefit, it is determined that the boundary vertex to be migrated of the computing node meets the first sub-migration condition.

[0063] According to an embodiment of the present invention, when it is determined that there is no migration interference between the boundary vertices to be migrated of the computing node and the boundary vertices to be migrated of the adjacent computing node to be migrated, it can be determined that the boundary vertices to be migrated of the computing node meet the first sub-migration condition.

[0064] According to an embodiment of the present invention, when it is determined that the target first expected benefit is less than or equal to the second expected benefit, the computing node can return a migration consent message to the adjacent computing node to be migrated to indicate that it agrees to the vertex migration operation of the adjacent computing node to be migrated.

[0065] According to an embodiment of the present invention, in some embodiments, when it is determined that the boundary vertices to be migrated meet the first sub-migration condition, it is determined that the boundary vertices to be migrated meet the migration condition, so that whether the migration can be carried out can be judged only by the computing node itself, so that while reducing a certain communication overhead, the redivision overhead is made as small as possible, that is, the speed of subgraph data redivision is improved.

[0066] According to an embodiment of the present invention, by determining whether there is a connecting edge between the boundary vertex to be migrated in the computing node and the boundary vertex to be migrated that is intended to migrate into the computing node, it is possible to determine whether there is migration interference between the two. If there is migration interference, based on the target first expected benefit and the second expected benefit, one of the two is determined to be migrated, thereby avoiding redundant migration due to migration interference, which makes it impossible to reduce the communication overhead as expected, or even increases.

[0067] According to an embodiment of the present invention, the migration condition further includes a second sub-migration condition, and the graph data partitioning method may further include the following operations.

[0068] In response to receiving a migration consent message sent by a target adjacent computing node, it is determined that the boundary vertex to be migrated of the computing node meets the second sub-migration condition, wherein the migration consent message is sent by the target adjacent computing node when it is determined based on the migration notification message that there is a connecting edge between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the target adjacent computing node, and the target first expected benefit is greater than the third expected benefit of the boundary vertex to be migrated of the target adjacent computing node.

[0069] According to an embodiment of the present invention, the third expected benefit is also the maximum expected benefit of migrating the boundary vertices to be migrated in the target adjacent computing node.

[0070] According to an embodiment of the present invention, a migration consent message may be sent by a target adjacent computing node when it is determined that there is no migration interference between a boundary vertex to be migrated in a computing node and a boundary vertex to be migrated in a target adjacent computing node, that is, there is no connecting edge between the two. It may also be sent by a target adjacent computing node when it is determined that there is migration interference between a boundary vertex to be migrated in a computing node and a boundary vertex to be migrated in a target adjacent computing node, but the target first expected benefit is greater than the third expected benefit of the boundary vertex to be migrated of the target adjacent computing node.

[0071] According to an embodiment of the present invention, when a computing node does not receive a migration consent message, it is considered that the boundary vertex to be migrated of the computing node does not meet the second sub-migration condition.

[0072] In some embodiments, when it is determined that the boundary vertex to be migrated satisfies the first sub-migration condition and the second sub-migration condition, it is determined that the boundary vertex to be migrated satisfies the migration condition, so that the migrating parties can reach an agreement and improve the migration success rate.

[0073] According to an embodiment of the present invention, by enabling a computing node to receive messages respectively sent by an adjacent computing node to be migrated and / or a target adjacent computing node, and determining locally at the computing node whether there is migration interference between the boundary vertices to be migrated of the adjacent computing node to be migrated, the computing node and the target adjacent computing node, and in the case of migration interference, migrating the boundary vertices to be migrated with the largest expected benefit, it is possible to reduce redundant migrations by collecting pre-migration information in a distributed manner, thereby improving the quality of graph partitioning. Compared with a solution in which global information of graph data is stored in a master node to determine which computing node a vertex in the graph should be assigned to, it has higher optimization efficiency, can better meet the characteristics of rapid changes in dynamic graphs, and is more in line with the original intention of using existing partitioning information to rapidly optimize subgraph data in computing nodes.

[0074] According to an embodiment of the present invention, at the same time, since the memory bandwidth inside the computing node is very high and will increase with the year-on-year growth of the number of computing nodes, the Internet bandwidth between different computing nodes is limited. Especially for in-memory computing systems, if a master node is set to collect pre-migration information and make judgments, when the number of computing nodes is large, the bandwidth and computing power of the master node will become the bottleneck of the system. However, through the distributed method of reducing redundant migration of the present invention, there is no need to set up a master node to collect pre-migration information and process it. Each computing node sends the information of the boundary vertices to be migrated to the target adjacent computing node, and the target adjacent computing node processes the information locally and determines whether the local computing node to be migrated needs to be migrated, so that each node processes information independently, the system has good scalability, and the target adjacent computing node, the adjacent computing node to be migrated and the local computing node will not have memory access conflicts.

[0075] According to an embodiment of the present invention, a computing node and at least one adjacent computing node are connected in sequence according to a preset topological structure; and the benefit weight is determined in the following manner.

[0076] Using a central node, based on a preset topological structure, a computing node and at least one adjacent computing node are mapped into a coordinate system to obtain the coordinates of the computing node and the coordinates of at least one adjacent computing node, wherein the central node is communicatively connected with the computing node and at least one adjacent computing node; the coordinates of the computing node and the coordinates of at least one adjacent computing node are sent to the computing node, so that the computing node determines the communication cost value between the computing node and at least one adjacent computing node based on the coordinates of the computing node and the coordinates of at least one adjacent computing node, and determines the benefit weight of the connecting edge between a boundary vertex and at least one neighbor vertex of the boundary vertex based on the communication cost value between the computing node and at least one adjacent computing node.

[0077] According to an embodiment of the present invention, the preset topology structure includes the respective positions of the computing node and the adjacent computing nodes, and the respective connection relationships between the computing node and the adjacent computing nodes.

[0078] According to an embodiment of the present invention, the communication overhead between a computing node and an adjacent computing node is determined based on the distance between the two. The coordinates of the computing node and the adjacent computing node respectively can represent the positions of the two in the coordinate system. By performing target operations on the coordinates of the two, a communication overhead value proportional to the distance between the two can be obtained.

[0079] According to an embodiment of the present invention, in the case where the computing node is a computing node included in a graph accelerator, the preset topology is determined based on the type of the graph accelerator, and the preset topology is not limited, and may be a mesh structure, a ring structure, etc. The communication between the computing node and the adjacent computing node may be data transmission according to the order of the preset topology, that is, communication is performed by hop according to the on-chip network formed by the preset topology, such as: in the preset topology, the computing node is connected to the adjacent computing node 1, and the adjacent computing node 1 is connected to the adjacent computing node 2. When the computing node communicates with the adjacent computing node 2, the data needs to be first transmitted to the adjacent computing node 1 and then transmitted to the adjacent computing node 2 through the adjacent computing node 1.

[0080] According to an embodiment of the present invention, when a computing node is not a computing node included in a graph accelerator, a preset topological structure between the computing node and at least one adjacent computing node can be obtained through the distance and position distribution between the computing node and the adjacent computing node.

[0081] According to an embodiment of the present invention, a computing node can determine the communication overhead value between the computing node and the adjacent computing node by performing a target operation on the coordinates of the computing node and the coordinates of the adjacent computing node. The target operation may include a subtraction operation and an addition operation. For example, for the coordinates (2, 2) and (8, 8), the communication overhead value can be determined to be 12 by calculating 8-2+8-2=12.

[0082] According to an embodiment of the present invention, the communication overhead value between a computing node and an adjacent computing node can be used as the benefit weight of the connecting edge between a boundary vertex and a neighbor vertex located at the adjacent computing node, or the communication overhead value can be normalized and then used as the benefit weight.

[0083] According to an embodiment of the present invention, when there are more computing nodes in a distributed system, the positional relationship between a computing node and an adjacent computing node is more important. The communication overhead between two computing nodes that are far away may be much greater than the overhead of two computing nodes that are close. Especially in an in-memory computing system, cross-node communication is performed by hops in the on-chip network. The communication overhead between two vertices that are far away may be dozens of times that of two vertices that are close. Therefore, when dynamically partitioning, it is necessary not only to reduce the number of edge cuts, that is, the number of communications across computing nodes, but also to consider the position of the computing node where the boundary vertex is located in the distributed system, and give priority to eliminating edge cuts that are far away. Therefore, the present invention adopts a benefit weight to determine the migration benefit of migrating boundary vertices, which not only considers the number of communications across nodes, but also considers the average transmission distance of cross-node messages, thereby achieving priority elimination of edge cuts that are farther away, and giving priority to migrating boundary vertices with the highest benefits, thereby improving the efficiency of dynamic partitioning.

[0084] According to an embodiment of the present invention, determining the first expected benefit of migrating a boundary vertex to at least one adjacent computing node based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex may include the following operations.

[0085] Determine an initial communication cost value of the boundary vertex based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex;

[0086] Simulate the migration of boundary vertices to at least one adjacent computing node respectively to obtain simulated boundary vertices corresponding to at least one adjacent computing node respectively; determine the simulated weighted benefits of the connecting edges between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex based on the first node identifier of the computing node and the second node identifier of at least one adjacent computing node respectively; for each simulated boundary vertex, determine the simulated communication cost value of migrating the boundary vertex to the adjacent computing node based on the simulated weighted benefits of the connecting edges between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex; determine the first expected benefit of migrating the boundary vertex to at least one adjacent computing node respectively based on the initial communication cost and the simulated communication cost value of migrating the boundary vertex to at least one adjacent computing node respectively.

[0087] According to an embodiment of the present invention, the benefit weights of the connecting edges between a boundary vertex and at least one neighbor vertex of the boundary vertex may be added together to obtain an initial communication overhead value.

[0088] According to an embodiment of the present invention, the method of simulating migration is not limited. The simulated migration of the computing node can be achieved by establishing a virtual adjacent computing node in the computing node, and establishing a simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex in the virtual adjacent computing node.

[0089] According to an embodiment of the present invention, there is no limitation on the identification types of the first node identifier and the second node identifier, which may be unique identifiers used to characterize computing nodes or adjacent computing nodes. For example, the first node identifier and the second node identifier may be node names, which may correspond to the coordinates of the computing nodes, that is, the coordinates of the computing node corresponding to the first node identifier may be determined by the first node identifier, and the coordinates of the adjacent computing node corresponding to the second node identifier may be determined by the second node identifier, and the simulated weighted benefit may be determined by the coordinates of the computing node and the coordinates of the adjacent computing nodes.

[0090] According to an embodiment of the present invention, the first node identifier can be obtained by the coordinate transformation of the computing node, and the second node identifier can be obtained by the coordinate transformation of the adjacent computing node. The simulation weight benefit can be determined by the first node identifier and the second node identifier. For example, if the coordinate of the computing node is (1, 1), the first vertex identifier can be set to 1-1, and the coordinate of the adjacent computing node is (2, 2), the second vertex identifier can be set to 2-2. The communication overhead value of the computing node and the adjacent computing node can be determined by 2-1+2-1=2, thereby determining the simulation benefit weight more quickly.

[0091] According to an embodiment of the present invention, the simulated communication overhead value for migrating the boundary vertex to the adjacent computing node can be obtained by accumulating the simulated weighted benefits of the connecting edges between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex.

[0092] According to an embodiment of the present invention, a subtraction operation is performed between the simulated communication cost value and the initial communication cost value, thereby obtaining the first expected benefits of migrating the boundary vertices to the adjacent computing nodes.

[0093] According to the embodiment of the present invention, by performing simulated migration in computing nodes and calculating communication overhead values, it is possible to quantify the communication overhead between computing nodes, thereby improving the accuracy of selecting nodes to be migrated.

[0094] According to an embodiment of the present invention, before determining at least one boundary vertex from the subgraph data of the computing node, the following operations may be included.

[0095] In response to the computing node receiving the dynamic graph update message, the original subgraph data in the computing node is updated based on the to-be-updated vertices and the to-be-updated connecting edges of the to-be-updated vertices included in the dynamic graph update message to obtain the subgraph data.

[0096] According to an embodiment of the present invention, a dynamic graph update request may be sent by a central node after receiving a dynamic graph update request sent by a client. In some embodiments, the dynamic graph update request may include graph data to be updated. The central node compares the graph data to be updated with the original graph data, and determines the vertices that are different from the original graph data as vertices to be updated, and the connecting edges of the vertices to be updated as connecting edges to be updated; and packages the vertices to be updated and the connecting edges to be updated of the vertices to be updated into a dynamic graph update message, and sends the dynamic graph update message to the computing node with the most neighbor vertices connected to the connecting edges to be updated. The original subgraph data is the data obtained after segmenting the original graph data, and the subgraph data included in each of the multiple computing nodes constitutes the dynamic graph to be updated.

[0097] According to an embodiment of the present invention, in some embodiments, a dynamic graph update request may include vertices to be updated and edges connecting the vertices to be updated.

[0098] According to an embodiment of the present invention, when a computing node receives a dynamic graph update message and performs an update, operations S101 to S104 are performed to achieve re-division of sub-graph data, thereby solving the problem of resource waste caused by the need to re-divide the graph data as a whole when graph data is updated in the related art. The present invention obtains the vertices to be updated and the connecting edges to be updated through a central node and sends them to the computing node to trigger the computing node to perform a re-division operation, thereby saving certain computing resources and better meeting the characteristics of the huge and increasing scale of graph data in the real world. It has the beneficial effects of low partitioning overhead and high dynamic partitioning quality.

[0099] According to an embodiment of the present invention, determining the boundary vertices to be migrated and the target adjacent computing nodes based on the first expected benefit of migrating at least one boundary vertex to at least one adjacent computing node may include the following operations.

[0100] The first expected benefits of migrating boundary vertices to at least one adjacent computing node are sorted to obtain a sorting result; the adjacent computing node with the highest first expected benefit represented by the sorting result is used as a candidate adjacent computing node for the boundary vertex; based on the first migration benefit of at least one boundary vertex migrating to the candidate adjacent computing node of at least one boundary vertex, the boundary vertex to be migrated is determined, wherein the boundary vertex to be migrated is the boundary vertex with the highest first expected benefit; the candidate adjacent computing node of the boundary vertex to be migrated is used as the target adjacent computing node.

[0101] According to an embodiment of the present invention, the first expected benefits of migrating each boundary vertex to at least one adjacent computing node may be sorted first, thereby determining the highest first expected benefit of each boundary vertex and the candidate adjacent computing node corresponding to the highest first expected benefit.

[0102] According to an embodiment of the present invention, the computing node can sort the highest first expected benefits of each boundary vertex, use the boundary vertex corresponding to the highest first migration benefit as the boundary vertex to be migrated, and use the candidate adjacent computing node of the boundary vertex to be migrated as the target adjacent computing node.

[0103] According to an embodiment of the present invention, migrating a boundary vertex to be migrated to a target adjacent computing node may include the following operations.

[0104] The boundary vertices to be migrated and the connecting edges of the boundary vertices to be migrated in the subgraph data of the computing node are packaged into a migration data packet; the migration data packet is sent to the target adjacent computing node; and the boundary vertices to be migrated and the connecting edges of the boundary vertices to be migrated in the subgraph data of the computing node are deleted.

[0105] According to an embodiment of the present invention, the computing node packages the data related to the boundary vertices to be migrated and the data related to the connecting edges of the boundary vertices to be migrated in the subgraph data into a migration data packet, and deletes it from the subgraph data, thereby realizing the migration of the boundary vertices to be migrated, and further realizing the re-division of the subgraph data.

[0106] According to an embodiment of the present invention, after the target adjacent computing node receives the migration data packet, the subgraph data of the target adjacent computing node may be updated based on the migration data packet.

[0107] Figure 2 A schematic diagram of a graph data partitioning method according to an embodiment of the present invention is shown.

[0108] like Figure 2 As shown, the subgraph data stored in computing node P1, computing node P2, and computing node P3 are respectively shown in 210 and 220. In 210, the subgraph data in P1 includes vertex 1 and vertex 2, the subgraph data in P2 includes vertex 3, vertex 4, and vertex 5, and the subgraph data in P3 includes vertex 6, vertex 7, and vertex 8.

[0109] According to an embodiment of the present invention, in 210, P1, P2, and P3 are adjacent computing nodes, the distances between P1, P2, and P3 are equal, that is, the communication overheads are equal, and the benefit weights of the connecting edges between the boundary vertices in the computing nodes and at least one neighbor vertex of the boundary vertex are all 1.

[0110] According to an embodiment of the present invention, by executing operations S101 to S103, it can be known that the boundary vertex to be migrated in P1 is vertex 2, the target adjacent calculation node is P3, and the target first migration benefit is 1; by executing operations S101 to S103, it can be known that the boundary vertex to be migrated in P2 is vertex 4, the target adjacent calculation node is P1, and the target first migration benefit is 2; by executing operations S101 to S103, it can be known that the boundary vertex to be migrated in P3 is vertex 8, the target adjacent calculation node is P2, and the target first migration benefit is 1.

[0111] According to an embodiment of the present invention, P1 informs P3 that it will move vertex 2 to P3 by sending a migration notification message to P3, and the target first migration benefit is 1. P2 informs P1 that it will move vertex 4 to P1 by sending a migration notification message to P1, and the target first migration benefit is 2. P3 informs P2 that it will move vertex 8 to P2 by sending a migration notification message to P2, and the migration benefit is 1. Since P1 is preparing to migrate its own boundary vertex to be migrated to P3, it can be known that relative to P3, P1 is P3's adjacent computing node to be migrated, and relative to P1, P3 is P1's target adjacent computing node, and the same is true for other computing nodes.

[0112] According to an embodiment of the present invention, each computing node locally processes pre-migration information and determines whether the local vertex needs to be migrated, that is, determines whether the local boundary vertices to be migrated meet the migration conditions. P1 detects that vertex 2 is connected to vertex 4, interfering with each other, and the benefit of migrating vertex 2 is lower than that of vertex 4, so it stops migrating vertex 2. P2 detects that vertex 4 is connected to vertex 8, interfering with each other, but the benefit of migrating vertex 4 is higher than that of vertex 8, so it continues to migrate vertex 4. P3 detects that vertex 2 is connected to vertex 8, interfering with each other, but the benefit of migrating vertex 8 is not higher than that of vertex 2, so it stops migrating vertex 8.

[0113] According to an embodiment of the present invention, it can be found through 220 that after the above-mentioned processing process inside the computing node, only vertex 4 is migrated to realize the redivision of the subgraph data, and after the division, the total number of cross-computing node communications in the distributed system is 5, and the communication overhead is also 5. If no redivision is performed, the number of cross-computing node communications is 7, and the communication overhead is also 7, that is, there is a large communication overhead.

[0114] According to an embodiment of the present invention, although the above partitioning result is not a global optimal solution, the global optimal solution requires storing the global information of the graph in the memory of a master node, and using the global information of the graph to determine which partition the vertices in the graph should be assigned to. This method is consistent with the static graph partitioning method, and is completely opposite to the original intention of dynamic graph partitioning to quickly optimize the graph partitioning using existing partitioning information. Therefore, in dynamic graph partitioning, the present invention uses the local information of the graph to reduce redundant vertex migration, and collects pre-migration information in a distributed manner, because distributed collection of pre-migration information is more suitable for scenarios with a large number of computing nodes. Especially in in-memory computing systems, the memory bandwidth inside the computing nodes is very high, and will increase with the year-on-year increase in the number of computing nodes, but the Internet bandwidth between different nodes is limited. If a master node is set to collect pre-migration information and make judgments, the bandwidth and computing power of the master node will become the bottleneck of the system when there are a large number of computing nodes.

[0115] Figure 3A flowchart of a graph data partitioning method according to another embodiment of the present invention is shown.

[0116] like Figure 3 As shown, the method includes operation 301 to operation S309.

[0117] In operation S301 , at least one boundary vertex is determined from subgraph data of a computation node.

[0118] In operation S302 , a first expected benefit of migrating the boundary vertex to at least one adjacent computing node is determined based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex.

[0119] In operation S303, based on a first expected benefit of migrating at least one boundary vertex to at least one adjacent computing node, a boundary vertex to be migrated and a target adjacent computing node are determined.

[0120] In operation S304, a target migration notification message is received from an adjacent computing node to be migrated, wherein the adjacent computing node to be migrated is an adjacent computing node that takes the computing node as a target adjacent computing node, and the target migration notification message includes a second vertex identifier of a boundary vertex to be migrated of the adjacent computing node to be migrated and a second expected benefit of migrating the boundary vertex to be migrated of the adjacent computing node to be migrated into the computing node.

[0121] In operation S305, it is determined whether there is a connecting edge between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the adjacent computing node to be migrated. If there is a connecting edge, operation S306 is performed; if there is no connecting edge, operation S307 is performed.

[0122] In operation S306, it is determined whether the target first expected benefit is greater than the second expected benefit. If the target first expected benefit is greater than the second expected benefit, operation S307 is performed; if the target first expected benefit is less than or equal to the second expected benefit, operation S309 is performed.

[0123] In operation S307, it is determined whether the boundary vertex to be migrated of the computing node has received a migration consent message. If the boundary vertex to be migrated of the computing node has received a migration consent message, operation S308 is performed; if the boundary vertex to be migrated of the computing node has not received a migration consent message, operation S309 is performed.

[0124] In operation S308 , the boundary vertices to be migrated are migrated to the target adjacent computing nodes to re-partition the subgraph data of the computing nodes.

[0125] In operation S309 , the migration operation of the boundary vertices to be migrated of the computing node is suspended.

[0126] According to an embodiment of the present invention, the above-mentioned graph data partitioning method can realize sub-graph data re-partitioning without modifying the hardware platform.

[0127] Based on the above graph data partitioning method, the present invention also provides a graph data partitioning device. Figure 4 The device is described in detail.

[0128] Figure 4 A structural block diagram of a graph data partitioning device according to an embodiment of the present invention is shown.

[0129] like Figure 4 As shown, the graph data partitioning device 400 of this embodiment includes a first determination module 410 , a second determination module 420 , a third determination module 430 and a migration module 440 .

[0130] The first determination module 410 is used to determine at least one boundary vertex from the subgraph data of the computing node, wherein the boundary vertex has at least one neighbor vertex, there is a connecting edge between the boundary vertex and the neighbor vertex, at least one target neighbor vertex of at least one neighbor vertex exists in the subgraph data of at least one adjacent computing node, the boundary vertex or the neighbor vertex represents a data object, and the connecting edge represents the communication relationship between the data objects.

[0131] The second determination module 420 is used to determine the first expected benefit of migrating the boundary vertex to at least one adjacent computing node based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex, wherein the benefit weights are determined based on the communication overhead between the adjacent computing nodes and the computing nodes, and the first expected benefit represents the value of the saved communication overhead.

[0132] The third determination module 430 is configured to determine a boundary vertex to be migrated and a target adjacent computing node based on a first expected benefit of migrating at least one boundary vertex to at least one adjacent computing node.

[0133] The migration module 440 is used to migrate the boundary vertex to be migrated to the target adjacent computing node to re-divide the subgraph data of the computing node when it is determined that the boundary vertex to be migrated meets the migration condition.

[0134] According to an embodiment of the present invention, the graph data partitioning device 400 further includes: a first packaging module and a first sending module.

[0135] The first packaging module is used to package the first vertex identifier of the boundary vertex to be migrated and the target first expected benefit of migrating the boundary vertex to be migrated to the target adjacent computing node into a migration notification message.

[0136] The first sending module is used to send the migration notification message to the target adjacent computing node.

[0137] According to an embodiment of the present invention, the migration condition includes a first sub-migration condition, at least one adjacent computing node includes an adjacent computing node to be migrated; the subgraph data of the adjacent computing node to be migrated includes the boundary vertices to be migrated of the adjacent computing node to be migrated; the boundary vertices to be migrated of the adjacent computing node to be migrated are the boundary vertices of the prepared migration computing node determined from at least one boundary vertex of the adjacent computing node to be migrated; the graph data partitioning device 400 also includes: a first receiving module, a comparison module and a fourth determination module.

[0138] The first receiving module is used to receive a target migration notification message sent by an adjacent computing node to be migrated, wherein the target migration notification message includes a second vertex identifier of a boundary vertex to be migrated of the adjacent computing node to be migrated and a second expected benefit of migrating the boundary vertex to be migrated of the adjacent computing node to be migrated into the computing node.

[0139] The comparison module is used to compare the target first expected benefit with the second expected benefit to obtain a comparison result when it is determined based on the second vertex identifier that there is a connecting edge between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the adjacent computing node to be migrated.

[0140] The fourth determination module is used to determine that the boundary vertex to be migrated of the computing node meets the first sub-migration condition when it is determined that the comparison result indicates that the first expected benefit of the target is greater than the second expected benefit.

[0141] According to an embodiment of the present invention, the migration condition further includes a second sub-migration condition, and the graph data partitioning apparatus 400 further includes a second receiving module.

[0142] A second receiving module is used to determine that the boundary vertices to be migrated of the computing node meet the second sub-migration condition in response to receiving a migration consent message sent by a target adjacent computing node, wherein the migration consent message is sent by the target adjacent computing node when it is determined based on the migration notification message that there is a connecting edge between the boundary vertices to be migrated of the computing node and the boundary vertices to be migrated of the target adjacent computing node, and the target first expected benefit is greater than the third expected benefit of the boundary vertices to be migrated of the target adjacent computing node.

[0143] According to an embodiment of the present invention, the second determination module 420 includes: a first cost determination submodule, a simulated migration submodule, a benefit determination submodule, a second cost determination submodule and a benefit determination submodule.

[0144] The first cost determination submodule is used to determine the initial communication cost value of the boundary vertex based on the benefit weights of the connecting edges between the boundary vertex and at least one neighbor vertex of the boundary vertex.

[0145] The simulation migration submodule is used to simulate the migration of boundary vertices to at least one adjacent computing node, and obtain simulated boundary vertices corresponding to the at least one adjacent computing node.

[0146] The benefit determination submodule is used to determine the simulated weighted benefits of the connecting edges between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex based on the first node identifier of the computing node and the second node identifier of at least one adjacent computing node.

[0147] The second cost determination submodule is used to determine, for each simulated boundary vertex, a simulated communication cost value for migrating the boundary vertex to an adjacent computing node based on the simulated weighted benefits of the connecting edges between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex.

[0148] The benefit determination submodule is used to determine a first expected benefit of migrating the boundary vertex to each of the at least one adjacent computing nodes based on the initial communication cost and the simulated communication cost value of migrating the boundary vertex to each of the at least one adjacent computing nodes.

[0149] According to an embodiment of the present invention, a computing node and at least one adjacent computing node are sequentially connected according to a preset topology. The revenue weight is determined in the following manner:

[0150] A coordinate determination module is used to use a central node to map a computing node and at least one adjacent computing node into a coordinate system based on a preset topological structure, thereby obtaining the coordinates of the computing node and the coordinates of at least one adjacent computing node, wherein the central node is communicatively connected with the computing node and at least one adjacent computing node.

[0151] The second sending module is used to send the coordinates of the computing node and the coordinates of at least one adjacent computing node to the computing node, so that the computing node determines the communication cost value between the computing node and at least one adjacent computing node based on the coordinates of the computing node and the coordinates of at least one adjacent computing node, and determines the benefit weight of the connecting edge between the boundary vertex and at least one neighbor vertex of the boundary vertex based on the communication cost value between the computing node and at least one adjacent computing node.

[0152] According to an embodiment of the present invention, the graph data partitioning device 400 further includes: an updating module.

[0153] The update module is used to update the original subgraph data in the computing node in response to the computing node receiving the dynamic graph update message, based on the to-be-updated vertices and the to-be-updated connecting edges of the to-be-updated vertices included in the dynamic graph update message, to obtain the subgraph data.

[0154] According to an embodiment of the present invention, the third determination module 430 includes: a sorting submodule, a first determination submodule, a second determination submodule, a second determination submodule, and a third determination submodule.

[0155] The sorting submodule is used to sort the first expected benefits of migrating the boundary vertices to at least one adjacent computing node to obtain a sorting result.

[0156] The first determination submodule is used to use the adjacent computing node with the highest first expected benefit represented by the sorting result as a candidate adjacent computing node for the boundary vertex.

[0157] The second determination submodule is used to determine the boundary vertex to be migrated based on the first migration benefit of at least one boundary vertex migrating to the candidate adjacent computing node of at least one boundary vertex, wherein the boundary vertex to be migrated is the boundary vertex with the highest first expected benefit.

[0158] The third determination submodule is used to use the candidate adjacent computing node of the boundary vertex to be migrated as the target adjacent computing node.

[0159] According to an embodiment of the present invention, the migration module 440 includes: a second packaging module, a third sending module and a deletion module.

[0160] The second packaging module is used to package the boundary vertices to be migrated and the connecting edges of the boundary vertices to be migrated in the subgraph data of the computing node into a migration data packet.

[0161] The third sending module is used to send the migration data packet to the target adjacent computing node.

[0162] The deletion module is used to delete the boundary vertices to be migrated and the connecting edges of the boundary vertices to be migrated in the subgraph data of the computing node.

[0163] According to an embodiment of the present invention, any multiple modules of the first determination module 410, the second determination module 420, the third determination module 430, and the migration module 440 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the first determination module 410, the second determination module 420, the third determination module 430, and the migration module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a proper combination of any of them. Alternatively, at least one of the first determination module 410 , the second determination module 420 , the third determination module 430 , and the migration module 440 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0164] Figure 5 A block diagram of an electronic device suitable for implementing a graph data partitioning method according to an embodiment of the present invention is shown.

[0165] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0166] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0167] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage portion 508 as needed.

[0168] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0169] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0170] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the graph data partitioning method provided by the embodiment of the present invention.

[0171] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0172] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0173] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0174] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0175] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0177] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A graph data partitioning method, characterized in that: The method comprises: Determine at least one boundary vertex from the subgraph data of the computing node, wherein the boundary vertex has at least one neighbor vertex, there is a connecting edge between the boundary vertex and the neighbor vertex, at least one target neighbor vertex of at least one of the neighbor vertices exists in the respective subgraph data of at least one adjacent computing node, the boundary vertex or the neighbor vertex represents a data object, and the connecting edge represents a communication relationship between the data objects; Determine, based on the benefit weights of the connecting edges between the boundary vertex and at least one of the neighboring vertices of the boundary vertex, a first expected benefit of migrating the boundary vertex to each of the at least one adjacent computing nodes, wherein the benefit weight is determined according to the communication overhead between the adjacent computing node and the computing node, and the first expected benefit represents the value of the saved communication overhead; Determine a boundary vertex to be migrated and a target adjacent computing node based on a first expected benefit of migrating at least one of the boundary vertices to at least one of the adjacent computing nodes; When it is determined that the boundary vertex to be migrated meets the migration condition, the boundary vertex to be migrated is migrated to the target adjacent computing node to re-divide the subgraph data of the computing node; The determining, based on the benefit weights of the connecting edges between the boundary vertex and at least one of the neighboring vertices of the boundary vertex, of the first expected benefit of migrating the boundary vertex to each of the at least one adjacent computing nodes comprises: Determining an initial communication cost value of the boundary vertex based on the benefit weights of the connecting edges between the boundary vertex and at least one of the neighboring vertices of the boundary vertex; Simulating migration of the boundary vertices to at least one of the adjacent computing nodes to obtain simulated boundary vertices corresponding to the at least one adjacent computing node; Determine, based on the first node identifier of the computing node and the second node identifier of at least one of the adjacent computing nodes, a simulated weighted benefit of a connection edge between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex; For each of the simulated boundary vertices, determining a simulated communication cost value for migrating the boundary vertex to the adjacent computing node based on simulated weighted gains of respective connecting edges between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex; Based on the initial communication cost and the simulated communication cost value of migrating the boundary vertex to each of the at least one adjacent computing nodes, a first expected benefit of migrating the boundary vertex to each of the at least one adjacent computing nodes is determined.

2. The method according to claim 1, characterized in that: The method further comprises: Packing the first vertex identifier of the boundary vertex to be migrated and the target first expected benefit of migrating the boundary vertex to be migrated to the target adjacent computing node into a migration notification message; The migration notification message is sent to the target adjacent computing node.

3. The method according to claim 2, characterized in that The migration condition includes a first sub-migration condition; at least one of the adjacent computing nodes includes an adjacent computing node to be migrated; the subgraph data of the adjacent computing node to be migrated includes the boundary vertices to be migrated of the adjacent computing node to be migrated; the boundary vertices to be migrated of the adjacent computing node to be migrated are boundary vertices to be migrated to the computing node determined from at least one boundary vertex of the adjacent computing node to be migrated; The method further comprises: Receive a target migration notification message sent by the adjacent computing node to be migrated, wherein the target migration notification message includes a second vertex identifier of a boundary vertex to be migrated of the adjacent computing node to be migrated and a second expected benefit of migrating the boundary vertex to be migrated of the adjacent computing node to be migrated into the computing node; When it is determined based on the second vertex identifier that there is a connecting edge between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the adjacent computing node to be migrated, the target first expected benefit and the second expected benefit are compared to obtain a comparison result; When it is determined that the comparison result indicates that the target first expected benefit is greater than the second expected benefit, it is determined that the boundary vertex to be migrated of the computing node satisfies the first sub-migration condition.

4. The method according to claim 3, characterized in that The migration condition further includes a second sub-migration condition, and the method further includes: In response to receiving a migration consent message sent by the target adjacent computing node, it is determined that the boundary vertex to be migrated of the computing node meets the second sub-migration condition, wherein the migration consent message is sent by the target adjacent computing node when it is determined based on the migration notification message that there is a connecting edge between the boundary vertex to be migrated of the computing node and the boundary vertex to be migrated of the target adjacent computing node, and the target first expected benefit is greater than the third expected benefit of the boundary vertex to be migrated of the target adjacent computing node.

5. The method according to claim 1, characterized in that The computing node and at least one of the adjacent computing nodes are sequentially connected according to a preset topological structure; the benefit weight is determined in the following manner: Using the central node, Based on the preset topological structure, the computing node and at least one of the adjacent computing nodes are mapped into a coordinate system to obtain the coordinates of the computing node and the coordinates of each of the at least one adjacent computing node, wherein the central node is in communication connection with the computing node and at least one of the adjacent computing nodes; The coordinates of the computing node and the coordinates of at least one of the adjacent computing nodes are sent to the computing node, so that the computing node determines the communication cost value between the computing node and at least one of the adjacent computing nodes based on the coordinates of the computing node and the coordinates of at least one of the adjacent computing nodes, and determines the benefit weight of the connecting edge between the boundary vertex and at least one neighbor vertex of the boundary vertex based on the communication cost value between the computing node and at least one of the adjacent computing nodes.

6. The method according to claim 1, characterized in that Before determining at least one boundary vertex from the subgraph data of the computing node, the method further comprises: In response to the computing node receiving a dynamic graph update message, the original subgraph data in the computing node is updated based on the to-be-updated vertices and the to-be-updated connecting edges of the to-be-updated vertices included in the dynamic graph update message to obtain the subgraph data.

7. The method according to claim 1, characterized in that The determining of the boundary vertex to be migrated and the target adjacent computing node based on the first expected benefit of migrating at least one of the boundary vertices to at least one of the adjacent computing nodes comprises: sorting the first expected benefits of migrating the boundary vertices to at least one of the adjacent computing nodes to obtain a sorting result; The adjacent computing node whose first expected benefit is represented by the sorting result is the highest is used as a candidate adjacent computing node for the boundary vertex; Determine a boundary vertex to be migrated based on a first migration benefit of at least one boundary vertex respectively migrating to the candidate adjacent computing node of at least one boundary vertex, wherein the boundary vertex to be migrated is a boundary vertex with the highest first expected benefit; The candidate adjacent computing node of the boundary vertex to be migrated is used as the target adjacent computing node.

8. The method according to claim 1, characterized in that The step of migrating the boundary vertex to be migrated to the target adjacent computing node includes: Packing the boundary vertices to be migrated and the connecting edges of the boundary vertices to be migrated in the subgraph data of the computing node into a migration data packet; Sending the migration data packet to the target adjacent computing node; The boundary vertices to be migrated and the connecting edges of the boundary vertices to be migrated in the subgraph data of the computing node are deleted.

9. A graph data partitioning device, characterized in that: The device comprises: A first determination module is used to determine at least one boundary vertex from the subgraph data of the computing node, wherein the boundary vertex has at least one neighbor vertex, there is a connecting edge between the boundary vertex and the neighbor vertex, at least one target neighbor vertex of at least one of the neighbor vertices exists in the respective subgraph data of at least one adjacent computing node, the boundary vertex or the neighbor vertex represents a data object, and the connecting edge represents a communication relationship between the data objects; A second determination module is used to determine a first expected benefit of migrating the boundary vertex to at least one of the adjacent computing nodes based on the benefit weights of the connecting edges between the boundary vertex and at least one of the neighboring vertices of the boundary vertex, wherein the benefit weight is determined according to the communication overhead between the adjacent computing node and the computing node, and the first expected benefit represents the value of the saved communication overhead; A third determination module, configured to determine a boundary vertex to be migrated and a target adjacent computing node based on a first expected benefit of migrating at least one of the boundary vertices to at least one of the adjacent computing nodes; A migration module, for migrating the boundary vertices to be migrated to the target adjacent computing node to re-divide the subgraph data of the computing node when it is determined that the boundary vertices to be migrated meet the migration conditions; the second determination module includes: a first cost determination submodule, a simulated migration submodule, a benefit determination submodule, a second cost determination submodule and a benefit determination submodule; The first cost determination submodule is used to determine the initial communication cost value of the boundary vertex based on the benefit weight of the connecting edge between the boundary vertex and at least one of the neighbor vertices of the boundary vertex; The simulated migration submodule is used to simulate the migration of the boundary vertices to at least one of the adjacent computing nodes, and obtain simulated boundary vertices corresponding to the at least one adjacent computing node; The benefit determination submodule is used to determine the simulated weighted benefit of the connecting edge between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex based on the first node identifier of the computing node and the second node identifier of at least one of the adjacent computing nodes; The second cost determination submodule is used to determine, for each of the simulated boundary vertices, a simulated communication cost value for migrating the boundary vertex to the adjacent computing node based on the simulated weighted benefits of the respective connecting edges between the simulated boundary vertex and at least one simulated neighbor vertex of the simulated boundary vertex; The benefit determination submodule is used to determine a first expected benefit of migrating the boundary vertex to each of the at least one adjacent computing nodes based on the initial communication cost and a simulated communication cost value of migrating the boundary vertex to each of the at least one adjacent computing nodes.

Images