Method, device, apparatus and program product for graph partitioning

The method partitions large graphs by assigning edges based on vertex degrees and thresholds, enhancing distributed graph processing efficiency and reducing processing time, addressing the scalability issues of existing graph data processing methods.

WO2024237803A1PCT designated stage expired Publication Date: 2024-11-21HUAWEI TECH CO LTD +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
PCT/RU2023/000146
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-18
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing methods for querying and processing large-scale graph data are time-consuming and do not scale well to web-scale applications, necessitating an efficient way to partition large graphs for distributed processing.

Method used

A method that partitions graphs by assigning edges to corresponding partitions based on vertex degrees and thresholds, using identity information and group indices to optimize partitioning, thereby accelerating graph partitioning tasks and improving performance for distributed graph processing.

Benefits of technology

This approach reduces end-to-end graph processing time and improves partitioning metrics such as balance and replication factor, while maintaining similar quality and reducing partition time complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure RU2023000146_21112024_PF_FP_ABST
    Figure RU2023000146_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure relate to a method, a device, an apparatus, and a program product for graph partitioning. The method comprises obtaining a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold. The method further comprises identifying a partition of the graph based on degrees of vertices of each of the plurality of edges and the at least one threshold. The method further comprises partitioning the graph by assigning each edge to a corresponding partition. The example embodiments of the present disclosure can accelerate graph partitioning tasks and performance for distributed graph processing. It also improves metrics for graph partitioning.
Need to check novelty before this filing date? Find Prior Art

Description

METHOD, DEVICE, APPARATUS AND PROGRAM PRODUCT FOR GRAPH PARTITIONINGFIELD

[0001] Embodiments of the present disclosure generally relate to the field of computer technology and in particular, to a method, a device, an apparatus, and a computer program product for graph partitioning.BACKGROUND

[0002] Large scale graph data, for example, where vertices representing documents, people, products, or other items are linked by edges, is generated in many application systems, for example, in social networking, information retrieval, video conferencing, product recommendation systems, knowledge management systems and others.

[0003] Existing approaches for querying graph data and carrying out computations using graph data in order to control application systems and others are typically time consuming and often do not scale up well to web-scale applications where massive amounts of data are involved. There is need for efficiently splitting large scale graph data into smaller graphs to be processed on computing devices.

[0004] The embodiments described below are not limited to implementations which solve any or all of the disadvantages of known ways of processing graph data.SUMMARY

[0005] In general, embodiments of the present disclosure provide a solution for partitioning graph.

[0006] In a first aspect, there is provided a method. The method comprises: obtaining a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold; identifying a partition of the graph based on degrees of vertices of each of the plurality of edges and the at least one threshold; and partitioning the graph by assigning each edge to a corresponding partition. In this way, it allows to accelerate graph partitioning tasks and performance for distributed graph processing. It also improves metrics for graph partitioning.

[0007] In some embodiments of the first aspect, identifying the partition of the graph may comprise determining a vertex of the edge based on the degrees of the vertices of the edge and the at least one threshold and determining the identity information of the partition at least based on the identity (ID) of the determined vertex. In this way, the vertices can be cut based on ID of vertices without randomness in partition.

[0008] In some embodiments of the first aspect, the at least one threshold may comprise a first threshold, and determining the vertex of the edge may comprises: in a case that one or more of two vertices of the edge has a degree greater than the first threshold, determining the vertex as a vertex of the edge having a lower degree than the other vertex. Cutting vertices with higher degree leads to decreasing of the replication factor. But it is less important for the pair of vertices having close degrees; they can be hashed not according to their degree. In this way, it can keep similar quality but gain better partition time.

[0009] In some embodiments of the first aspect, the at least one threshold may further comprise a second threshold lower than the first threshold and determining the vertex of the edge may comprises: in a case that none of the two vertices has a degree greater than the first threshold and that one of the vertices has a degree less than the second threshold and the other one has a degree equal to or greater than the second threshold, determining the vertex as a vertex of the edge having a higher degree than the other vertex. In a number of graph-processing algorithms the number of replications of vertices with a very low degree is not important due to their often being excluded from the active set. Therefore, replication of edges between low-degree vertex and middle-degree vertex by the middle-degree vertex improves the actual algorithm runtime.

[0010] In some embodiments of the first aspect, determining the vertex of the edge may comprises: in a case that both of the vertices have a degree less than the second threshold, or in a case that both of the vertices have a degree equal to or less than the first threshold but equal to or greater than the second threshold, determining the vertex as a vertex of the edge having a lower ID than the other vertex. For the edge having two medium-degree vertices, the partition is determined based on a lower ID, regardless of comparison of the degrees of the two vertices. In this way, it can improve the partition time.

[0011] In some embodiments of the first aspect, the partitioning parameters may further comprise a number of groups, each group comprising a part of partitions of the graph, and the method may further comprise identifying a group for the edge based on the number ofgroups. In this way, it allows to decrease inner degree. Since a lot of large inner degrees on partition can case slowdown of computations, it can reduce partition time.

[0012] In some embodiments of the first aspect, identifying the group for the edge may comprises determining a group index based on the number of groups and IDs of both of vertices of the edge. In this way, it may calculate group index efficiently and distribute edges into groups more randomly.

[0013] In some embodiments of the first aspect, the group index may is a remainder of a sum of the IDs of the vertices divided by the number of groups. In this way, it may calculate group index efficiently and distribute edges into groups more randomly.

[0014] In some embodiments of the first aspect, identifying the partition of the graph may comprise: determining a vertex of the edge based on the degrees of vertices of the edge and the at least one threshold; determining a local index of the partition within the group based on the ID of the vertex of the edge and a number of partitions in the group; and determining identity information of the partition based on the local index and the group index. In this way, it allows to decrease inner degree. A lot of large inner degrees on partition can case slowdown of computations.

[0015] In some embodiments of the first aspect, the partitioning parameters may further comprise a number of partitions of the graph, and the method may further comprises determining the number of partitions in the group by dividing the number of partitions of the graph by the number of groups. In this way, the edges are distributed in a balanced manner.

[0016] In some embodiments of the first aspect, the local index may be a remainder of the ID of the vertex divided by the number of partitions in the group. In this way, the local index can be calculated efficiently.(0017] In some embodiments of the first aspect, the identity information of the partition may be the local index of the partition plus the number of partitions in the group multiplied by the group index. In this way, the identity information of the partition can be calculated efficiently.

[0018] In some embodiments of the first aspect, the method may further comprise: performing graph processing by distributing the partitioned graph to multiple computing devices. In this way, the partitioned graph can be processed in parallel and end-to-end graph processing time is reduced.

[0019] In a second aspect, there is provided an electronic device. The electronic device comprises a processor and a memory coupled to the processor, wherein the memory has instructions stored therein which, when executed by the processor, cause the device to perform actions. The actions comprise: obtaining a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold; identifying a partition of the graph based on degrees of vertices of each of the plurality of edges and the at least one threshold; and partitioning the graph by assigning each edge to a corresponding partition.

[0020] In a third aspect, there is provided an apparatus. The apparatus comprises: means for obtaining a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold; means for identifying a partition of the graph based on degrees of vertices of each of the plurality of edges and the at least one threshold; and means for partitioning the graph by assigning each edge to a corresponding partition.

[0021] In a fourth aspect, there is provided a computer program product tangibly stored on a computer-readable medium and comprising machine-executable instructions, wherein the machine-executable instructions, when executed, cause a machine to perform the method according to the first aspect of the present disclosure.

[0022] In a fifth aspect, there is provided a computer-readable medium comprising machine-executable instructions, wherein the machine-executable instructions, when executed, cause a machine to perform the method according to the first aspect of the present disclosure.

[0023] It is to be understood that the Summary section is not intended to identify key or essential features of embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become easily comprehensible through the following description.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Some embodiments will now be described with reference to the accompanying drawings, where:

[0025] FIG. 1 illustrates a schematic diagram of an example environment in which a plurality of embodiments of the present disclosure can be implemented;

[0026] FIG. 2 illustrates a flowchart of an example method for partitioning a graph according to some embodiments of the present disclosure;

[0027] FIG. 3 illustrates a schematic diagram of an example workflow for determining a partition for an edge of a graph according to some embodiments of the present disclosure;

[0028] FIG. 4A illustrates a flowchart of a process for partitioning based on a lower degree according to some embodiments of the present disclosure;

[0029] FIG. 4B illustrates a flowchart of a process for partitioning based on a higher degree according to some embodiments of the present disclosure;

[0030] FIG. 4C illustrates a flowchart of a process for partitioning based on a vertex identity (ID) according to some embodiments of the present disclosure;

[0031] FIG. 5 illustrates a flowchart of a process for identifying a partition of a graph based on degrees of vertices of an edge in the graph according to some embodiments of the present disclosure;

[0032] FIG. 6 illustrates a flowchart of a process for calculating identity information of a partition for an edge in the graph according to some embodiments of the present disclosure;

[0033] FIG. 7A shows the experiments results according to some embodiments of the present disclosure.

[0034] FIG. 7B shows runtime improvement according to some embodiments of the present disclosure.

[0035] FIG. 8 illustrates a schematic block diagram of a device that can be used to implement embodiments of the present disclosure.

[0036] Throughout all the drawings, the same or similar reference numerals represent the same or similar elements.DETAILED DESCRIPTION

[0037] The principle of the present disclosure will now be described with reference to some embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and to help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. The disclosure described herein can be implemented in various manners other than the ones described below.

[0038] In the following description and claims, unless defined otherwise, all technical andscientific terms used herein have the same meaning as commonly understood by one of the ordinary skills in the art to which this disclosure belongs.

[0039] References in the present disclosure to “one embodiment,” “some embodiments,” “an embodiment,” and the like indicate that the embodiment described may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment includes the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with some embodiments, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0040] It shall be understood that although the terms “first” and “second” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0041] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting to embodiments. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “has”, “having”, “includes” and / or “including”, when used herein, specify the presence of stated features, elements, and / or components, etc., but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0042] For better understanding embodiments of the disclosure, provided below is definitions of related terms that are mentioned in the disclosure.Graph: the pair of sets (V, E), V is the set of vertices, E is the set of pairs of vertices (v, u) - edges; v is called source, u is called destination.Subgraph: of graph G=(V,E): graph G'=(V’, E’) such that V' <= V, E' £= E.Partitioning: distributing graph elements to the smaller set. There are two types: vertex partition and edge partition, the latter case is considered in this disclosure. For example, given an input value p - number of partitions - each edge receives integernumber from 0 to n-1.Degree: number of edges in the graph containing vertex v.Inner degree: vertex degree in subgraph on one partition.Replication factor: average number of vertex replicas that is equal to the summarized number of vertices in all subgraphs V(Ej) formed by partitions Ei, i =0, ... , n — 1, divided by number of vertices |V|:It shows how much vertices are duplicated.Balance: maximal number of edges on one partition divided by average number of edges on one partition; better balance should be close to 1.0:MSID: Maximal sum of inner degrees squared on one partition:

[0043] The graph partitioning problem is a computational task of graph elements’ distribution to smaller disjoint sets. Graph partitions can be performed as vertex partitions and edge partitions. In this disclosure, the latter case is considered. Input of a graph partitioning system is graph represented as the set of edges and the number of partitions n. The result of the system is partitioned graph, which means that every edge corresponds to the number from 0 to n- 1.

[0044] Graph partitioning is an important step in distributed computations. This task is essential for the analysis of the large graphs due to large runtime. In practice, graphs are distributed on several machines in some way in order to optimize storage but also it can significantly speed up the computation. The way graph is partitioned influences critically on the load balance, on the amount of interactions between the machines which is important for the runtime. Unfortunately, the ideal separation of the graph between machines without any data duplication is in the most cases impossible.

[0045] After edge partition is performed some vertices are “cut” - they appear on different partitions. When such vertices are affected in the algorithms there is interaction betweendifferent machines. To make algorithm performance faster, it makes sense to reduce number of repeating vertices; average number of vertex replicas is called replication factor. Also, in some algorithms and implementations it is important not to have a lot of high-degree vertices inside one partition, known as inner degree, because it can significantly slow down the process. The problem becomes even more complicated because of the non-planar nature of many real-world graphs which invalidates many naive “cluster-based” solutions. The power-law distribution of degrees of real-world graphs adds extra challenge since the high-degree and low-degree vertices behave differently during many algorithms’ runs.

[0046] There are a lot of partitioning methods in articles and frameworks which, being applied to some graph algorithm, show runtime acceleration. There are investigations concerning graph partition quality metrics such as balance and replication factor. Methods usually aimed to optimizing these metrics. But such improvements highly depend on graph structure and target application that needs to be accelerated. There is no universal method for all cases. In addition, partitioning methods which achieve better partition quality usually use more complex calculations and so could be time consuming. High partition time is essential in the task where partition is performed before each application run. That’s why complicated graph partitioning methods cannot be used in scenarios where partition is part of the whole computation pipeline. In addition, a lot of target real-world graphs have power-law distribution, and in such graphs high-degree and low-degree vertices behave differently in a number of algorithms.

[0047] In view of this, there is provided an approach for partitioning a graph. In this approach, a device obtains a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold. Each of the edges may be represented by a pair of vertices. The vertices have corresponding identities (IDs). For each edge, the device identifies a partition of the graph based on degrees of vertices of the edge and the at least one threshold. For example, the device may compare one or more of the degrees of vertices with the at least one threshold, and determine the partition based on the comparison and IDs of the vertices. The device then may assign the edge to the corresponding partition. By assigning all edges of the graph to corresponding partitions, the device generates partitioned graph.

[0048] Principles and embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Reference is first made to FIG. 1, which illustrates a schematic diagram of an example environment 100 in which a pluralityof embodiments of the present disclosure can be implemented.

[0049] The example environment 100 includes an application system 110 and a distributed computer system 120. The distributed computer system 120 includes a plurality of computing devices 101 and 102 which may be arranged at same or different geographic locations. In the distributed computer system 120, a computing device 101 is coupled to an application system 110 and configured to perform graph partitioning methods.

[0050] The computing device 110 takes as input the graph 106 comprising a plurality of edges each represented by a pair of vertices. The graph 106 may be a directed graph where some of the edges have directions. The graph 106 may be an undirected graph where none of the edges has a direction. In both cases, each of the edges may be represented by a source (src) vertex and destination (dst) vertex. The vertices of the graph 106 have identities. The identities of vertices may comprise one or more of numbers, strings, characters, and the like. The computing device 101 may receive or access the graph 106 as a data stream from a system 110 to be controlled or influenced using the graph. In some examples the graph may be stored (at the computing device 101 or elsewhere). The computing device 101 may partition the graph 106 into subgraphs 108 and distribute subgraphs 108 to computing devices 102 associated with graph analytics applications.

[0051] The application system 110 may comprise a plurality of applications 112 to be controlled or influenced using the graph 106. The application system 110 may be any computer implemented system which observes and / or records events that may be recorded as one or more connections between items represented as graph vertices. Depending particular applications, each graph vertex may represent at least one item, such as a person, product, document, email account, video conference account, or other item. Each graph edge represents a relationship between items represented by the vertices it connects.

[0052] For example, one of the applications 112 may be a social network system which stores user accounts and enables connections to be made between user accounts and / or communications to be sent between user accounts. In this case the graph 106 may be a social graph where operations are user interactions, defined through social engagements represented with graph edges. The applications 112 may be an information retrieval system which obtains addresses of documents or other items and information about links or other connections between the documents. The documents may be represented using graph vertices and the links between documents may be represented as edges. The applications112 may be a video conferencing system which represents video conferencing accounts associated with people by graph vertices. Communication events between users of the video conferencing system may be represented using graph edges. The application 112 may be a recommendation system which uses features of users and products in order to recommend products to users. For example, a graph vertex may represent a user and / or a product and the edges may represent events such as whether a user has purchased a given product. The system may be a knowledge management system. For example, the graph may be a knowledge graph where vertices represent people, places and items and edges represent relationships such as “likes”, “dislikes”, “lives”, or “works”.

[0053] Once the graph 106 is available it is possible to use the graph 106 to control the application system 110. For example, queries on the graph 106 may yield results which are used to improve information retrieval results, improve recommendations, suggest new friends in social networks and for other purposes. However, for many situations the scale of the graph 106 is so massive that it is not practical to carry out computations on the graph data in practical time scales.

[0054] As mentioned, the graph 106 may be partitioned into subgraphs 108 whereby the subgraphs 108 are smaller in size than the overall graph 106. Each subgraph may be stored at, or accessible to, one or more computing devices 102 at a data center or distributed at various locations and in communication with one another. Computations may be carried out at the individual computing devices 102 and the results aggregated so as to enable massive scale graph data to be accommodated. For example, queries on the graph data may be large and each query may be carried out on many of the clusters. Computation results 109 may be sent to the application system 110 and used to control that system.

[0055] In some embodiments, the computing devices 101 and 102 can be implemented as various user terminals or service terminals having the computing capability. The service terminals can be servers, large-scale computing devices, and the like provided by a variety of service providers. The user terminal may be, for example, a mobile terminal, a fixed terminal, or a portable terminal of any type, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a Personal Communication System (PCS) device, a personal navigation device, a Personal Digital Assistant (PDA), an audio / video player, a digital camera / video, a positioning device, a television receiver, a radio broadcast receiver, an electronic bookdevice, a gaining device or any other combinations thereof, including accessories and peripherals of these devices or any other combinations thereof. It can also be appreciated that the computing devices 101 and 102 can support any type of user-specific interfaces (such as “wearable” circuits and the like).

[0056] Alternatively, or in addition, the functionality of the computing devices 101 and 102 described herein can be performed, at least in part, by one or more hardware logic components of an electronic device. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program- specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), Graphics Processing Units (GPUs).

[0057] It is to be understood that the architecture and functions in the environment 100 are described for illustrative purposes only without suggesting any limitations. There may also be other devices, systems, or components that are not shown in the environment 100. Furthermore, embodiments of the present disclosure may also be applied to other environments having different structures and / or functions.

[0058] Reference is now made to FIG. 2 which illustrates a flowchart of an example method 200 for partitioning a graph according to some embodiments of the present disclosure. The example method 200 may be performed, for example, by the computing device 101 shown in FIG. 1. It should be understood that the method 200 may also include additional actions not shown, and the scope of the present disclosure is not limited in this regard. The method 200 is described in detail below in conjunction with the example environment 100 of FIG. 1.

[0059] At block 210, a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold is obtained. The computing device 101 may receive a graph from the application system 110 or read the graph stored locally or elsewhere. The graph comprises a plurality of edges where each edge may be represented by an ordered pair of vertices with IDs. The computing device 101 may obtain a set of partitioning parameters including a number of partitions and one or more thresholds to be compared with degrees of the vertices of the graph. In some embodiments, the partitioning parameters may include a number of groups. Each group is configured to include a subset of the partitions of the graph. For example, let an integer numParts be the number of partitions ofthe graph and an integer spread be the number of groups, then each group has numP arts / spread partitions (suppose numParts is divisible by spread).

[0060] At block 220, a partition of the graph based on degrees of vertices of each of the plurality of edges and the at least one threshold is identified. In some embodiments, the partitions of the graph may be represented by their identity information which could be indexes ranging from 0 to numParts - 1. The computing device 101 may determine a partition for each edge by calculating an index of a partition.

[0061] In some embodiments, the computing device 101 may calculate the identity information of partition based on degrees of vertices of the edge and the obtained one or more thresholds. In some embodiments, the computing device 101 may select one of the two vertices based on the comparison between the degrees of the two vertices and the thresholds, and use the ID of the selected vertex to calculate, for example, by a hash function, the identity information of the partition for the edge. The hash function takes IDs of a vertex as input and outputs identity information of partitions. As an example, the identity information of partitions may be indexes range from 0 to numParts - 1, and the hash function is expected to output such integers evenly. In some embodiments, the hash function may be modulo operation of number of partitions in consideration. In some embodiments, the identity information may be a remainder of the ID of the vertex divided by the number of partitions (numParts) or the number of partitions in the group (numParts / spread). In the latter case, the identity information may be referred as a local index in the group.

[0062] If the partitions are grouped (in case where the parameter spread indicating a number of groups is obtained), the identity information may be the local index of the partition plus the number of partitions in the group (e.g. numParts / spread ) multiplied by a group index. The group index indicates which group of partitions the edge belongs to. For example, the group index may be 0, 1 ..., spread-) . In some embodiments, the group index may be a remainder of a sum of the IDs of the vertices divided by the number of groups.

[0063] In some embodiments, the thresholds may include a first threshold and a second threshold lower than the first one. The first threshold (high threshold) defines a set of high-degree vertices having degrees greater than it. The second threshold (low threshold) defines a set of low-degree vertices having degrees less than it. Other vertices in the graph may be referred as medium vertices. Thus, the vertices in the graph may be divided intothree classifications: high-degree, medium-degree, and low-degree vertices.

[0064] The computing device 101 may calculate identity information of partition, in particular, the local index based on the classifications of vertices. In some embodiments, if the edge has one or more high-degree vertices, the computing device 101 may compare the degrees of the vertices and select a lower degree vertex, and then calculate the identity information of partition based on the ID of the selected vertex. In some embodiments, if the edge has one medium-degree vertex and one low-degree vertex, the computing device 101 may compare the degrees of the vertices and select a higher degree vertex, and then calculate the identity information of partition based on the ID of the selected vertex. In some embodiments, if both vertices of the edge are medium-degree vertices, or low-degree vertices, the computing device 101 may select either of the vertices (e.g. one with lower ID, or alternatively, with higher ID), and calculate the identity information of partition based on the ID of the selected vertex.

[0065] The computing device 101 may combine the group index and the local index to obtain the identity information of partition. In some embodiments, the identity information of the partition is the local index of the partition plus the number of partitions in the group multiplied by the group index, according to equation (1) below:where partition denotes the identity information of partition to which the edge is assigned, local denotes the local index within a group, numParts denotes partitions of graph in total, spread denotes a number of partition groups, whichGroup denotes group index, and [ ] is floor function.

[0066] At block 230, by assigning each edge to a corresponding partition, the graph is partitioned. The computing device 101 associates the identity information of the partition with the edge to include the edge in the corresponding partition. By processing each and every edge in the graph, the graph is partitioned into a set of subgraphs. In some embodiments, the computing device 101 may distribute the partitioned graph, i.e. the subgraphs to multiple computing devices 102. The computing devices 102 perform graph algorithms with the subgraphs.

[0067] The solution for partitioning a graph in accordance with embodiments of the disclosure has been described above with reference to Figs. 1 to 2. In comparison toconventional methods, it can achieve significant improvements on runtime and other partitioning metrics. In some embodiments, it does not need know degrees of all vertices, which is less important for the pair of vertices having close degrees, though requires some substantial partition time. It can keep similar quality but gain better partition time. Besides, in a number of graph-processing algorithms the number of replications of vertices with a very low degree is not important due to their often being excluded from the active set. Therefore, replication of edges between low-degree vertex and middle-degree vertex by the middle-degree vertex improves the actual algorithm runtime. In some embodiments, each edge is first distributed into one of the groups and only afterward distributed to one of the partitions within this group.

[0068] FIG. 3 illustrates a schematic diagram of an example workflow for determining a partition for an edge of a graph according to some embodiments of the present disclosure. FIG. 3 specifies the edge assigning according to such factors as having high- or low-degree vertices.

[0069] At block 301, an edge is represented as pair of source and destination (src and dst). More detailed, at block 303, if the edge has a high degree vertex, which means its degree is greater than the high threshold, then block 304 replicating by lower degree is used, which is described in FIG. 4A. If not and at block 305, it is determined that the edge has a low degree vertex, which means its degree is less than the low threshold, then block 306 replicating by higher degree is used, which is described in the FIG. 4B. If not, then block 307 replicating according to vertices Ids is used, which is described in FIG. 4C. Result of this replication will be used together with group index defined by vertices indices in block 302.

[0070] FIG. 4A illustrates a flowchart of a process for partitioning based on a lower degree according to some embodiments of the present disclosure. It is schematic depiction if this algorithm functions. Edge as a pair of vertices is the input 410. In some embodiments degrees of all vertices can be stored in memory but it is sufficient to store only degrees high-degree vertices, for the rest it is enough to remember flag that it is not high-degree. So, in the present flowchart, at block 412 it is written check if degree of the first vertex is less than degree of the second vertex not specifying if all exact values of these degrees are known. If yes, then at block 414 the hash of the edge is equal to the hash of the first vertex. If not, then at block 416 the hash of the edge is equal to the hash of the second vertex.

[0071] FIG. 4B illustrates a flowchart of a process for partitioning based on a higher degree according to some embodiments of the present disclosure. It is schematic depiction if this algorithm functions. Edge as a pair of vertices is the input 420. In the present flowchart at block 422 it is determined if degree of the first vertex is less than degree of the second vertex. If yes, then at block 424 the hash of the edge is equal to the hash of the second vertex. If not, then at block 426 the hash of the edge is equal to the hash of the first vertex.

[0072] FIG. 4C illustrates a flowchart of a process for partitioning based on a vertex identity (ID) according to some embodiments of the present disclosure. It is schematic depiction if this algorithm functions. Edge as a pair of vertices is the input 430. In the present flowchart at block 432 it is determined if the ID of the first vertex is less than the ID of the second vertex. If yes, then at block 434 the hash of the edge is equal to the hash of the first vertex. If not, then at block 436 the hash of the edge is equal to the hash of the second vertex.

[0073] FIG. 5 illustrates a flowchart of a process for identifying a partition of a graph based on degrees of vertices of an edge in the graph according to some embodiments of the present disclosure. The process 500 may be performed, for example, by the computing device 101 shown in FIG. 1. It should be understood that the process 500 may also include additional actions not shown, and the scope of the present disclosure is not limited in this regard. The process 500 is an example implementation of block 220 in FIG. 2.

[0074] To partition a graph by edges (vj, Vj), the computing device 101 first obtains a set of partitioning parameters including one or more of a number of partitions (numParts), a number of groups of partitions (spread), a high threshold (highThreshold) and a low threshold (lowThreshold).

[0075] At block 502, the computing device 101 determines a group index for the edge. The computing device 101 may calculate the number of partitions per group according to equation (2) and the global index according to equation (3) as below. numPartsl = numParts / spread (2)where Vj and Vj are the IDs of the vertices, numPartsl is number of partitions per group, whichGroup is the global index.

[0076] At block 504, the computing device 101 determines if there is any vertex has a degree greater than the high threshold. If yes, the process 500 proceeds to block 506 where the computing device 101 calculates local index based on ID of a vertex with a lower degree. The local index may be the remainder of the ID of the lower degree vertex divided by the number of partitions per group.

[0077] In this case, the process 500 proceeds to block 514 where the computing device 101 calculates the identity information of the partition based on the group index and the local index. For example, the identity information of the partition may be the local index plus the number of partitions in the group multiplied by the group index.

[0078] If it is determined that none of the vertices has a degree greater than the high threshold, the process 500 proceeds to block 508 where the computing device 101 determines if one degree of vertex is less than the low threshold and the other is not. If yes, the process 500 proceeds to block 510 where the computing device 101 calculates the local index based on ID of a vertex with a higher degree. Similarly, the local index may be the remainder of the ID of the higher degree vertex divided by the number of partitions per group.

[0079] For example, the computing device 101 may check if the degree of the destination vertex dj is less than lowThreshold and the degree of the source vertex dj is equal to or greater than lowThreshold. If yes, the computing device 101 calculates the local index based on the ID of the source vertex. If not, the computing device 101 may further check if the degree of the source vertex dj is less than lowThreshold and the degree of the destination vertex dj is equal to or greater than lowThreshold. If yes, the computing device 101 calculates the local index based on the ID of the destination vertex. Afterwards, the process 500 proceeds to block 514 where the computing device 101 obtains the ID of partition based on the group index and the local index.

[0080] If it is determined at block 508 that the condition is not satisfied, it means that the degrees of both of the vertices are either below the low threshold, or between the low threshold and the high threshold. In this case, the process 500 proceeds to block 512 where the computing device 101 calculates the local index based on the ID of vertex regardless of degree. For example, the local index may be calculated based on the either a lower or higher ID of the vertices. Afterwards, the process 500 proceeds to block 514 where the computing device 101 obtains the ID of partition based on the group index and the localindex.

[0081] The process 500 may be summarized by the example pseudo code below.edge hash(yt,Vj ) = vertex hash[v(, numPartsl) + whichGroup * numPartsl• else if dt< lowThreshold && dj > lowThreshold edge hash[yi,Vj ) = vertex hash(yj, numPartsl) + whichGroup * numPartsl• else

[0082] FIG. 6 illustrates a flowchart of a process for calculating identity information of a partition for an edge in the graph according to some embodiments of the present disclosure. The process 600 may be performed, for example, by the computing device 101 shown in FIG. 1. It should be understood that the process 500 may also include additional actions not shown, and the scope of the present disclosure is not limited in this regard. The process 500 is another example implementation of block 220 in FIG. 2.

[0083] At block 601, the computing device 101 takes an edge including a source vertex and a destination vertex as input. At block 602, the computing device 101 performs a modulo operation (src + dst) % spread and at block 610 obtains the group index of the edge.

[0084] At block 603, the computing device 101 determines if the degree of the source vertex is greater than the high threshold. If not, the computing device 101 at block 604 determines if the degree of the source vertex is less than the low threshold. If yes, the computing device 101 at block 605 determines if the degree of the destination vertex is less than the low threshold. If yes, which means both of degrees are less than the low threshold, the computing device 101 at block 606 further check if the destination vertex has an ID higher than the source vertex. If yes, at block 630 the local index of the edge is calculated as the hash of the destination vertex; otherwise at block 620 it is calculated as the hash of the source vertex.

[0085] Back to block 605, if it is determined that the degree of the destination vertex is not less than the low threshold, the process proceeds to block 630 where the local index of the edge is calculated as the hash of the destination vertex. Further back to block 604, if it is determined the degree of the source vertex is not less than the low threshold, the computing device 101 checks if the degree of the destination vertex is less than the low threshold or the degree of the source vertex is greater than the high threshold. If this condition is satisfied, the process goes to block 620 where the local index is calculated as the hash of the source vertex. If the condition at block 607 is not meet, the process goes to block 606 again, where a lower ID of vertex is found and used for calculation at one of blocks 620 or 630.

[0086] Return to block 603. If it is determined that the degree of the source vertex is greater than the high threshold. The process goes to block 608 where the computing device 101 checks if the degree of the destination vertex is higher than the high threshold. If yes, it means both degrees are greater than the high threshold. In this case, the computing device 101 at block 609 further determines if the degree of the destination vertex is equal to or higher than that of the source vertex. If yes at block 609, the process goes to block 630 where the local index of the edge is calculated as the hash of the destination vertex; otherwise at block 620 it is calculated as the hash of the source vertex.

[0087] Back to block 608, if it is determined that the degree of the destination vertex is not higher than the high threshold, the process goes to block 630 where the local index of the edge is calculated as the hash of the destination vertex, i.e. the vertex with a lower degree. At block 640, the computing device 101 may finally determine the identity information of partition by combining the group index from block 610 and the local index from block 620 or 630.

[0088] FIG. 7A shows the experiments results according to some embodiments of the present disclosure. FIG. 7A shows most importance metric in graph partitioning. The metrics are Balance, Replication Factor (RF), and Maximal sum of inner degrees squared (MSIDS) on one partition. The solution has significant improvements on these metrics over the related partition algorithms.

[0089] FIG. 7B shows runtime improvement according to some embodiments of the present disclosure. The algorithm runtime and end-to-end runtime are significantly improved.

[0090] In some example embodiments, an apparatus capable of performing the method 200 (for example, the computing device 101) may comprise means for performing the respective steps of the method 200. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module.

[0091] In some example embodiments, the apparatus comprises means for obtaining a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold; means for identifying a partition of the graph based on degrees of vertices of each of the plurality of edges and the at least one threshold; and means for partitioning the graph by assigning each edge to a corresponding partition.

[0092] In some example embodiments, the means for identifying a partition of the graph may comprise means for determining a vertex of the edge based on the degrees of the vertices of the edge and the at least one threshold and means for determining the identity information of the partition at least based on the identity (ID) of the determined vertex.

[0093] In some example embodiments, the at least one threshold may comprise a first threshold, and the means for determining the vertex of the edge may comprise means for, in a case that one or more of two vertices of the edge has a degree greater than the first threshold, determining the vertex as a vertex of the edge having a lower degree than the other vertex.

[0094] In some example embodiments, the at least one threshold may further comprise a second threshold lower than the first threshold, and the means for determining the vertex of the edge may comprise means for, in a case that none of the two vertices has a degree greater than the first threshold and that one of the vertices has a degree less than the second threshold and the other one has a degree equal to or greater than the second threshold, determining the vertex as a vertex of the edge having a higher degree than the other vertex.

[0095] In some example embodiments, the means for determining the vertex of the edge may comprise means for, in a case that both of the vertices have a degree less than the second threshold, or in a case that both of the vertices have a degree equal to or less than the first threshold but equal to or greater than the second threshold, determining the vertex as a vertex of the edge having a lower ID than the other vertex.

[0096] In some example embodiments, the partitioning parameters may further comprise a number of groups, each group comprising a part of partitions of the graph, and apparatus may further comprise means for identifying a group for the edge based on the number ofgroups.

[0097] In some example embodiments, the means for identifying a group for the edge based on the number of groups may comprise means for determining a group index based on the number of groups and IDs of both of vertices of the edge.

[0098] In some example embodiments, the group index may be a remainder of a sum of the IDs of the vertices divided by the number of groups.

[0099] In some example embodiments, means for identifying a partition of the graph may comprise means for determining a vertex of the edge based on the degrees of vertices of the edge and the at least one threshold; means for determining a local index of the partition within the group based on the ID of the vertex of the edge and a number of partitions in the group; and means for determining identity information of the partition based on the local index and the group index.

[0100] In some example embodiments, the partitioning parameters may further comprise a number of partitions of the graph, and the apparatus may further comprise means for determining the number of partitions in the group by dividing the number of partitions of the graph by the number of groups.

[0101] In some example embodiments, the local index may be a remainder of the ID of the vertex divided by the number of partitions in the group.

[0102] In some example embodiments, the identity information of the partition may be the local index of the partition plus the number of partitions in the group multiplied by the group index.

[0103] In some example embodiments, the apparatus may further comprise means for performing graph processing by distributing the partitioned graph to multiple computing devices.

[0104] FIG. 8 illustrates a schematic block diagram of a device 800 that may be used to implement embodiments of the present disclosure. The device 800 may be the device or apparatus described in the embodiments of the present disclosure, such as the computing device 110. As shown in FIG. 8, the device 800 includes a processor 801, which may execute various appropriate actions and processing in accordance with computer program instructions to perform the methods (e.g., the method 200) of the present disclosure. The computer program instructions may be stored in a Read-Only Memory (ROM) 802 or beloaded onto a Random Access Memory (RAM) 803 from a storage unit 808, for example. The processor 801, the ROM 802, and the RAM 803 are connected to each other via bus 804. An Input / Output (I / O) interface 805 is also connected to bus 804. In addition, although not shown in FIG. 8, the device 800 may also include a co-processor.

[0105] A plurality of components in device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard and a mouse; an output unit 807, such as various types of displays and speakers; the storage unit 808, such as a magnetic disk and an optical disc; and a communication unit 809, such as a network card, a modem, and a wireless communication transceiver. The communication unit 809 allows the device 800 to exchange information / data with other devices via a computer network, such as the Internet, and / or various telecommunication networks.

[0106] The various methods or processes described above may be performed by the processor 801. For example, in some embodiments, the method may be embodied as a computer software program that is tangibly included in a machine-readable medium, such as in the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 800 through the ROM 802 and / or communication unit 809. When the computer program is loaded into the RAM 803 and executed by the processor 801, one or more steps or actions of the methods or processes described herein may be executed.

[0107] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for performing various aspects of the present disclosure are loaded.

[0108] The computer-readable storage medium may be a tangible device that may retain and store instructions used by an instruction-executing device. For example, the computer-readable storage medium may be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer- readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory(CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical coding device, for example, a punch card or a raised structure in a groove with instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not to be interpreted as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber-optic cables), or electrical signals transmitted through electrical wires.

[0109] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0110] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages as well as conventional procedural programming languages. The computer-readable program instructions may be executed entirely on a user computer, partly on a user computer, as a stand-alone software package, partly on a user computer and partly on a remote computer, or entirely on a remote computer or a server. In a case where a remote computer is involved, the remote computer can be connected to a user computer through any kind of networks, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, connected through the Internet using an Internet service provider). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is customized by utilizing status information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructionsso as to implement various aspects of the present disclosure.

[0111] These computer-readable program instructions may be provided to a processing unit of a general-purpose computer, a special-purpose computer, or a further programmable data processing apparatus, thereby producing a machine, such that these instructions, when executed by the processing unit of the computer or the further programmable data processing apparatus, produce means for implementing functions / actions specified in one or more blocks in the flow charts and / or block diagrams. These computer-readable program instructions may also be stored in a computer-readable storage medium, and these instructions cause a computer, a programmable data processing apparatus, and / or other devices to operate in a specific manner; and thus the computer-readable medium having instructions stored includes an article of manufacture that includes instructions that implement various aspects of the functions / actions specified in one or more blocks in the flow charts and / or block diagrams.

[0112] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatuses, or other devices, such that a series of operating steps may be executed on the computer, the other programmable data processing apparatuses, or the other devices to produce a computer-implemented process, such that the instructions executed on the computer, the other programmable data processing apparatuses, or the other devices may implement the functions / actions specified in one or more blocks in the flow charts and / or block diagrams.

[0113] The flow charts and block diagrams in the drawings illustrate the architectures, functions, and operations of possible implementations of the devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow charts or block diagrams may represent a module, a program segment, or part of an instruction, and the module, program segment, or part of an instruction includes one or more executable instructions for implementing specified logical functions. In some alternative implementations, functions marked in the blocks may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive blocks may in fact be executed substantially concurrently, and sometimes they may also be executed in a reverse order, depending on the functions involved. It should be further noted that each block in the block diagrams and / or flow charts as well as a combination of blocks in the block diagrams and / or flow charts may be implemented using a dedicated hardware-based system that executes specified functions or actions, or using acombination of special hardware and computer instructions.

[0114] Various embodiments of the present disclosure have been described above. The foregoing description is illustrative rather than exhaustive, and is not limited to the disclosed various embodiments. Numerous modifications and alterations are apparent to persons of ordinary skill in the art without departing from the scope and spirit of the illustrated embodiments. The selection of terms as used herein is intended to best explain the principles and practical applications of the various embodiments or the technical improvements to technologies on the market, or to enable other persons of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: obtaining a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold; identifying a partition of the graph based on degrees of vertices of each of the plurality of edges and the at least one threshold; and partitioning the graph by assigning each edge to a corresponding partition.

2. The method of claim 1, wherein identifying the partition of the graph comprises: determining a vertex of the edge based on the degrees of the vertices of the edge and the at least one threshold; and determining the identity information of the partition at least based on the identity (ID) of the determined vertex.

3. The method of claim 2, wherein the at least one threshold comprises a first threshold, and determining the vertex of the edge comprises: in a case that one or more of two vertices of the edge have a degree greater than the first threshold, determining the vertex as a vertex of the edge having a lower degree than the other vertex.

4. The method of claim 3, wherein the at least one threshold further comprises a second threshold lower than the first threshold and determining the vertex of the edge comprises: in a case that none of the two vertices has a degree greater than the first threshold and that one of the vertices has a degree less than the second threshold and the other one has a degree equal to or greater than the second threshold, determining the vertex as a vertex of the edge having a higher degree than the other vertex.

5. The method of claim 4, wherein determining the vertex of the edge comprises: in a case that both of the vertices have a degree less than the second threshold, or in a case that both of the vertices have a degree equal to or less than the first threshold butequal to or greater than the second threshold, determining the vertex as a vertex of the edge having a lower ID than the other vertex.

6. The method of any of claims 1 to 5, wherein the partitioning parameters further comprise a number of groups, each group comprising a part of partitions of the graph, the method further comprising: identifying a group for the edge based on the number of groups.

7. The method of claim 6, wherein identifying the group for the edge comprises: determining a group index based on the number of groups and IDs of both of vertices of the edge.

8. The method of claim 7, wherein the group index is a remainder of a sum of the IDs of the vertices divided by the number of groups.

9. The method of any of claims 6 to 8, wherein identifying the partition of the graph comprises: determining a vertex of the edge based on the degrees of vertices of the edge and the at least one threshold; determining a local index of the partition within the group based on the ID of the vertex of the edge and a number of partitions in the group; and determining identity information of the partition based on the local index and the group index.

10. The method of claim 9, wherein the partitioning parameters further comprise a number of partitions of the graph, the method further comprises: determining the number of partitions in the group by dividing the number of partitions of the graph by the number of groups.

11. The method of claims 10 or 11, wherein the local index is a remainder of the ID of the vertex divided by the number of partitions in the group.

12. The method of any of claims 9 to 11, wherein the identity information of thepartition is the local index of the partition plus the number of partitions in the group multiplied by the group index.

13. The method of any of claims 1 to 12, further comprising: performing graph processing by distributing the partitioned graph to multiple computing devices.

14. An electronic device, comprising: a processor; and a memory coupled to the processor, wherein the memory has instructions stored therein which, when executed by the processor, cause the device to execute actions comprising: obtaining a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold; identifying a partition of the graph based on degrees of vertices of each of the plurality of edges and the at least one threshold; and partitioning the graph by assigning each edge to a corresponding partition.

15. An apparatus, comprising: means for obtaining a graph comprising a plurality of edges and partitioning parameters comprising at least one threshold; means for identifying a partition of the graph based on degrees of vertices of each of the plurality of edges and the at least one threshold; and means for partitioning the graph by assigning each edge to a corresponding partition.

16. A computer program product tangibly stored on a computer-readable medium and comprising machine-executable instructions, wherein the machine-executable instructions, when executed, cause a machine to perform the method according to any one of claims 1 to 13.

Citation Information

Patent Citations

  • Graph partitioning

    WO2017127106A1