Efficient graph partitioning method, device and equipment for large-scale graphs

Abstract

The present invention relates to an efficient graph partitioning method, device and equipment for large-scale graphs. According to the purpose of graph partitioning and in combination with the power law of large-scale graphs, mirror vertices are created for high-degree vertices on each computing node. After the high-degree vertices in the large-scale graph are marked, the high-degree vertices and the edges connected to the high-degree vertices are temporarily removed. Then, the remaining vertices and edges in the large-scale graph are first partitioned to each computing node to improve the partitioning efficiency. Then, the high-degree vertex-low-degree vertex edges are allocated to the computing nodes where the low-degree vertices are located. Then, the high-degree vertex-high-degree vertex edges are allocated to the computing nodes with fewer stored edges to balance the number of edges actually stored in each computing node. Finally, the high-degree vertices are allocated to the computing nodes with fewer stored vertices to balance the number of vertices actually stored in each computing node, thereby greatly reducing the partitioning time overhead and improving the partitioning quality, thereby achieving improved computing performance for processing large-scale graphs.

Description

Technical Field

[0001] The present invention belongs to the field of graph computing technology and relates to an efficient graph partitioning method, device and equipment for large-scale graphs. Background Art

[0002] A graph is an abstract data structure used to represent the relationship between objects. It is described using vertices and edges: vertices represent objects, and edges represent the relationship between objects. Data that can be abstracted into a graph is called graph data. Graph computing is the process of using a graph as a data model to express and solve problems. With the continuous development of intelligent computing, the scale of graph data continues to increase, far exceeding the upper limit of single-machine storage. As a result, distributed graph computing was born, which requires dividing the graph into several subgraphs (a subset of the graph) and distributing them to various computing nodes for computing.

[0003] In the process of graph partitioning, two vertices of some edges belong to two different computing nodes, which is called cut. In the actual graph computing process, the cross-node communication overhead (data interaction in graph computing is usually performed along the edge) will be much higher than the computing overhead. Therefore, the purpose of graph partitioning is to reduce the cross-node communication overhead as much as possible. Traditional graph partitioning techniques include 1D partitioning, 2D partitioning, metis (a software tool commonly used for graph partitioning) partitioning, hybrid partitioning, and graph preprocessing methods that use cluster partitioning subgraphs. Taking the graph preprocessing method as an example, it automatically divides the graph into multiple clusters (subgraphs) by deleting high-degree vertices, divides the clusters, and then restores the originally deleted vertices and edges at the same time; the idea of ​​restoration is to reduce mirror vertices as much as possible. However, traditional graph partitioning techniques still have the technical problem of low computing performance when processing large-scale graphs. Summary of the invention

[0004] In response to the problems existing in the above-mentioned traditional methods, the present invention proposes an efficient graph partitioning method for large-scale graphs, an efficient graph partitioning device for large-scale graphs, and a computer device, which can greatly improve the computing performance of processing large-scale graphs.

[0005] In order to achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] On the one hand, an efficient graph partitioning method for large-scale graphs is provided, comprising the steps of:

[0007] Get a large-scale graph to be processed;

[0008] Traverse the entire large-scale graph and mark all high-degree vertices in the graph, temporarily remove high-degree vertices and edges connected to high-degree vertices;

[0009] Divide the remaining vertices and edges in the large-scale graph into various computing nodes; each computing node creates a mirror vertex for the high-degree vertex;

[0010] Assign high-degree vertex-low-degree vertex edges to the computational nodes where the low-degree vertices are located;

[0011] Assign high-degree vertex-high-degree vertex edges to computation nodes with fewer stored edges;

[0012] Assign high-degree vertices to compute nodes that have fewer stored vertices.

[0013] In one of the embodiments, in the step of partitioning the remaining vertices and edges in the large-scale graph to the respective computing nodes, TopoX partitioning is used for partitioning processing.

[0014] In one of the embodiments, in the process of traversing the entire large-scale graph and marking all high-degree vertices in the graph, the product of the average degree in the graph and a set additional parameter is selected as a threshold; the threshold is used to mark high-degree vertices.

[0015] On the other hand, an efficient graph partitioning device for large-scale graphs is also provided, including:

[0016] A graph acquisition module, used to acquire large-scale graphs to be processed;

[0017] The vertex marking module is used to traverse the entire large-scale graph and mark all high-degree vertices in the graph, temporarily removing high-degree vertices and edges connected to high-degree vertices;

[0018] The low-degree partitioning module is used to partition the remaining vertices and edges in the large-scale graph to each computing node; each computing node creates a mirror vertex for the high-degree vertex;

[0019] The high-low edge partitioning module is used to allocate high-degree vertex-low-degree vertex edges to the computing nodes where the low-degree vertices are located;

[0020] A high-edge partitioning module is used to allocate high-vertex-high-vertex edges to computing nodes with fewer stored edges;

[0021] The high vertex partitioning module is used to distribute high-degree vertices to computing nodes with fewer stored vertices.

[0022] In one embodiment, the low-degree partitioning module uses TopoX partitioning to partition the remaining vertices and edges in the large-scale graph to each computing node.

[0023] In one embodiment, the vertex marking module selects the product of the average degree in the graph and a set additional parameter as a threshold when traversing the entire large-scale graph and marking all high-degree vertices in the graph; the threshold is used to mark high-degree vertices.

[0024] On the other hand, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned efficient graph partitioning method for large-scale graphs when executing the computer program.

[0025] One of the above technical solutions has the following advantages and beneficial effects:

[0026] The above-mentioned efficient graph partitioning method, device and equipment for large-scale graphs, according to the purpose of graph partitioning and combined with the power-law of large-scale graphs, creates mirror vertices for high-degree vertices on each computing node to reduce the number of maximum cuts from the number of neighbors of the vertex to the number of computing nodes in the computing cluster. After first marking the high-degree vertices in the large-scale graph, the high-degree vertices and the edges connected to the high-degree vertices are temporarily removed to sparse the graph as much as possible (rather than obtaining grouped clusters). In order to accelerate the next step of partitioning, the remaining vertices and edges in the large-scale graph are first divided into each computing node, and the mirror vertex allocation of the high-degree vertices is not considered for the time being, so as to optimize First, ensure the balance of edges and vertices and improve the partitioning efficiency. Then, assign the high-degree vertex-low-degree vertex edges to the computing nodes where the low-degree vertices are located. Then assign the high-degree vertex-high-degree vertex edges to the computing nodes with fewer stored edges to balance the number of edges actually stored in each computing node. Finally, assign the high-degree vertices to the computing nodes with fewer stored vertices to balance the number of vertices actually stored in each computing node. The number of maximum cuts can be reduced from the number of neighbors of the vertex to the number of computing nodes in the computing cluster. The graph is more balanced and the computational efficiency is higher. Ultimately, the partitioning time overhead is greatly reduced and the partitioning quality is improved, achieving improved computing performance for processing large-scale graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 A schematic diagram of a process flow of an efficient graph partitioning method for large-scale graphs in one embodiment;

[0029] Figure 2 A diagram of the vertices and edges of a graph for one of the datasets;

[0030] Figure 3 A schematic diagram of a process of temporarily removing high-degree vertices in one embodiment;

[0031] Figure 4 A schematic diagram of a rough division process in an embodiment;

[0032] Figure 5 A schematic diagram of a process of allocating high-degree vertex-low-degree vertex edges in one embodiment;

[0033] Figure 6 A schematic diagram of a process of allocating high-degree vertex-high-degree vertex edges in one embodiment;

[0034] Figure 7 A schematic diagram of a process for allocating high-degree vertices in one embodiment;

[0035] Figure 8 It is a structural block diagram of an efficient graph partitioning device for large-scale graphs in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0037] It should be noted that the reference to "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The presentation of this phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It will be appreciated by those skilled in the art that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the specification of the present invention and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0038] The following will describe the implementation of the present invention in detail with reference to the accompanying drawings in the embodiment diagram of the present invention.

[0039] There are many graph partitioning methods, such as 1D partitioning, 2D partitioning, metis (software tool commonly used for graph partitioning) partitioning and hybrid partitioning. 1D partitioning and 2D partitioning belong to hash partitioning. 1D partitioning refers to directly assigning edges to corresponding computing nodes according to the hash value of the starting vertex or destination vertex ID (identity identifier). 2D partitioning, on the other hand, is to form all computing nodes into a two-dimensional array (that is, a cluster with M computing nodes is arranged into an m×n array, where m and n are both positive integers), and hash the starting vertex and destination vertex of an edge respectively to obtain the specific nodes (m0, n0) to which the edge should be assigned.

[0040] Metis partitioning is a classic partitioning method, and its calculation process can be roughly divided into three parts: coarsening, rough partitioning, and refinement. Its disadvantage is that the larger the scale of the graph and the number of subgraphs to be divided, the lower the calculation efficiency and the lower the partitioning quality. Therefore, it is almost impossible to directly use metis partitioning when processing large-scale graphs, which is very time-consuming. There are also many hybrid partitioning methods, such as the representative TopoX (graph partitioning for distributed graph computing) partitioning method.

[0041] However, 1D partitioning and 2D partitioning do not consider the topological relationship within the graph, which will generate a large number of cuts and lead to poor computing performance. Although the overall partitioning quality of metis partitioning is better than that of 1D partitioning and 2D partitioning, when processing large-scale graphs, the time cost is too high and the partitioning quality is too low, which makes it unable to meet the actual application requirements of large-scale graph partitioning. Hybrid partitioning lacks parallelism, resulting in high partitioning time cost. Therefore, a new solution will be given below to meet the application requirements of large-scale graph partitioning.

[0042] In one embodiment, Figure 1 As shown, an efficient graph partitioning method for large-scale graphs is provided, which may include the following processing steps S10 to S20:

[0043] S10, obtaining a large-scale graph to be processed;

[0044] S12, traverse the entire large-scale graph and mark all high-degree vertices in the graph, temporarily remove high-degree vertices and edges connected to high-degree vertices;

[0045] S14, dividing the remaining vertices and edges in the large-scale graph into various computing nodes; each computing node creates a mirror vertex for the high-degree vertex;

[0046] S16, assigning high-degree vertex-low-degree vertex edges to the computing nodes where the low-degree vertices are located;

[0047] S18, assigning high-degree vertex-high-degree vertex edges to computing nodes with fewer stored edges;

[0048] S20, allocating high-degree vertices to computing nodes that have fewer stored vertices.

[0049] It can be understood that a power-law graph refers to a graph in which the degree of vertices in the graph data is unevenly distributed, such as a social network, web network, road network, or other similar large-scale network graph. In a graph, the degree represents the number of edges connecting the vertex, and also represents the number of its neighbors. The higher the vertex degree, the higher the computational load during the calculation process, and the higher the storage overhead. Graph calculations are usually performed along the edges. Some concepts are as follows: BFS (Breath First Search) is an important algorithm in graph theory, which transmits activation messages along the edges; CC (Connected Component) is used to count the number of connected components in the current graph set, and it searches for connected components along the edges; PageRank (an algorithm for evaluating network weights based on random walks) aggregates the weights of neighbors along the edges; GNN (graph neural network) aggregates the embeddings of neighbors along the edges.

[0050] In distributed graph computing, vertices and edges are assigned to different computing nodes. When the vertices at both ends of an edge belong to two different computing nodes, cross-node communication will occur. Usually, the communication overhead is much greater than the computing overhead. The purpose of graph partitioning is to minimize cross-node communication overhead. The higher the degree, the more neighbors there are, and the more difficult it is to store all neighbors in a few computing nodes. Aggregating the neighbors of highly-determined vertices as much as possible will lead to a bottleneck in the amount of computing on a single node.

[0051] In this embodiment, high-degree vertices and low-degree vertices are processed separately (degree represents the number of adjacent vertices connected to the vertex), high-degree vertices are dispersed as much as possible, and low-degree vertices are aggregated as much as possible. The edges in the entire graph can be divided into three categories: high-degree vertex-high-degree vertex, high-degree vertex-low-degree vertex, and low-degree vertex-low-degree vertex. The design concept is as follows: The purpose of graph partitioning is to reduce the number of cuts. For high-degree vertices, if the number of their cuts is reduced as much as possible, all their neighbors need to be gathered in a few computing nodes as much as possible, which will cause computing and storage bottlenecks in these computing nodes. This is a very costly solution. Therefore, from the perspective of balancing the amount of computing, no matter how the high-degree vertices are divided, it is difficult to avoid a large number of cuts. However, by setting mirror vertices on each computing node, the maximum number of cuts can be reduced from the number of neighbors of the vertex to the number of computing nodes in the computing cluster.

[0052] For example, in a data set, the largest user has 200 million fans. If his fans are distributed to a cluster of 100 computing nodes, 198 million cuts will be generated in an even distribution. By creating a mirror vertex for each high-degree vertex on each computing node, it is only necessary to synchronize the user status of the 200 million fans to the other 99 cut computing nodes, and the total number of cuts will be reduced to 99 (the cut at this time is vertex-mirror vertex). Therefore, for high-degree vertices, it is assumed that they have mirror vertices on all computing nodes, that is, the maximum cut between them and their neighbors is fixed as (M-1, M is the number of cluster nodes). Figure 2 As shown, the circle numbers represent the vertices, the vertices in the shaded part represent high-degree vertices, the vertices in the other transparent parts represent low-degree vertices, and the part framed by the square is a schematic diagram of a pair of high-degree vertices and high-degree vertices, and the same applies to the following text.

[0053] For low-degree vertices, since they have fewer neighbors, they can be divided into the same computing node with a small amount of cuts. Especially for high-degree vertex-low-degree vertex edges, since high-degree vertices have mirror vertices in all computing nodes, they only need to be allocated following low-degree vertices. Therefore, we first consider the division of low-degree vertices to ensure that there are not too many cuts on low-degree vertices.

[0054] Specifically, first traverse the entire large-scale graph and mark all high-degree vertices in the graph to temporarily remove high-degree vertices and edges connected to high-degree vertices. Usually, a threshold can be set for marking high-degree vertices. For example, the average degree in the graph can be selected as the threshold to achieve accurate high-degree vertex partitioning. At this point, only low-degree vertices and low-degree vertex-low-degree vertex type edges remain in the remaining graph, such as Figure 3 shown.

[0055] Then, the large-scale graph is roughly divided. Usually, metis division or other heuristic division methods can be used to initially divide the remaining vertices and edges in the large-scale graph into various computing nodes, such as Figure 4 As shown in Figure 2, large-scale graphs usually have power-law properties, and edges connected by highly vertices account for the vast majority of the graph. In this case, the partitioning efficiency is high and the quality is good.

[0056] Then assign high-degree vertex-low-degree vertex edges to distribute these edges to the computation nodes where the low-degree vertices are located, for example Figure 5 Then, we allocate high-degree vertex-high-degree vertex edges. At this time, we can count the number of edges actually stored on each computing node and allocate high-degree vertex-high-degree vertex edges to computing nodes with fewer stored edges to balance the number of edges actually stored on each computing node. For example, Figure 6 As shown in Figure 2. Since high-degree vertices appear in all computational nodes, the number of cuts will not be affected by where high-degree vertex-high-degree vertex edges are allocated.

[0057] Finally, assign high-degree vertices. At this point, you can count the number of vertices actually stored on each computing node, and assign high-degree vertices to computing nodes with fewer stored vertices to balance the number of vertices actually stored on each computing node, such as Figure 7 As shown in Figure 2. Since high-degree vertices appear in all computational nodes, where high-degree vertices are assigned will not affect the number of cuts.

[0058] The above-mentioned efficient graph partitioning method for large-scale graphs, according to the purpose of graph partitioning and combined with the power-law of large-scale graphs, creates mirror vertices for high-degree vertices on each computing node, and first marks the high-degree vertices in the large-scale graph, and then temporarily removes the high-degree vertices and the edges connected to the high-degree vertices. Then, the remaining vertices and edges in the large-scale graph are first divided to each computing node to improve the partitioning efficiency. Then, the high-degree vertex-low-degree vertex edges are assigned to the computing nodes where the low-degree vertices are located, and then the high-degree vertex-high-degree vertex edges are assigned to the computing nodes with fewer stored edges to balance the number of edges actually stored in each computing node. Finally, the high-degree vertices are assigned to the computing nodes with fewer stored vertices to balance the number of vertices actually stored in each computing node. The number of maximum cuts can be reduced from the number of neighbors of the vertex to the number of computing nodes in the computing cluster, and finally the partitioning time overhead is greatly reduced and the partitioning quality is improved, thereby achieving improved computing performance in processing large-scale graphs.

[0059] In one embodiment, in the step of partitioning the remaining vertices and edges in the large-scale graph to the respective computing nodes, TopoX partitioning is used for partitioning processing.

[0060] It can be understood that in this embodiment, the conventional TopoX partitioning is used for coarse partitioning, which can further improve the partitioning efficiency and enhance the partitioning quality.

[0061] In one embodiment, in the process of traversing the entire large-scale graph and marking all high-degree vertices in the graph, the product of the average degree in the graph and a set additional parameter is selected as a threshold; the threshold is used to mark high-degree vertices.

[0062] It can be understood that in this embodiment, the product of the average degree in the graph and the additional parameter a can also be selected as the threshold to mark the high-degree vertices. By setting the additional parameter a (which can generally be configured to be 0.2, which can be obtained through experiments), the threshold can be flexibly set, allowing the user to adjust it according to the actual situation, so that the above-mentioned traversal and marking processing can better adapt to the processing of different large-scale graphs, and achieve more accurate and flexible high-degree vertex marking.

[0063] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0064] In one embodiment, Figure 8 As shown, an efficient graph partitioning device 100 for large-scale graphs is also provided, including a graph acquisition module 11, a vertex marking module 13, a low-degree partitioning module 15, a high-low edge partitioning module 17, a high-edge partitioning module 19 and a high-vertex partitioning module 21. Among them, the graph acquisition module 11 is used to obtain a large-scale graph to be processed. The vertex marking module 13 is used to traverse the entire large-scale graph and mark all high-degree vertices in the graph, temporarily remove the high-degree vertices and the edges connected to the high-degree vertices. The low-degree partitioning module 15 is used to partition the remaining vertices and edges in the large-scale graph to each computing node; each computing node creates a mirror vertex for the high-degree vertex. The high-low edge partitioning module 17 is used to assign the high-degree vertex-low-degree vertex edge to the computing node where the low-degree vertex is located. The high-edge partitioning module 19 is used to assign the high-degree vertex-high-degree vertex edge to the computing node with fewer stored edges. The high-vertex partitioning module 21 is used to assign the high-degree vertex to the computing node with fewer stored vertices.

[0065] The above-mentioned efficient graph partitioning device 100 for large-scale graphs, according to the purpose of graph partitioning and combined with the power-law of large-scale graphs, creates mirror vertices for high-degree vertices on each computing node, and first marks the high-degree vertices in the large-scale graph, and then temporarily removes the high-degree vertices and the edges connected to the high-degree vertices. Then, the remaining vertices and edges in the large-scale graph are first divided to each computing node to improve the partitioning efficiency. Then, the high-degree vertex-low-degree vertex edges are allocated to the computing nodes where the low-degree vertices are located, and then the high-degree vertex-high-degree vertex edges are allocated to the computing nodes with fewer stored edges to balance the number of edges actually stored in each computing node. Finally, the high-degree vertices are allocated to the computing nodes with fewer stored vertices to balance the number of vertices actually stored in each computing node. The number of maximum cuts can be reduced from the number of neighbors of the vertex to the number of computing nodes in the computing cluster, and finally the partitioning time overhead is greatly reduced and the partitioning quality is improved, thereby achieving improved computing performance in processing large-scale graphs.

[0066] In one embodiment, the low-degree partitioning module 15 uses TopoX partitioning to partition the remaining vertices and edges in the large-scale graph to various computing nodes.

[0067] In one embodiment, the vertex marking module 13 selects the product of the average degree in the graph and a set additional parameter as a threshold when traversing the entire large-scale graph and marking all high-degree vertices in the graph; the threshold is used to mark high-degree vertices.

[0068] For the specific limitations of the efficient graph partitioning device 100 for large-scale graphs, please refer to the limitations of the efficient graph partitioning method for large-scale graphs above, which will not be repeated here. The various modules in the above-mentioned efficient graph partitioning device 100 for large-scale graphs can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0069] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following processing steps when executing the computer program: obtaining a large-scale graph to be processed; traversing the entire large-scale graph and marking all high-degree vertices in the graph, temporarily removing the high-degree vertices and the edges connected to the high-degree vertices; dividing the remaining vertices and edges in the large-scale graph into various computing nodes; each computing node creates a mirror vertex for the high-degree vertex; assigning the high-degree vertex-low-degree vertex edge to the computing node where the low-degree vertex is located; assigning the high-degree vertex-high-degree vertex edge to the computing node with fewer stored edges; assigning the high-degree vertex to the computing node with fewer stored vertices.

[0070] It can be understood that the above-mentioned computer device, in addition to the memory and processor mentioned above, may also include other software and hardware components not listed in this specification. The specific details can be determined according to the model of the specific graphics computing device in different application scenarios, and this specification will not list them one by one in detail.

[0071] In one embodiment, when the processor executes the computer program, it can also implement the steps or sub-steps added in each embodiment of the above-mentioned efficient graph partitioning method for large-scale graphs.

[0072] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following processing steps are implemented: obtaining a large-scale graph to be processed; traversing the entire large-scale graph and marking all high-degree vertices in the graph, temporarily removing the high-degree vertices and the edges connected to the high-degree vertices; dividing the remaining vertices and edges in the large-scale graph into various computing nodes; each computing node creates a mirror vertex for the high-degree vertex; assigning the high-degree vertex-low-degree vertex edge to the computing node where the low-degree vertex is located; assigning the high-degree vertex-high-degree vertex edge to the computing node with fewer stored edges; assigning the high-degree vertices to the computing nodes with fewer stored vertices.

[0073] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (RambusDRAM, referred to as RDRAM) and interface dynamic random access memory (DRDRAM).

[0074] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection of the invention. It should be pointed out that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. An efficient graph partitioning method for large-scale graphs, characterized in that: Includes steps: Get a large-scale graph to be processed; Traversing the entire large-scale graph and marking all high-degree vertices in the graph, temporarily removing the high-degree vertices and edges connected to the high-degree vertices; Divide the remaining vertices and edges in the large-scale graph into various computing nodes; each computing node creates a mirror vertex for the high-degree vertex; Assign high-degree vertex-low-degree vertex edges to the computational nodes where low-degree vertices are located; degree represents the number of adjacent vertices connected to a vertex, the high-degree vertex is a vertex with a large number of adjacent vertices connected to it, and the low-degree vertex is a vertex with a small number of adjacent vertices connected to it; Assign high-degree vertex-high-degree vertex edges to computation nodes with fewer stored edges; The high-degree vertices are distributed to computing nodes that have fewer stored vertices.

2. The efficient graph partitioning method for large-scale graphs according to claim 1, characterized in that: In the step of dividing the remaining vertices and edges in the large-scale graph into various computing nodes, TopoX partitioning is used for partitioning processing.

3. The efficient graph partitioning method for large-scale graphs according to claim 1 or 2, characterized in that: In the process of traversing the entire large-scale graph and marking all high-degree vertices in the graph, the product of the average degree in the graph and the set additional parameter is selected as the threshold; the threshold is used to mark the high-degree vertices.

4. An efficient graph partitioning device for large-scale graphs, characterized in that: include: A graph acquisition module, used to acquire large-scale graphs to be processed; A vertex marking module, used to traverse the entire large-scale graph and mark all high-degree vertices in the graph, and temporarily remove the high-degree vertices and the edges connected to the high-degree vertices; A low-degree partitioning module is used to partition the remaining vertices and edges in the large-scale graph to each computing node; each computing node creates a mirror vertex for the high-degree vertex; A high-low edge partitioning module is used to allocate high-degree vertex-low-degree vertex edges to computing nodes where low-degree vertices are located; the degree represents the number of adjacent vertices connected to a vertex, the high-degree vertex is a vertex with a large number of adjacent vertices connected to it, and the low-degree vertex is a vertex with a small number of adjacent vertices connected to it; A high-edge partitioning module is used to allocate high-vertex-high-vertex edges to computing nodes with fewer stored edges; The high vertex partitioning module is used to distribute the high-degree vertices to computing nodes with fewer stored vertices.

5. The efficient graph partitioning device for large-scale graphs according to claim 4, characterized in that: The low-degree partitioning module uses TopoX partitioning to partition the remaining vertices and edges in the large-scale graph to various computing nodes.

6. The efficient graph partitioning device for large-scale graphs according to claim 4 or 5, characterized in that: The vertex marking module selects the product of the average degree in the graph and the set additional parameter as the threshold value in the process of traversing the entire large-scale graph and marking all high-degree vertices in the graph; The threshold is used to mark high-degree vertices.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the processor implements the steps of the efficient graph partitioning method for large-scale graphs as described in any one of claims 1 to 3.

Images