A graph data partitioning method based on a cache mechanism

Abstract

The application discloses a graph data partitioning method based on a cache mechanism, comprising the following steps: obtaining graph data, obtaining the total number of graph vertices, the neighbor list of each vertex, the initial vertex partition set and the initial partition number according to the graph data; calculating the upper limit of the load; calculating the correlation score of the vertex in each vertex set which does not reach the upper limit of the load according to the vertex and the neighbor list of the vertex; judging whether the vertex meets the cache condition, if yes, partitioning according to the correlation score; if not, the vertex is partitioned into the cache module, when the cache module reaches the maximum capacity limit, the cache module returns the vertex and the neighbor list which are most suitable for the current partitioning, and the process returns; judging whether all vertices have been partitioned, if yes, the final graph data partition set is obtained; if not, the process returns. The application improves the partitioning precision of the Fennel algorithm under the condition that the size of the graph data is too large.

Description

Technical Field

[0001] This invention relates to the field of graph computing technology, and more specifically, to a graph data partitioning method based on a caching mechanism. Background Technology

[0002] Graph partitioning is a preliminary step in distributed graph processing. In recent years, to address the efficiency issues in partitioning large-scale graph data caused by the continuous expansion of data size, scholars have proposed concise and efficient streaming graph partitioning algorithms. However, the partitioning results are still significantly lower than those of traditional offline algorithms. There is still considerable room for research on how to improve partitioning performance while maintaining the conciseness and efficiency of streaming partitioning.

[0003] Fennel's algorithm transforms the graph flow partitioning problem into a modularity maximization problem based on the data flow. Its core idea is to maximize the number of adjacent vertices and minimize the number of non-adjacent vertices, thus deriving the relevant objective function formula. Although computationally complex, it achieves partitioning results with a relatively low edge cut rate compared to other vertex partitioning algorithms under load balancing, making it one of the best-performing algorithms to date. Its drawbacks include the need to obtain the total number of vertices in the graph beforehand, its inability to handle dynamic graphs, and its failure to fully utilize the local structural information of the graph data.

[0004] Existing technology discloses a graph partitioning method, apparatus, and computer-readable storage medium. The method includes pre-determining graph partitioning rules for proportionally distributing graph data to each computing node in a parallel computing system based on a preset metric mechanism, according to the computing performance of each node. During graph partitioning, the data reading range of each loading thread is first determined based on the number of graph vertices and the number of loading threads. Simultaneously, multiple loading threads are invoked to read the graph data to be partitioned in parallel, vertice by vertex, and the vertex data read by each loading thread is passed to the corresponding partitioning thread. Each partitioning thread, based on the currently partitioned graph data and according to the graph partitioning rules, determines the subgraph to which each vertex data passed in within its thread belongs. Finally, based on the correspondence between subgraphs and computing nodes, the graph vertex data is partitioned to the corresponding computing nodes. This method can efficiently and accurately partition a large graph into a series of subgraphs with vertices or edges satisfying arbitrary proportional relationships. However, this method still fails to fully utilize the local structural information of the graph data when the graph data is too large, resulting in poor partitioning performance. Summary of the Invention

[0005] This invention provides a graph data partitioning method based on a caching mechanism, which improves the partitioning accuracy of the Fennel algorithm when the graph data is too large.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A graph data partitioning method based on a caching mechanism includes the following steps:

[0008] S1: Obtain graph data. Based on the graph data, obtain the total number of vertices in the graph, the neighbor list of each vertex, the initial vertex partition set, and the initial number of partitions. The vertex partition set includes several vertex sets.

[0009] S2: Calculate the load limit based on the total number of vertices and the initial number of partitions in the graph data;

[0010] S3: Calculate the associativity score of the vertex in each set of vertices that has not yet reached the load limit, based on the vertex and its neighbor list;

[0011] S4: Determine whether the vertex meets the caching conditions. If it does, no caching is performed, and the vertex is directly divided according to the correlation score. If it does not meet the conditions, the vertex is divided into the cache module. When the cache module reaches the maximum capacity limit, the cache module returns the list of vertices and their neighbors that are most suitable for the current division and returns to step S3.

[0012] S5: Determine whether all vertices have been partitioned. If yes, obtain the final graph data partition set; otherwise, return to step S3.

[0013] Preferably, the graph data obtained in step S1 is an edge set array.

[0014] The graph data partitioning method based on the caching mechanism is characterized in that, in step S2, the load limit is calculated based on the total number of vertices and the initial number of partitions in the graph data, specifically as follows:

[0015]

[0016] In the formula, Maximum load, The total number of vertices in the graph data. This is the initial number of partitions. To adjust the parameters.

[0017] Preferred, Set to 0.001~0.005.

[0018] Preferably, in step S3, the correlation score of a vertex in each set of vertices that has not yet reached the load limit is calculated based on the vertex and its neighbor list. Specifically:

[0019]

[0020]

[0021] In the formula, Represents vertices The current load limit has not been reached. vertex set The correlation score on the surface Including scoring items and penalty items , Represents vertices The list of neighbors, The penalty coefficient is... This is the initial number of partitions. This indicates the number of edges in the graph data. The total number of vertices in the graph data. For cached parameters.

[0022] Preferably, in step S3, when calculating the correlation score, a temporary space is created, which records the current load limit before it is reached. The set of vertices and the score of the set of vertices that have not yet reached the load limit. .

[0023] Preferably, if the caching conditions are met in step S4, no caching processing is performed, and the partitioning is carried out directly, specifically as follows:

[0024] like:

[0025]

[0026] In the formula, Represents vertices In the neighbor set, it is assigned to the vertex set. The number of vertices;

[0027] In this case, caching is unnecessary; simply determine the relevance score and find... The optimal partitioning of the vertex set, this vertex set satisfy:

[0028] .

[0029] Preferably, if the condition is not met in step S4, the vertex is assigned to the cache module. When the cache module reaches its maximum capacity limit, the cache module returns the most suitable vertex and its neighbors for the current partitioning, specifically:

[0030] like:

[0031]

[0032] If the cache module has not reached its maximum capacity limit at this time, then the vertices are directly... Add a caching module to process vertices in the next vertex stream; if the caching module reaches its maximum capacity limit, then move the vertices to the next vertex stream. Add the caching module and return the vertices from the caching module that best suit the current partitioning. and its neighbor list, and for vertices Divide into sections.

[0033] Preferably, the cc-Fennel method is recursively called on the vertices. Divide into sections.

[0034] Preferably, the vertex most suitable for the current partitioning is returned in the cache module. and its neighbors list Specifically:

[0035] The CVA method returns the next vertex to be partitioned. The CVA method calculates the score of each vertex in the cache module, calculates the correlation between the vertices in the cache module and the partitioned vertices, and returns the vertex with the highest score.

[0036] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0037] This invention changes the order of vertex partitioning, determines the allocation rule for a single vertex in the data stream, and then reads the flow graph by repeating the above series of operations. ,until Once all nodes are assigned, the flow graph can be partitioned, improving the partitioning accuracy of the Fennel algorithm when the graph data size is too large. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the method flow provided in an embodiment of the present invention.

[0039] Figure 2 The following is a pseudocode diagram provided for an embodiment of the present invention. Detailed Implementation

[0040] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0041] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions.

[0042] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0043] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] Example 1

[0045] A graph data partitioning method based on a caching mechanism, such as Figure 1 As shown, it includes the following steps:

[0046] S1: Obtain graph data. Based on the graph data, obtain the total number of vertices in the graph, the neighbor list of each vertex, the initial vertex partition set, and the initial number of partitions. The vertex partition set includes several vertex sets.

[0047] S2: Calculate the load limit based on the total number of vertices and the initial number of partitions in the graph data;

[0048] S3: Calculate the associativity score of the vertex in each set of vertices that has not yet reached the load limit, based on the vertex and its neighbor list;

[0049] S4: Determine whether the vertex meets the caching conditions. If not, do not perform caching and directly divide it according to the correlation score. If it meets the conditions, divide the vertex into the cache module. When the cache module reaches the maximum capacity limit, the cache module returns the most suitable vertex and its neighbor list for the current division and returns to step S3.

[0050] S5: Determine whether all vertices have been partitioned. If yes, obtain the final graph data partition set; otherwise, return to step S3.

[0051] pseudocode such as Figure 2 As shown in the figure, Indicates the input image. Represents the vertices of the graph. express The edge between, This indicates the number of edges in the graph. This indicates the number of vertices in the graph. Represents vertices The neighborhood group, Indicates the number of partitions. , This indicates that partitions have been created. Indicates the first One partition, .

[0052] Example 2

[0053] This embodiment, based on Embodiment 1, continues to disclose the following content:

[0054] The graph data obtained in step S1 is of the type of edge set array.

[0055] In step S2, the load limit is calculated based on the total number of vertices and the initial number of partitions in the graph data. Specifically:

[0056]

[0057] In the formula, Maximum load, The total number of vertices in the graph data. This is the initial number of partitions. To adjust the parameters.

[0058] Set to 0.001~0.005.

[0059] In step S3, the associativity score of a vertex in each set of vertices that has not yet reached the load limit is calculated based on the vertex and its neighbor list. Specifically:

[0060]

[0061]

[0062] In the formula, Represents vertices The current load limit has not been reached. vertex set The correlation score on the surface Including scoring items and penalty items , Represents vertices The list of neighbors, The penalty coefficient is... This is the initial number of partitions. This indicates the number of edges in the graph data. The total number of vertices in the graph data. For cached parameters.

[0063] In step S3, when calculating the correlation score, a temporary space is created. This temporary space records data that has not yet reached its load limit. The set of vertices and the score of the set of vertices that have not yet reached the load limit. .

[0064] Example 3

[0065] Based on Examples 1 and 2, this embodiment continues to disclose the following content:

[0066] If the caching conditions are not met in step S4, no caching processing will be performed, and the data will be directly partitioned. Specifically:

[0067] like:

[0068]

[0069] In the formula, Represents vertices In the neighbor set, it is assigned to the vertex set. The number of vertices;

[0070] In this case, caching is unnecessary; simply determine the relevance score and find... The optimal partitioning of the vertex set, this vertex set satisfy:

[0071] .

[0072] If the conditions are met in step S4, the vertex is added to the cache module. When the cache module reaches its maximum capacity limit, the cache module returns the most suitable vertex and its neighbors for the current partitioning, specifically:

[0073] like:

[0074]

[0075] If the cache module has not reached its maximum capacity limit at this time, then the vertices are directly... Add a caching module to process vertices in the next vertex stream; if the caching module reaches its maximum capacity limit, then move the vertices to the next vertex stream. Add the caching module and return the vertices from the caching module that best suit the current partitioning. and its neighbor list, and for vertices Divide into sections.

[0076] Recursively call the cc-Fennel method on vertices Divide into sections.

[0077] The most suitable vertex for the current partitioning in the return cache module and its neighbors list Specifically:

[0078] The CVA method returns the next vertex to be partitioned. The CVA method calculates the score of each vertex in the cache module, calculates the correlation between the vertices in the cache module and the partitioned vertices, and returns the vertex with the highest score.

[0079] In this specific embodiment, the platform used was a Macbook Pro running macOS Monterey 12.1, with an Apple M1 processor and 8GB of memory. The programs used in the experiment were written in Python 3.8, utilizing not only native Python libraries but also the numpy (1.19.2) and networkx (2.6.3) libraries.

[0080] To address the issue of poor partitioning performance of the Fennel algorithm, this invention provides an improved method based on a caching strategy, which includes:

[0081] First, it is necessary to refer to the diagram. The total number of vertices in the graph and number of partitions Calculate the load limit ,in The parameters are manually set to adjust the load; the larger the parameter value, the more lenient the load limit used by the algorithm.

[0082] When reading flow graph data, load vertices all at once. and its neighbors list The data is used for subsequent calculations. Let the current vertex partition set be denoted as . ,definition As vertex In each current load limit not yet reached vertex set The score is calculated using a scoring function consisting of two parts: a scoring term and a penalty term.

[0083] Among the scoring items Represents vertices The neighboring vertices have now been assigned to partitions. The number of items in the partition is determined by this strategy based on the difference between two attributes, i.e., the partition number. Attraction to neighbors and repulsion from non-neighbors.

[0084] Penalty items This is based on partitioning Penalty for current load, where , The parameters are adjusted to control the weights associated with maximizing the number of neighbors of the input vertex and minimizing the number of non-neighbors during the partitioning process.

[0085] For each load limit not reached vertex set To facilitate subsequent CVA condition checks during score calculation, a temporary space is used simultaneously with the score function calculation. Record the set of vertices that have not reached the load limit, and the vertices... Scoring items in these vertex sets Score.

[0086] After traversing all sets of nodes that have not reached their load limits, it is necessary to determine whether the vertex meets the caching conditions. (in Represents vertices In the neighbor set, it was divided into The number of vertices; that is, If several vertices in the neighboring vertices have already been partitioned, then caching is unnecessary; only the scoring function needs to be used to determine which vertices are found. The optimal partitioning can be achieved. This partition must meet the following conditions: If there is This indicates that the vertex If the caching conditions are met, the CVA method must be called. At this point, if the cache... If the maximum capacity limit (MS) has not been reached and there is still space to accommodate another vertex, then directly... Add to the buffer, and the algorithm begins processing the next vertex in the vertex stream; if the buffer has reached MS, then... Add the vertex to the buffer and return the vertex in the buffer that best fits the current partitioning. and its neighbors list and recursively call the cc-Fennel method to... Divide into sections.

[0087] This establishes the allocation rules for a single vertex in the data stream. Subsequent steps involve repeating the above series of operations to read the flow graph. ,until Once all nodes are assigned, the flow graph partitioning is complete.

[0088] The data used in the examples are shown in Table 1, and the results are shown in Tables 2 and 3. The cc-Fennel algorithm is the best performing of the three algorithms, even reducing the edge cutting rate by 25.8% compared to the traditional Fennel algorithm in the BerkStan network graph. It can be seen that the optimization strategy proposed for the traditional Fennel algorithm is effective, reducing the generation of edges to a certain extent while maintaining the balance effect of the original algorithm.

[0089] Table 1

[0090]

[0091] Table 2

[0092]

[0093] Table 3

[0094]

[0095] The same or similar labels correspond to the same or similar parts;

[0096] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0097] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A graph data partitioning method based on a caching mechanism, characterized in that, Includes the following steps: S1: Obtain graph data. Based on the graph data, obtain the total number of vertices in the graph, the neighbor list of each vertex, the initial vertex partition set, and the initial number of partitions. The vertex partition set includes several vertex sets. S2: Calculate the load limit based on the total number of vertices and the initial number of partitions in the graph data; S3: Calculate the associativity score of the vertex in each set of vertices that has not yet reached the load limit, based on the vertex and its neighbor list; S4: Determine whether the vertex meets the caching conditions. If not, do not perform caching and directly divide it according to the correlation score. If it meets the conditions, divide the vertex into the cache module. When the cache module reaches the maximum capacity limit, the cache module returns the most suitable vertex and its neighbor list for the current division and returns to step S3. S5: Determine whether all vertices have been partitioned. If yes, obtain the final graph data partition set; otherwise, return to step S3.

2. The graph data partitioning method based on a caching mechanism according to claim 1, characterized in that, The graph data obtained in step S1 is of the type of edge set array.

3. The graph data partitioning method based on a caching mechanism according to claim 1, characterized in that, In step S2, the load limit is calculated based on the total number of vertices and the initial number of partitions in the graph data. Specifically: In the formula, Maximum load, The total number of vertices in the graph data. This is the initial number of partitions. To adjust the parameters.

4. The graph data partitioning method based on a caching mechanism according to claim 3, characterized in that, Set to 0.001~0.

005.

5. The graph data partitioning method based on a caching mechanism according to claim 1, characterized in that, In step S3, the associativity score of a vertex in each set of vertices that has not yet reached the load limit is calculated based on the vertex and its neighbor list. Specifically: In the formula, Represents vertices The current load limit has not been reached. vertex set The correlation score on the surface Including scoring items and penalty items , Represents vertices The list of neighbors, The penalty coefficient is... This is the initial number of partitions. This indicates the number of edges in the graph data. The total number of vertices in the graph data. For cached parameters.

6. The graph data partitioning method based on a caching mechanism according to claim 5, characterized in that, In step S3, when calculating the correlation score, a temporary space is created. This temporary space records data that has not yet reached its load limit. The set of vertices and the score of the set of vertices that have not yet reached the load limit. .

7. The graph data partitioning method based on a caching mechanism according to claim 6, characterized in that, If the caching conditions are not met in step S4, no caching processing will be performed, and the data will be directly partitioned. Specifically: like: In the formula, Represents vertices In the neighbor set, it is assigned to the vertex set. The number of vertices; In this case, caching is unnecessary; simply determine the relevance score and find... The optimal partitioning of the vertex set, this vertex set satisfy: 。 8. The graph data partitioning method based on a caching mechanism according to claim 7, characterized in that, If the conditions are met in step S4, the vertex is added to the cache module. When the cache module reaches its maximum capacity limit, the cache module returns the most suitable vertex and its neighbors for the current partitioning, specifically: like: If the cache module has not reached its maximum capacity limit at this time, then the vertices are directly... Add a caching module to process vertices in the next vertex stream; if the caching module reaches its maximum capacity limit, then move the vertices to the next vertex stream. Add the caching module and return the vertices from the caching module that best suit the current partitioning. and its neighbor list, and for vertices Divide into sections.

9. The graph data partitioning method based on a caching mechanism according to claim 8, characterized in that, The most suitable vertex for the current partitioning in the return cache module And its neighbor list, specifically: The CVA method returns the next vertex to be partitioned. The CVA method calculates the score of each vertex in the cache module, calculates the correlation between the vertices in the cache module and the partitioned vertices, and returns the vertex with the highest score.

Images