Large-scale graph partitioning method using streaming clustering in a GPU environment

Abstract

This invention relates to a method for large-scale graph partitioning using streaming clustering in a GPU environment. [Solution] The present invention relates to a technology that takes into account the structural environment of the GPU, gradually compresses graph data using the CPU, then transmits it to GPU memory to perform graph partitioning, thereby making the most efficient use of GPU memory and enabling rapid graph processing. As a result, the connectivity information for all clusters can be processed effectively in parallel using the high computational power of the GPU, and calculations requiring a large amount of memory space can be processed by the CPU, thereby overcoming the limitations of GPU memory.

Description

Technical Field

[0006]

[0001] The present invention relates to a method for large-scale graph partitioning by streaming clustering in a GPU environment. More specifically, the graph is compressed by clustering so as to be able to perform large-scale graph partitioning within the limited memory resources of a GPU, and relates to a method for large-scale graph partitioning by streaming clustering in a GPU environment capable of improving data processing efficiency.

Background Art

[0007] Nevertheless, loading large graph data on a GPU clearly has limitations. Generally, GPUs have relatively less memory capacity than CPUs, making it impossible to load large graph data into GPU memory. For example, the conventional technique Thanos proposes a rapid graph partitioning algorithm using a cross-partitioning algorithm, but when the data becomes extremely large, it faces the problem of being difficult to partition due to insufficient GPU memory (GPU out of memory).

[0008] In this regard, Patent Document 1 ("Hybrid GPU / CPU Data Processing Method") discloses a technique for performing large-scale graph searches on a parallel processing platform. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 6346893 (Registration Date: June 1, 2018) [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] Therefore, the present invention was derived to solve the problems of the prior art described above, and aims to provide a graph partitioning technique based on streaming clustering that can complement the memory limitations of GPUs and efficiently process large graphs. [Means for solving the problem]

[0011] The present invention, which provides a method for large-scale graph partitioning by streaming clustering in a GPU environment to achieve the above-mentioned objectives, preferably includes: an input step (S100) in which a host CPU receives original graph data; a clustering step (S200) in which the CPU generates a cluster map array for the received original graph data using a streaming clustering algorithm and generates cluster graph data using the cluster map array generated for the original graph data; a transmission step (S300) in which the GPU receives the generated cluster graph data; an initial partitioning step (S400) in which the GPU applies a pre-stored algorithm and performs initial partitioning on the transmitted cluster graph data; and an optimization step (S500) in which the CPU receives the transmission of the assigned partition label data, which is the result of the initial partitioning, and optimizes the assigned partition label data using the generated cluster map array.

[0012] Furthermore, it is preferable that the clustering step (S200) assumes that the original graph data is in an Edge Stream (ES) form where edges are input sequentially, provides edge information in batch units, and groups closely connected vertices into the same cluster for each vertex.

[0013] Furthermore, it is preferable that the initial partitioning step (S400) applies a pre-stored label propagation algorithm to repeatedly update the labels of each node according to the label distribution of neighboring nodes.

[0014] Furthermore, it is preferable that the optimization step (S500) uses the assigned partition label data to determine whether the quality of the divided graph would improve if each vertex were assigned to another partition of an adjacent vertex, rather than the partition to which the vertex is currently assigned, and then optimizes the graph division by vertex duplication based on the determination result. [Effects of the Invention]

[0015] According to the present invention, a large-scale graph partitioning method using streaming clustering in a GPU environment has the advantage of overcoming the memory limitations of GPU-based graph partitioning techniques and utilizing the memory access method of a hybrid graph partitioning technique for processing large-scale graph data.

[0016] This offers the advantage of leveraging the rapid computing power of GPUs to compress transmitted data into essential information, reducing memory limitations and data transmission overhead, and enabling efficient partitioning of large-scale graph data.

[0017] Furthermore, clustering similar data and performing partitioning on a GPU offers the advantage of supporting efficient parallel processing tailored to the data size. [Brief explanation of the drawing]

[0018] [Figure 1] This is an illustrative diagram showing a conventional graph partitioning method. [Figure 2] This is a sequential diagram illustrating a method for large-scale graph partitioning using streaming clustering in a GPU environment according to one embodiment of the present invention. [Figure 3]It is an exemplary diagram showing the structure of a large-scale graph partitioning method by streaming clustering in a GPU environment according to an embodiment of the present invention. [Figure 4] In the large-scale graph partitioning method by streaming clustering in a GPU environment according to an embodiment of the present invention, it is an exemplary diagram showing the process of generating a cluster graph by a clustering step (S200). [Figure 5] In the large-scale graph partitioning method by streaming clustering in a GPU environment according to an embodiment of the present invention, it is an exemplary diagram showing the data conversion process by an initial partitioning step (S400). [Figure 6] In the large-scale graph partitioning method by streaming clustering in a GPU environment according to an embodiment of the present invention, it is an exemplary diagram showing the partition assignment process by an initial partitioning step (S400). [Figure 7] In the large-scale graph partitioning method by streaming clustering in a GPU environment according to an embodiment of the present invention, it is an exemplary diagram showing the vertex replication process by an optimization step (S500).

Embodiments for Carrying Out the Invention

[0019] Hereinafter, the large-scale graph partitioning method by streaming clustering in a GPU environment according to the present invention having the above-described configuration will be described in detail with reference to the accompanying drawings. The drawings introduced below are provided as examples in order to sufficiently convey the idea of the present invention to those skilled in the art. Therefore, the present invention is not limited to the drawings presented below and can also be embodied in other forms. Also, the same reference numerals throughout the specification indicate the same components.

[0020] Here, in technical terms and scientific terms used, unless otherwise defined, they have the meanings commonly understood by those having ordinary knowledge in the technical field to which the present invention pertains, and in the following description and the accompanying drawings, descriptions of known functions and configurations that may obscure the gist of the present invention are omitted.

[0021] Furthermore, a system refers to a collection of components, including devices, mechanisms, and means, that are organized and interact regularly in order to perform a required function.

[0022] One embodiment of the present invention proposes a large-scale graph partitioning method using streaming clustering in a GPU environment that overcomes the memory limitations of the GPU and enables efficient processing of large-scale graph data. Considering the structural environment of the GPU, the graph data is gradually compressed by the CPU, then transmitted to the GPU memory for graph partitioning, thereby making the most efficient use of the GPU memory and enabling rapid graph processing. As a result, the connectivity information for all clusters can be processed effectively in parallel using the high computational power of the GPU, and calculations requiring a large amount of memory space can be processed by the CPU, thereby overcoming the memory limitations of the GPU.

[0023] Conventional graph partitioning methods can be broadly classified into vertex-centered and edge-centered methods. Referring to Figure 1, if graph data like that shown in Figure 1a) is input, vertex-centered partitioning is as shown in Figure 1b), and edge-centered partitioning is as shown in Figure 1c). In the case of vertex-centered partitioning, since information about neighboring vertices is mainly utilized, partitioning is possible by analyzing the role and importance of vertices, and this is advantageous for community identification and subgraph identification including important vertices. In the case of edge-centered partitioning, partitioning is performed by focusing on the distribution and connectivity of edges. This is mainly advantageous when trying to minimize the communication cost of the graph.

[0024] While vertex-centered partitioning requires storing all vertex position information for each cut edge, edge-centered partitioning saves storage space by storing less of this information. Edge-centered partitioning stores some vertices in other parts (partitions), which helps reduce the overall amount of data. Edge-centered partitioning is particularly efficient in graphs that follow a power-function distribution, in which case it can better balance partitions when there is a large difference in the number of neighbors between vertices.

[0025] Minimizing the number of duplicated vertices is a key factor in improving the efficiency of edge-centered partitioning, and such optimizations become even more important in distributed environments.

[0026] The fewer vertices assigned to each partition, and the fewer duplicated vertices, the lower the communication costs and the more efficient the overall system becomes.

[0027] As mentioned above, graph partitioning within a GPU environment is a crucial consideration from a data processing perspective. Processing a graph requires transmitting all data, including vertex and edge information, to GPU memory for processing. Effective graph partitioning in this process necessitates efficient use of GPU memory and minimizing memory usage.

[0028] Typically, for algorithm execution, graph data is moved from CPU memory to GPU memory, and after graph partitioning is performed, the results are allocated to GPU memory. The partitioned graph then moves back to CPU memory. Most graph partitioning algorithms that utilize GPU data parallelism employ an in-memory access method. The memory limitations of graph partitioning in a GPU environment manifest themselves in two main ways. Firstly, GPU memory capacity is limited compared to CPU memory, generally less than 20% of the CPU memory size. GPU memory is limited to a few GB to tens of GB, which may be insufficient to store all large graph data. Secondly, the data transfer speed between the CPU and GPU can be relatively slow, and overhead occurs during data transfer due to the use of separate memory spaces. To reduce this overhead, data replication must be minimized, and the memory copy process must be optimized.

[0029] As an example of a graph partitioning method optimized for a GPU environment, a multi-level access method is adopted to efficiently partition large-scale graph data, accelerating the graph partitioning process by leveraging the parallel processing capabilities of the GPU. Graph partitioning is performed in three steps: shrinking, initial partitioning, and expansion. Given an input graph, the shrinking step groups vertex pairs around edges with high weight values ​​and dynamically allocates GPU SIMT threads to distribute the load. In the initial partitioning step, the graph is partitioned using parallel BFS, and partition weight values ​​and cut edges are estimated based on the grouped vertices during this process. Finally, in the expansion step, the information obtained in the initial partitioning is used to establish the shrunk vertices, and edge cuts are minimized by repeatedly moving to boundary vertices.

[0030] Based on such conventional technologies, the large-scale graph partitioning method using streaming clustering in a GPU environment according to one embodiment of the present invention has as its primary technical objective the ability to perform graph partitioning within the limited memory resources of the GPU. GPU-based graph partitioning techniques generally compress graph data in the form of CSR (Compressed Sparse Row) or COO (Coordinate List). Such memory compression techniques are advantageous for minimizing memory usage within the GPU and loading graph data. However, loading large-scale graph data into GPU memory remains a difficult challenge. Even when graphs are compressed into COO or CSR form, memory equivalent to |E|, which corresponds to the total number of edges, is required, and the adjacency matrix requires space equal to the square of the number of vertices. For example, the NVIDIA GeForce RTX 3060 (12GB) is unable to batch process data with approximately 300 million or more edges.

[0031] To solve these problems, the method for large-scale graph partitioning using streaming clustering in a GPU environment according to one embodiment of the present invention maximizes the computing power of the GPU by compressing large-scale graph data through clustering, thereby maximizing data processing efficiency.

[0032] A method for large-scale graph partitioning by streaming clustering in a GPU environment according to one embodiment of the present invention includes an input step (S100), a clustering step (S200), a transmission step (S300), an initial partitioning step (S400), and an optimization step (S500), as illustrated in Figure 2. It goes without saying that each step is performed by a computing means including a CPU or a GPU.

[0033] Figure 3 is an illustrative diagram showing the structure of a large-scale graph partitioning method using streaming clustering in a GPU environment according to one embodiment of the present invention. Briefly, the CPU receives graph data as input. This graph consists of many edges, and the CPU gradually reads each edge of the graph, grouping edges with similar characteristics together to generate cluster data. That is, the input data is processed as an edge stream (ES), and cluster information is generated by streaming clustering. The generated cluster information is managed by the CPU as an array called a cluster map, which displays the cluster affiliation of each vertex. In other words, it shows which cluster each vertex belongs to. Next, the input graph is converted into a cluster graph using the cluster map. Before transmitting to the GPU, the data size is compressed to match the cluster size and GPU memory. That is, vertices belonging to the same cluster generate the initial graph again on a cluster basis, according to a predetermined maximum size setting. This reduces the size of the data. On the GPU, initial graph partitioning is performed based on the transmitted cluster information, and it is determined which part each cluster belongs to. The partition labels resulting from this process are returned to the CPU. During the CPU optimization process, ES is again utilized to perform optimizations that take into account the replication coefficient and load balancing when vertex duplication.

[0034] This allows the CPU to effectively transmit cluster information to the GPU via streaming clustering, minimizing data transfer between the CPU and GPU and improving overall performance. The cluster graph can reflect the overall information of the data, enabling high-quality graph partitioning on the GPU.

[0035] Based on this, to explain each step in detail, the input step (S100) involves the host CPU receiving the original graph data.

[0036] In the clustering step (S200), it is preferable that the CPU uses a streaming clustering algorithm to divide all of the original graph data received as input into smaller clusters and compress them.

[0037] In other words, a cluster map array is generated for the original graph data that has been received as input, and the cluster graph data is generated using the cluster map array generated for the original graph data.

[0038] In detail, the clustering step (S200) preferably assumes that the original graph data is in an Edge Stream (ES) form where edges are input sequentially, provides edge information in batch units, and groups closely connected vertices into the same cluster for each vertex.

[0039] The core of the clustering step (S200) is to group closely connected vertices in the graph data into the same cluster. That is, by grouping closely connected vertices in the graph data into the same cluster, the overall size of the graph data can be reduced while maintaining the relationships between vertices within each cluster. As a result, the clustering process allows for the identification of concentrated connectivity among vertices in the graph data, and by reflecting this in the partitioning process, more efficient graph partitioning can be achieved.

[0040] In detail, the CPU assumes that the input data is in an Edge Stream (ES) form, where edges are input sequentially. The ES provides edge information in batches, such as {(v1, v2), (v2, v3), ...}, indicating which vertices each vertex is connected to. Figure 4 shows an example of a cluster graph for compressing the input graph data. Figure 4a) represents the input graph data, where small circles represent vertices and large circles represent clusters. Figure 4b) shows an example of the input graph data converted to a CSR representation. Here, a vertex pointer (Vertex ptr) represents the starting point of the neighbor array (Neighbor) for each vertex. For example, the neighbors of vertex 0 are vertices 1, 2, and 4. Figure 4b) shows the compressed cluster data.

[0041] The aforementioned clustering step (S200) is designed by extending previous research on streaming cluster algorithms. Table 1 below shows the algorithm illustrating the clustering process.

[0042] [Table 1]

[0043] As shown in Table 1 above, the clustering step (S200) uses `edge_stream`, which is a list containing batches of edges, and `max_vol`, which is an integer indicating the maximum allowed volume of a cluster, as input. The output generates `unique_edges`, which is two-dimensional data indicating the edges between clusters, and `edge_weight`, which is a one-dimensional array indicating the weight values ​​of the edges between each cluster.

[0044] The main variables include an array `d` that stores the degree of each vertex, an array `vol` that stores the volume of each cluster, an array `v2c` that stores the ID of the cluster to which each vertex belongs, an array `cluster_weights` that shows the number of vertices belonging to each cluster as weighted values, and a list `edge_index_list` that stores edge information between clusters.

[0045] The main functions of the algorithm are the `process_edges` function, which processes a given batch of edges, merges and updates clusters, and the `find_smaller_volume_vertices` function, which identifies the vertices belonging to the cluster with the smaller volume among the two endpoints of each edge.

[0046] Based on these points, the processing process of the algorithm in the clustering step (S200) is as follows: First, the d, vol, and v2c arrays are initialized to 0, and cluster_weights and edge_index_list are initialized to empty lists. For each batch of edge_stream, the edge batch is moved to the GPU, the vertex degree is updated, and then the process_edges function is called to merge and update the clusters. After that, edge information between clusters is extracted and added to edge_index_list, and then the cluster weights are updated.

[0047] After processing all edge batches, the edge information between clusters stored in edge_index_list is merged into a single array, duplicate edges are removed, and then the weight value of each edge is calculated to generate unique_edges and edge_weight.

[0048] This algorithm efficiently clusters large graphs through GPU acceleration and batch processing, offering the advantage of optimizing memory usage and resolving memory shortage issues. It also applies cluster volume constraints (max_vol) to generate balanced clusters.

[0049] The size of the compressed information takes into account the maximum cluster size (Max_vol) and the compression ratio (CR) which takes into account the available memory space (GM, Global Memory) of the GPU. This can be calculated using Equation 1 below.

[0050] In Equation 1 below, the compression ratio is calculated by dividing the available memory of the GPU by the size of the graph. Also, in Equation 1 below, the maximum volume of the cluster (Max_vol) is calculated by multiplying the number of edges in the graph data by the compression ratio.

[0051]

number

[0052] The aforementioned formula 1 allows us to reduce the data size to the maximum size that the GPU can process, minimizing communication costs between the CPU and GPU. It works efficiently even with large graphs that can be entered into ES and can be used as a solution to dynamic clustering problems.

[0053] The clustering step (S200) obtains information about the connected edges between each cluster once all edges have been input and all vertices have been assigned to clusters.

[0054] In other words, the clustering step (S200) compresses the vertices that match in a cluster in order to form compressed graph data. That is, it identifies edges when a vertex belonging to C(i) and a vertex belonging to C(j) are connected to each other (when there are adjacent vertices inside a cluster).

[0055] In the transmission step (S300), the GPU receives the transmission of the cluster graph data generated in the clustering step (S200).

[0056] The initial partitioning step (S400) uses a pre-saved Label Propagation algorithm on the GPU to repeatedly update the labels of each node according to the label distribution of neighboring nodes, thereby generating initial partitions that intuitively reflect the structure of the graph data, i.e., generating initially assigned partition label data.

[0057] In other words, the initial partitioning step (S400) applies the label propagation algorithm to effectively utilize local information from the original graph data and repeatedly updates the labels of nodes according to the label distribution of neighboring nodes. Furthermore, by utilizing the cluster graph to reduce the graph size, the use of GPU memory can be optimized and the efficiency of parallel processing can be improved.

[0058] The initial partitioning step (S400) is a core process that performs the initial partitioning of the cluster graph, as shown in Table 2 below. The initial partitioning step (S400) utilizes a label propagation algorithm, which has a simple structure and is very suitable for parallel execution on a GPU.

[0059] [Table 2]

[0060] As shown in Table 2 above, the inputs used are the cluster graph's edge_index and edge_weight, the desired number of clusters (num_clusters), the maximum number of iterations (max_iterations), and the batch size (batch_size). The output returns the cluster labels assigned to each node.

[0061] The main processing steps of the algorithm in the initial partitioning step (S400) are as follows:

[0062] During the initialization process (lines 1-3 in Table 2), the number of nodes is calculated, and a unique label is assigned to each node. A sparse adjacency matrix is ​​also generated.

[0063] In the label propagation process (lines 5-17 in Table 2), the process is repeated for the maximum number of iterations, with each iteration processing the nodes by dividing them by a batch. The product of the adjacency matrix for each batch and the current label matrix is ​​calculated, and the index of the maximum value in the calculation is assigned as the new label.

[0064] In the post-processing stage (lines 19-26 in Table 2), the number of unique labels generated is checked, and the labels are adjusted as needed to match the desired number of clusters.

[0065] Finally, in the results return process (line 28), the cluster labels for each node that were ultimately determined are returned.

[0066] As shown in Table 2, the algorithm is designed to run efficiently on a GPU and optimizes memory usage even for large graphs through batch processing. Furthermore, during the label propagation process, it considers the structural characteristics of the graph to generate effective initial partitions, and the post-processing process ensures that the desired number of clusters is obtained.

[0067] Furthermore, the initial partitioning step (S400), as shown in Figure 5, receives the transmission of COO-formatted graph data (cluster graph data) generated by the clustering step (S200) and converts it to CSR format. The CSR representation is advantageous for parallel processing of cluster pointers (Cluster_ptr), allowing multiple GPU threads to effectively access the cluster pointer array. In Figure 5b), the arrows indicate GPU threads. Each thread uses its thread ID as an index for the cluster pointer to access the neighboring list in CSR format. The thread then accesses the weight values ​​of adjacent clusters to evaluate the connectivity between clusters. The cluster weight values ​​indicate how many times a cluster is connected to the vertices of other clusters, allowing the use of GPU-compressed cluster data and indirectly leveraging the overall information of the graph data.

[0068] The initial partitioning step (S400) calculates which partition is best suited to assign a particular cluster to, using the following formula 2.

[0069]

number

[0070] In the above equation 2, vol(P(i), C(j)) represents the number of vertices included when cluster C(j) merges with partition P(i). D(P(i), C(j)) represents how closely one cluster is connected to clusters belonging to other partitions.

[0071] In other words, when cluster C(j) is added to partition P(i), the first part calculates the partition's volume (load balancing), and the second part calculates the cluster's degree (connectivity). This is done for all cluster and partition combinations to calculate the optimal partitioning result.

[0072] Figure 6 is an illustrative diagram showing the process of determining which partition to assign cluster C(j) to at a specific point in time, as described in the initial partitioning step (S400). As illustrated in Figure 6, the entire graph is currently divided into four partitions, and ten clusters have been determined. The small black circles are vertices, and the larger red and yellow circles containing them represent clusters. The dotted lines indicate that the state of determining the Gain with the partition is idle, and the solid lines indicate that the relationship value is currently being determined. Here, the dotted lines are further divided into black and blue, where black indicates that the value has been determined, and blue indicates that the value has not yet been determined. Clusters {C(1), C(2), C(3), C(4), C(5)} belong to partition P(1), and clusters {C(6), C(7), C(8), C(9)} belong to partition P(2). In other words, currently, cluster C(j) has already calculated its gain with C(3) belonging to partition P(1), and is in the process of calculating its gain with clusters belonging to P(2). At the next point in time, C(j) will calculate its gains with P(3) and P(4), and use these results to decide which partition to assign C(j) to.

[0073] In the optimization step (S500), the CPU receives the transmission of the assigned partition label data, which is the result of the initial partitioning, and optimizes the assigned partition label data using the generated cluster map array.

[0074] In other words, the optimization step (S500) uses the assigned partition label data to determine whether the quality of the divided graph would improve if each vertex were assigned to another partition of an adjacent vertex instead of the partition to which the vertex is currently assigned, and then optimizes the graph partitioning by duplicating vertices based on the determination result.

[0075] In detail, the optimization step (S500) assigns vertices to each partition according to the partitioning result and performs fine-tuning at the vertex level to improve the quality of the partitioning result. Partitioning uses partition label data assigned from the GPU along with a cluster map array in CPU memory to assign each vertex to the appropriate partition. Fine-tuning is necessary in terms of edge distribution and vertex replication between partitions, as the GPU performs graph partitioning based on summarized information. Fine-tuning improves the quality of graph partitioning through load balancing and vertex replication. Such an optimization step (S500) is more efficient when performed on the CPU than on the GPU in order to minimize the amount of data transmitted between the CPU and GPU.

[0076] The optimization step (S500) determines, using the following formula 3, whether the quality of the divided graph improves when each vertex v(i) and v(j), which are assigned to different partitions, is assigned to another partition of an adjacent vertex instead of the conventional partition.

[0077]

number

[0078] As a result, if Score(Pi) is greater than Score(Pj), vertex v is copied to partition P(i); if Score(Pi) is less than Score(Pj), vertex v is copied to partition P(j).

[0079] In other words, the HDRF scoring technique, which minimizes the number of edge cuts, is applied to minimize communication costs between servers in graph partitioning. The partitioned data to date is partitioned based on clusters. The scoring function is activated only when clusters of adjacent vertices belong to different partitions. For this purpose, ES data is again utilized. For example, if vertices v(i) and v(j) are assigned to different partitions P(0) and P(2), the scoring function is performed to determine the partition with the highest score. Here, according to equation 3 above, if partition P(0) has a high score, vertex v(i) is duplicated, and if P(2) has a high score, vertex v(j) is duplicated.

[0080] Figure 7 shows the process by which vertex duplication is performed in the optimization step (S500). Here, if vertices v(i) and v(j) are assigned to the same partition, the operation for the optimization step (S500) is not performed, and a value is determined to perform the operation for the optimization step (S500) only when they are assigned to different partitions.

[0081] To demonstrate the superiority of this embodiment of the large-scale graph partitioning method using streaming clustering in a GPU environment, we performed a comparative performance evaluation with conventional graph partitioning techniques. Furthermore, we conducted a comparative performance evaluation of graph partitioning quality with conventional graph partitioning techniques using a quality evaluation method.

[0082] The main performance evaluation metrics considered were execution time, replication factor (RF), load balancing (LB), and locality partitioning score (LP Score). Comparative performance evaluations were conducted using data sets of various sizes.

[0083] Here, execution time is measured as the time it takes to perform the specified method on a given graph dataset, and the replication coefficient is a parameter indicating whether or not a copy of the vertices is maintained for each partition, defined as shown in Equation 4 below. Load balancing is defined as shown in Equation 5 below, and the locality score is an indicator for measuring the locality of the data set, representing the accuracy of whether the relevant clusters were correctly assigned to the partitioned areas. In other words, we use the community as the ground truth for the given data set to verify whether the graph was partitioned appropriately.

[0084]

number

[0085]

number

[0086] In particular, data locality refers to the physical proximity between data and the computing resources that process that data in a distributed processing system, and is an important metric considered in the field of computer science. Such a locality score is defined as shown in Equation 6 below. Here, V(P(i)) represents the set of vertices belonging to partition P(i), and V(C(j)) represents the set of vertices belonging to the top 5,000 highest-quality communities C(j). The higher the LP Score, the more similar partition P(i) and community C(j) are considered to be.

[0087]

number

[0088] Based on these points, comparative performance evaluations revealed that, in the case of execution time comparisons by graph size, the present invention exhibits a processing speed approximately 5-15% faster than conventional partitioning techniques performed in a CPU environment, regardless of graph size. Furthermore, for data sets with approximately 200 million or more edges, conventional GPU-based graph partitioning techniques cannot be used, meaning that it is impossible to load graph data containing approximately 200 million or more edges into GPU memory using conventional techniques. In the case of graph partitioning quality comparisons, the present invention was found to show a difference in the replication coefficient of up to approximately 1.9 times compared to the prior art, which is because the present invention uses a probabilistic methodology based on clustering. Graph partitioning techniques with a high replication coefficient have the advantage of improving the possibilities of graph data analysis and the efficiency of parallel processing. In the case of locality score comparison evaluations, the present invention was found to improve the number of locality scores by approximately 10-50% compared to the prior art. By utilizing compressed global information, it plays an important role in achieving locality in graph partitioning, meaning that it is possible to improve data analysis performance in distributed graph processing systems.

[0089] This method of large-scale graph partitioning using streaming clustering in a GPU environment, according to one embodiment of the present invention, is applicable to a wide range of fields, including social network analysis, medical information systems, recommendation systems, cybersecurity, scientific research and bioinformatics, transportation network analysis, and financial network analysis. For example, it can be used to analyze user relationships and identify communities on social networks such as Facebook and Twitter, and to improve personalized recommendation systems on e-commerce platforms. In medical information systems, it can be used to effectively partition patient records and track disease transmission routes, and in the field of cybersecurity, it can be used to analyze network traffic data to detect anomalies and understand cyberterrorism pathways. In short, this method of large-scale graph partitioning using streaming clustering in a GPU environment, according to one embodiment of the present invention, can help efficiently process and analyze large-scale graph data in a variety of application fields.

[0090] On the other hand, a large-scale graph partitioning method using streaming clustering in a GPU environment according to one embodiment of the present invention is implemented in a program instruction form that can be performed by various means of electronically processing information and can be recorded on a storage medium. The storage medium may include program instructions, data files, data structures, etc., either individually or in combination.

[0091] The program instructions recorded on the storage medium may be specifically designed and configured for the present invention, or they may be known and available to those skilled in the software art. Examples of storage mediums include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code produced by compilers, but also high-level language code executable by devices that process information electronically using interpreters, such as computers.

[0092] Although the present invention has been described above with reference to specific components and other details, and with drawings of limited embodiments, these are provided to facilitate a more general understanding of the invention. The invention is not limited to the above-described embodiments, and various modifications and variations can be made from this description by those with ordinary skill in the art to which the invention belongs.

[0093] Therefore, the concept of the present invention should not be limited to the embodiments described above. Not only the claims described later, but also all variations that are equivalent or comparable to the claims described herein can be said to fall within the scope of the concept of the present invention.

Claims

1. The host CPU performs an input step (S100) to receive the original graph data, A clustering step (S200) is performed in which the CPU generates a cluster map array for the original graph data received as input using a streaming clustering algorithm, and then generates cluster graph data using the cluster map array generated for the original graph data. A transmission step (S300) in which the cluster graph data generated by the GPU is received, An initial partitioning step (S400) is performed on the transmitted cluster graph data by applying a pre-stored algorithm on the GPU, A method for large-scale graph partitioning using streaming clustering in a GPU environment, comprising: an optimization step (S500) in which the CPU receives the transmission of assigned partition label data, which is the result of the initial partitioning, and optimizes the assigned partition label data using the generated cluster map array.

2. The clustering step (S200) is performed as follows: The method for large-scale graph partitioning by streaming clustering in a GPU environment according to claim 1, wherein the original graph data is assumed to be in an ES (Edge Stream) form in which edges are input sequentially, edge information is provided in batch units, and closely connected vertices are grouped into the same cluster for each vertex.

3. The initial division step (S400) is, A method for large-scale graph partitioning by streaming clustering in a GPU environment according to claim 2, wherein a pre-stored Label Propagation algorithm is applied to repeatedly update the labels of each node according to the label distribution of neighboring nodes.

4. The optimization step (S500) is, A method for large-scale graph partitioning by streaming clustering in a GPU environment according to claim 3, wherein, using the assigned partition label data, it is determined whether the quality of the partitioned graph would improve if each vertex were assigned to another partition of an adjacent vertex instead of the partition to which the vertex is assigned, and the graph partitioning is optimized by vertex duplication based on the determination result.

Images