Parallel graph computing processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a parallel graph computing and processing method, device, electronic device and storage medium, which can be applied to the computer field. The method includes: grouping a processing unit array graph data set to obtain a destination vertex set of each row of processing units and multiple source vertex sets corresponding to each destination vertex set; for each processing unit in each row of processing units, using a distribution scheduler to store the subgraph destination vertex set, subgraph source vertex set and coincident vertex set corresponding to the processing unit into a corresponding storage unit; using a computing module to read the edge data of the storage unit and the processing unit respectively, to obtain the initial vertex weights of the source and destination vertices of each vertex set and the edge weights of the edge data; based on the edge weights and vertex weights, using a computing module to determine the single-source shortest path between the source vertex and the destination vertex, to obtain the target vertex weights of the subgraph destination vertex set and the coincident vertex set; using the computing module to update the target vertex weights to the storage unit.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a parallel graph computing and processing method, device, electronic device, storage medium, and program product. Background Art

[0002] Graph computing is widely used on the internet. Graph data structures can effectively represent the relationships between data. Graph computing is typically performed using a central processing unit (CPU). However, graph computing processes large amounts of data, has low data relevance, and suffers from irregular memory access.

[0003] In the process of implementing the concept of the present disclosure, the inventors discovered that there are at least the following problems in the related art: due to the irregularity of memory access of graph algorithms, data access efficiency is low and graph calculation speed is slow. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a parallel graph computing processing method, apparatus, electronic device, storage medium and program product.

[0005] According to the first aspect of the present disclosure, a parallel graph computing processing method is provided, comprising: grouping the graph data set of the processing unit array based on a preset grouping rule to obtain the destination vertex set of each row of processing units in the above processing unit array, and multiple source vertex sets corresponding to each of the above destination vertex sets; for each processing unit in each row of processing units, using the distribution scheduler in the above processing unit to store the subgraph destination vertex set, subgraph source vertex set and overlapping vertex set corresponding to the above processing unit into the corresponding storage unit, wherein the above overlapping vertex set is the overlap between the above subgraph destination vertex set and the above subgraph source vertex set. a set of overlapping vertices; using the computing module in the processing unit to read the edge data corresponding to the storage unit and the graph data set corresponding to the processing unit respectively, and obtain the initial vertex weights corresponding to the source vertex and the destination vertex of each vertex set in the storage unit, as well as the edge weights of the edge data; based on the edge weights and the vertex weights, using the computing module to determine the single-source shortest path between the source vertex and the destination vertex, and obtain the target vertex weights corresponding to the subgraph destination vertex set and the overlapping vertex set; using the computing module to update the target vertex weights to the corresponding storage unit.

[0006] According to an embodiment of the present disclosure, the above-mentioned grouping of the graph data set of the processing unit array based on the preset grouping rules to obtain the destination vertex set of each row of processing units in the above-mentioned processing unit array, and multiple source vertex sets corresponding to each of the above-mentioned destination vertex sets, includes: based on the vertex numbers of the multiple destination vertices in the above-mentioned graph data set, uniformly grouping the above-mentioned destination vertices using the hash grouping rule to obtain the destination vertex set of each row of processing units in the above-mentioned processing unit array; based on the total in-degree number of the destination vertices in each of the above-mentioned destination vertex sets and the preset number of groups, uniformly grouping the multiple source vertices corresponding to the above-mentioned destination vertex set to obtain multiple of the above-mentioned source vertex sets.

[0007] According to an embodiment of the present disclosure, the above-mentioned multiple source vertices corresponding to the above-mentioned destination vertex set are uniformly grouped based on the total in-degree of the destination vertices in each of the above-mentioned destination vertex sets and the preset number of groups to obtain multiple above-mentioned source vertex sets, including: calculating the target out-degree of each source vertex set based on the above-mentioned total in-degree and the above-mentioned preset grouping; and uniformly grouping the multiple above-mentioned source vertices based on the above-mentioned target out-degree to obtain multiple above-mentioned source vertex sets.

[0008] According to an embodiment of the present disclosure, the above-mentioned distribution scheduler in the above-mentioned processing unit is used to store the subgraph destination vertex set, subgraph source vertex set and overlapping vertex set corresponding to the above-mentioned processing unit in the corresponding storage unit, including: using the above-mentioned distribution scheduler to store the above-mentioned subgraph destination vertex set in the on-chip static destination storage unit, storing the above-mentioned subgraph source vertex set in the on-chip static source storage unit, and storing the overlapping vertex set in the off-chip static coincidence storage unit.

[0009] According to an embodiment of the present disclosure, the above-mentioned use of the above-mentioned calculation module to update the above-mentioned target vertex weights to the corresponding storage unit includes: using the above-mentioned calculation module to update the target vertex weights corresponding to the above-mentioned subgraph destination vertex set to the above-mentioned on-chip static destination storage unit; in the case that the target vertex weights corresponding to the overlapping vertices in the above-mentioned overlapping vertex set are less than the initial vertex weights, using the above-mentioned calculation module to update the target vertex weights corresponding to the above-mentioned overlapping vertex set to the above-mentioned off-chip static overlapping storage unit.

[0010] According to an embodiment of the present disclosure, the above method also includes: for each row of processing units, when each of the above processing units completes the vertex weight update, selecting the fastest convergence processing unit in the row of processing units, and storing the iteration destination vertex set and iteration source vertex set corresponding to the above processing unit with the fastest convergence in the off-chip storage unit; for each iteration destination vertex set and iteration source vertex set in the above off-chip storage unit, using the above calculation module, iteratively processing the iteration destination vertex set and the iteration source vertex set to obtain a new iteration destination vertex set and a new iteration source vertex set; when the difference in vertex weights of corresponding vertices between the new iteration destination vertex set and the new iteration source vertex set generated in the i-th and i+1-th iterations respectively is less than a preset value, the new iteration destination vertex set and the new iteration source vertex set corresponding to the i-th iteration are determined as the graph calculation result.

[0011] The second aspect of the present disclosure provides a parallel graph computing processing device, including: a grouping module for grouping the graph data set of the processing unit array based on a preset grouping rule to obtain the destination vertex set of each row of processing units in the above processing unit array, and multiple source vertex sets corresponding to each of the above destination vertex sets; a storage module for storing the subgraph destination vertex set, subgraph source vertex set and overlapping vertex set corresponding to the above processing unit in the corresponding storage unit using the distribution scheduler in the above processing unit for each processing unit in each row of processing units, wherein the above overlapping vertex set is composed of vertices that overlap between the above subgraph destination vertex set and the above subgraph source vertex set. a set of; a reading module, used to use the calculation module in the above-mentioned processing unit to read the edge data corresponding to the above-mentioned storage unit and the above-mentioned graph data set respectively, to obtain the initial vertex weights corresponding to the source vertex and the destination vertex of each vertex set in the above-mentioned storage unit, and the edge weights of the above-mentioned edge data; a calculation module, used to determine the single-source shortest path between the above-mentioned source vertex and the above-mentioned destination vertex based on the above-mentioned edge weights and the above-mentioned vertex weights, using the above-mentioned calculation module, to obtain the target vertex weights corresponding to the above-mentioned subgraph destination vertex set and the above-mentioned overlapping vertex set; an updating module, used to use the above-mentioned calculation module to update the above-mentioned target vertex weights to the corresponding storage unit.

[0012] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method.

[0013] A fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0014] The fifth aspect of the present disclosure further provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0015] According to the parallel graph computing processing method, apparatus, electronic device, storage medium, and program product provided by the present disclosure, by grouping the graph data set of the processing unit array based on preset grouping rules, the data in the graph data set can be accessed sequentially based on the grouping results during traversal, so that the access process does not involve random address access, thereby speeding up access. During the data processing process, the vertex weights of the destination vertex and the source vertex are updated to more quickly determine the single-source shortest path, and by setting the overlapping vertex set, the weights of the overlapping vertices can be further determined during the update process, thereby speeding up the graph calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 A diagram schematically illustrates an application scenario of a parallel graph computing method, apparatus, electronic device, storage medium, and program product according to an embodiment of the present disclosure;

[0018] Figure 2 Schematically shows a flow chart of a parallel graph computing processing method according to an embodiment of the present disclosure;

[0019] Figure 3 Schematically shows a flow chart of a parallel graph computing processing method according to another embodiment of the present disclosure;

[0020] Figure 4 Schematically shows a diagram of grouping a graph data set according to an embodiment of the present disclosure;

[0021] Figure 5 The following schematically shows a structural block diagram of a parallel graph computing and processing device according to an embodiment of the present disclosure;

[0022] Figure 6 A block diagram of an electronic device suitable for implementing a parallel graph computing processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0028] Classic graph algorithms, such as Breadth First Search (BFS), Single-Source Shortest Paths (SSSP), and PageRank, are widely used on the Internet.

[0029] Graph algorithms need to process graph datasets with large data volumes, low data correlation, and irregular memory access. A graph dataset is a collection of vertices and edges, which can be stored using an adjacency matrix, an adjacency list, or directly using an edge list. Different storage methods essentially correspond to different algorithm execution strategies, and all execution strategies can generally be roughly divided into two types: vertex-centric processing and edge-centric processing. The former strategy is to traverse vertices and visit each destination vertex of the vertex in turn, and is more inclined to use adjacency matrix or adjacency list storage methods; while the latter traverses edges and visits the source and destination vertices of the edge, and is more inclined to use edge lists and other storage methods.

[0030] For architectures that pursue multi-core parallelism, an edge-centric strategy is clearly more suitable. This is because the edge list remains fixed during the graph algorithm iteration (only the vertex information is continuously updated), allowing multiple cores to independently iterate over different edge lists. When processing large-scale graph datasets, it is common to divide the graph into multiple subgraphs by grouping vertices. Currently, uniform vertex grouping is the main method used, ensuring that each vertex set has the same number of vertices.

[0031] In related technologies, central processing units (CPUs) are typically used to handle graph computations and other problems. However, this process encounters bottlenecks in parallelism and memory access. When using graphics processing units (GPUs) to handle graph computations and other problems, while GPUs offer far greater parallelism than CPUs, video memory is still essentially memory, and memory bottlenecks caused by irregular memory access persist. In fact, due to the discontinuous memory access of graph algorithms, traditional cache memory structures are extremely inefficient, with spatial locality almost completely lost, making storage systems designed for general-purpose computing unsuitable.

[0032] In view of this, embodiments of the present disclosure provide a parallel graph computing and processing method, a parallel graph computing and processing device, an electronic device, a readable storage medium, and a computer program product. Among them, the parallel graph computing processing method includes: grouping the graph data set of the processing unit array based on a preset grouping rule to obtain the destination vertex set of each row of processing units in the processing unit array, and multiple source vertex sets corresponding to each destination vertex set; for each processing unit in each row of processing units, using the distribution scheduler in the processing unit to store the subgraph destination vertex set, subgraph source vertex set and overlapping vertex set corresponding to the processing unit in the corresponding storage unit, wherein the overlapping vertex set is a set of vertices that overlap between the subgraph destination vertex set and the subgraph source vertex set; using the computing module in the processing unit to read the edge data corresponding to the storage unit and the processing unit in the graph data set respectively, to obtain the initial vertex weights corresponding to the source vertex and destination vertex of each vertex set in the storage unit, as well as the edge weights of the edge data; based on the edge weights and vertex weights, using the computing module to determine the single-source shortest path between the source vertex and the destination vertex, to obtain the target vertex weights corresponding to the subgraph destination vertex set and the overlapping vertex set; using the computing module to update the target vertex weights to the corresponding storage unit.

[0033] Figure 1 The application scenario diagram of the parallel graph computing processing method, device, electronic device, storage medium and program product according to the embodiments of the present disclosure is schematically shown.

[0034] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0035] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0036] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0037] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received user requests and other data, and feed back processing results (e.g., web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the parallel graph calculation processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the parallel graph calculation processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The parallel graph calculation processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the parallel graph calculation processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0040] The following will be based on Figure 1 The scene described by Figures 2 to 4 The parallel graph computing processing method of the disclosed embodiment is described in detail.

[0041] Figure 2 The flowchart of the parallel graph computing processing method according to an embodiment of the present disclosure is schematically shown.

[0042] like Figure 2 As shown, the method includes operations S201 to S205.

[0043] In operation S201 , the graph data set of the processing unit array is grouped based on a preset grouping rule to obtain a destination vertex set of each row of processing units in the processing unit array and a plurality of source vertex sets corresponding to each destination vertex set.

[0044] In operation S202, for each processing unit in each row of processing units, the distribution scheduler in the processing unit is used to store the subgraph destination vertex set, subgraph source vertex set and overlapping vertex set corresponding to the processing unit into the corresponding storage unit, wherein the overlapping vertex set is a set of vertices that overlap between the subgraph destination vertex set and the subgraph source vertex set.

[0045] In operation S203, the computing module in the processing unit is used to read the edge data corresponding to the processing unit in the storage unit and the graph data set respectively to obtain the initial vertex weights corresponding to the source vertices and destination vertices of each vertex set in the storage unit, as well as the edge weights of the edge data.

[0046] In operation S204 , a single-source shortest path between a source vertex and a destination vertex is determined using a calculation module based on edge weights and vertex weights, and target vertex weights corresponding to the subgraph destination vertex set and the coincident vertex set are obtained.

[0047] In operation S205 , the target vertex weight is updated to the corresponding storage unit using the calculation module.

[0048] According to an embodiment of the present disclosure, the processing unit array is composed of a plurality of processing units (Processing Element, referred to as PE), which can be a 4*4 PE array. A graph data set is obtained from an off-chip storage unit, and the graph data set of the processing unit array is grouped based on a preset grouping rule. The graph data set is a set of several vertices and edges, and includes vertex weights corresponding to the vertices and edge weights corresponding to the edges. By grouping the graph data set, the destination vertex set of each row of processing units in the processing unit array is the same, and the multiple source vertex sets corresponding to each row of the destination vertex set are continuous. This makes it possible that random address access is not involved in the access process, thereby speeding up the access speed.

[0049] According to an embodiment of the present disclosure, for each processing unit in each row of processing units, the distribution scheduler in the processing unit is used to store the subgraph destination vertex set, subgraph source vertex set and overlapping vertex set corresponding to the processing unit in the corresponding storage unit. The distribution scheduler includes a scheduling module and a merging module, and also includes a storage unit for storing processing unit execution information. The scheduling module is used to read vertex information from the off-chip storage unit and distribute it to the corresponding storage unit. The merging module is used to read the destination vertex in the storage unit after all processing units in the same row have completed execution, and send the destination vertex back to the off-chip storage unit.

[0050] According to the embodiments of the present disclosure, each vertex in a graph dataset can serve as either a destination vertex or a source vertex. Therefore, there may be intersections between the destination vertex set and the source vertex set generated by grouping. When overlapping vertices appear in the destination vertex set and the source vertex set, in addition to the destination vertices, some source vertices also need to be updated. Therefore, a storage unit accessible to multiple processing units is introduced in the processing unit to store the data of the overlapping vertices.

[0051] According to an embodiment of the present disclosure, a computing module in a processing unit is used to read a storage unit to obtain initial vertex weights corresponding to source vertices and destination vertices of each vertex set in the storage unit. A computing module in a processing unit is used to read edge data corresponding to the processing unit in a graph data set to obtain edge weights of the edge data. The weights in the vertex weights and edge weights can be represented by the distance in a single-source shortest path algorithm, by the page ranking (PR value) of a page in a page ranking algorithm, or by the number of layers in a breadth-first search algorithm.

[0052] According to an embodiment of the present disclosure, based on edge weights and vertex weights, a calculation module is used to determine the single-source shortest path between the source vertex and the destination vertex, and the target vertex weights corresponding to the subgraph destination vertex set and the overlapping vertex set are obtained. For example: assuming that the vertex weight of source vertex No. 1 is 10, the initial vertex weight of destination vertex No. 10 is 100, and the edge weight between source vertex No. 1 and destination vertex No. 10 is 50, then the single-source shortest path between source vertex No. 1 and destination vertex No. 10 is determined to be the sum of the vertex weight of source vertex No. 1 and the edge weight, which is 60. Therefore, the target vertex weight of destination vertex No. 10 is determined to be 60, and the weight of destination vertex No. 10 is then updated.

[0053] According to the parallel graph computing processing method, apparatus, electronic device, storage medium, and program product provided by the present disclosure, by grouping the graph data set of the processing unit array based on preset grouping rules, the data in the graph data set can be accessed sequentially based on the grouping results during traversal, so that the access process does not involve random address access, thereby speeding up access. During the data processing process, the vertex weights of the destination vertex and the source vertex are updated to more quickly determine the single-source shortest path, and by setting the overlapping vertex set, the weights of the overlapping vertices can be further determined during the update process, thereby speeding up the graph calculation.

[0054] Figure 3 The flowchart of a parallel graph computing method according to another embodiment of the present disclosure is schematically shown.

[0055] According to the embodiments of the present disclosure, Figure 3As shown, the destination vertex set and source vertex set in the off-chip memory unit 301 (OFF_CHIP MEMORY) are input into the dispatch scheduler 302 (Control Unit) of the processing element (PE) through a vertex-centric processing method. The dispatch scheduler 302 includes a dispatch module 303 (Dispatcher) and a merge module 304 (Merger), and also includes a storage unit 305 (mem_info, memory information SRAM) for storing processing unit execution information. The dispatch module 303 is used to read the vertex information from the off-chip memory unit, allocate the source vertex to the on-chip static source memory unit 306 (SRC_SRAM, source SRAM), and allocate the destination vertex set to the on-chip static destination memory unit 307 (DST_SRAM, destination SRAM). The on-chip static source memory unit 306 and the on-chip static destination memory unit 307 are both composed of a number of static memory banks (SRAM banks), which are used to store the source vertices and destination vertices from the off-chip memory unit 301, respectively.

[0056] According to an embodiment of the present disclosure, the edge data in the local dynamic storage unit 308 (Local DRAM) corresponding to the processing unit is input into the calculation module 309 (CAL, calculate) of the processing unit through edge-centric processing. The edge data includes the source vertex (SRC, source), the destination vertex (DST, destination) and the edge weight (weight). The calculation module 309 uses the edge data to access the on-chip static source storage unit 306 and the on-chip static destination storage unit 307 and complete the corresponding weight calculation, and writes the weight calculation result back to the on-chip static destination storage unit 307.

[0057] According to an embodiment of the present disclosure, the processing unit further introduces an off-chip static overlap storage unit 310 (SRAM for overlap) accessible to multiple PEs. The dispatch scheduler 302 inputs the overlapping vertices into the off-chip static overlap storage unit 310. The calculation module 309 uses the edge data to access the off-chip static overlap storage unit 310 and complete the corresponding weight calculation, and writes the weight calculation result back to the off-chip static overlap storage unit 310.

[0058] According to an embodiment of the present disclosure, grouping a graph data set of a processing unit array based on a preset grouping rule to obtain a destination vertex set for each row of processing units in the processing unit array and multiple source vertex sets corresponding to each destination vertex set may include the following operations:

[0059] Based on the vertex numbers of multiple destination vertices in the graph data set, the destination vertices are evenly grouped using hash grouping rules to obtain the destination vertex set of each row of processing units in the processing unit array; based on the total in-degree of the destination vertices in each destination vertex set and the preset number of groups, the multiple source vertices corresponding to the destination vertex set are evenly grouped to obtain multiple source vertex sets.

[0060] According to an embodiment of the present disclosure, for the destination vertex set, a method of uniformly grouping by vertex is used, that is, the destination vertex set of PE in each row has the same number of destination vertices, and hash grouping is performed based on the vertex number of the destination vertex (assuming that a vertex is numbered x and it is planned to be divided into N destination vertex sets, then the vertex belongs to the (x mod N)th vertex set). Hash grouping can, to a certain extent, avoid the continuous appearance of vertices with high in-and-out degrees in the original data set, which makes the in-and-out degrees of the corresponding vertex set too high.

[0061] According to an embodiment of the present disclosure, the source vertices are grouped uniformly by edges. Based on the total in-degree of the destination vertices in each destination vertex set and the preset number of groups, the multiple source vertices corresponding to the destination vertex set are uniformly grouped to obtain multiple source vertex sets, so that the edge set size of each subgraph processed by the same row of PE is the same.

[0062] Figure 4 A schematic diagram of grouping a graph dataset according to an embodiment of the present disclosure is shown schematically.

[0063] According to an embodiment of the present disclosure, based on the total in-degree of the destination vertices in each destination vertex set and a preset number of groups, uniformly grouping multiple source vertices corresponding to the destination vertex set to obtain multiple source vertex sets may include the following operations:

[0064] The target out-degree of each source vertex set is calculated based on the total in-degree and the preset grouping; multiple source vertices are evenly grouped based on the target out-degree to obtain multiple source vertex sets.

[0065] According to the embodiments of the present disclosure, the graph data set has the characteristics of power-law distribution. If the traditional grouping method of uniformly grouping by vertices is used, the out-degrees of different vertices vary greatly, and the sizes of the edge lists corresponding to each subgraph will also vary greatly, which will lead to large differences in the execution time of different PEs in the same row. The PE that finishes executing first needs to wait for the PE that finishes executing last to complete the synchronization of the destination vertex data. This embodiment proposes a grouping method of uniformly grouping by edges, which maintains the uniform grouping of the destination vertex set by the number of vertices, changes the grouping method of the source vertex set, obtains the total out-degree of all vertices to each destination vertex set and calculates the average, so that the out-degree of each source vertex set to the destination vertex set is equal to the average, because the out-degree is numerically equal to the number of edges of the subgraph. In this way, the size of the edge list iterated by each PE is controlled to be the same.

[0066] According to the embodiments of the present disclosure, Figure 4 As shown, assuming that the processing unit array is a 4*4 PE array, containing 16 PEx_y, and the position corresponding to each PE is Ex,y_x. Dx represents the destination vertex set corresponding to the x-th row PE, and Sx,y represents the y-th source vertex set separated according to Dx. Assuming that the total in-degree of the destination vertices in this row is in_dgr, and the preset number of groups is determined to be M, the out-degree of each source vertex set should be equal to in_dgr / M, so that the out-degree of each source vertex set relative to the destination vertex set in this row is the same. Figure 4 In the example, the y-th source vertex set derived from Dx must satisfy that the out-degree of all vertex sets Sx,0 to Sx,M to Dx is equal to in_dgr / M. This ensures that the edge sets of each subgraph processed by the same row of PEs have the same size, because the number of edges is equal to the out-degree of the source vertex set to the destination vertex set, that is, in_dgr / M.

[0067] According to an embodiment of the present disclosure, using a distribution scheduler in a processing unit to store the subgraph destination vertex set, subgraph source vertex set, and coincident vertex set corresponding to the processing unit in a corresponding storage unit may include the following operations:

[0068] The distribution scheduler is used to store the subgraph destination vertex set in the on-chip static destination storage unit, the subgraph source vertex set in the on-chip static source storage unit, and the coincident vertex set in the off-chip static coincidence storage unit.

[0069] According to an embodiment of the present disclosure, the processing unit includes an on-chip static destination storage unit and an on-chip static source storage unit, and an off-chip static coincidence storage unit is a storage unit accessible to the processing unit.

[0070] According to an embodiment of the present disclosure, updating the target vertex weight to the corresponding storage unit using the calculation module may include the following operations:

[0071] The calculation module is used to update the target vertex weights corresponding to the subgraph destination vertex set to the on-chip static destination storage unit; when the target vertex weights corresponding to the overlapping vertices in the overlapping vertex set are less than the initial vertex weights, the calculation module is used to update the target vertex weights corresponding to the overlapping vertex set to the off-chip static overlapping storage unit.

[0072] According to an embodiment of the present disclosure, the calculation module calculates the single-source shortest path and updates the target vertex weight corresponding to the obtained destination vertex to the on-chip static destination storage unit. For the overlapping vertices in the overlapping vertex set, the overlapping vertex can be either the source vertex or the destination vertex. If the weight of the overlapping vertex target vertex is less than the initial vertex weight, the target vertex weight corresponding to the overlapping vertex set is updated to the off-chip static overlapping storage unit, thereby speeding up the graph calculation.

[0073] According to an embodiment of the present disclosure, the method may further include the following operations:

[0074] For each row of processing units, when each processing unit completes the vertex weight update, the fastest converging processing unit in the row is selected, and the iteration destination vertex set and iteration source vertex set corresponding to the fastest converging processing unit are stored in the off-chip storage unit.

[0075] For each iteration destination vertex set and iteration source vertex set in the off-chip storage unit, the calculation module is used to iteratively process the iteration destination vertex set and the iteration source vertex set to obtain a new iteration destination vertex set and a new iteration source vertex set.

[0076] When the difference in vertex weights of corresponding vertices between the new iteration destination vertex set and the new iteration source vertex set generated in the i-th and i+1-th iterations respectively is less than a preset value, the new iteration destination vertex set and the new iteration source vertex set corresponding to the i-th iteration are determined as the graph calculation results.

[0077] According to the embodiments of the present disclosure, due to the uniform grouping by edges, the edge lists of the subgraphs processed by PEs in the same row are of the same size, so the time required for PE execution to complete is also similar, which greatly reduces the time PEs wait for synchronization. For each row of processing units, after all PEs in the same row complete the iteration of the edge list, they begin to synchronize the destination vertices. Because the destination vertex set of all PEs in the row is the same, all PEs need only read the destination vertices simultaneously and sequentially and write them back to the off-chip storage unit. Before writing to the off-chip storage, the result that makes the algorithm converge the fastest is selected as the destination vertex result of this iteration, that is, the iterative destination vertex set and the iterative source vertex set.

[0078] According to the embodiments of the present disclosure, since the goal of graph computation is to iterate until all vertices converge, vertices that converge the fastest are selected for updating. From loading vertices into the PE to writing them back to off-chip storage, a parallel iteration of multiple subgraphs is completed. Repeating these steps until the weights of all vertices no longer decrease indicates that the graph algorithm for the graph dataset has converged.

[0079] Based on the above parallel graph computing method, the present disclosure also provides a parallel graph computing device. Figure 5 The device is described in detail.

[0080] Figure 5 The structural block diagram of the parallel graph computing processing device according to an embodiment of the present disclosure is schematically shown.

[0081] like Figure 5 As shown, the parallel graph computing and processing device 500 of this embodiment includes a grouping module 510 , a storage module 520 , a reading module 530 , a computing module 540 and an updating module 550 .

[0082] Grouping module 510 is configured to group the graph dataset of the processing unit array based on a preset grouping rule, thereby obtaining a destination vertex set for each row of processing units in the processing unit array and multiple source vertex sets corresponding to each destination vertex set. In one embodiment, grouping module 510 may be configured to perform operation S201 described above and will not be further described herein.

[0083] Storage module 520 is configured to, for each processing unit in each row of processing units, utilize the distribution scheduler within the processing unit to store the subgraph destination vertex set, subgraph source vertex set, and overlapping vertex set corresponding to the processing unit in the corresponding storage unit. The overlapping vertex set is the set of vertices that overlap between the subgraph destination vertex set and the subgraph source vertex set. In one embodiment, storage module 520 can be configured to perform operation S202 described above and will not be further described here.

[0084] Reading module 530 is configured to use the computing module in the processing unit to read edge data corresponding to the processing unit from the storage unit and the graph data set, respectively, to obtain initial vertex weights corresponding to the source and destination vertices of each vertex set in the storage unit, as well as edge weights of the edge data. In one embodiment, reading module 530 can be configured to perform operation S203 described above and will not be further described here.

[0085] Calculation module 540 is configured to determine a single-source shortest path between a source vertex and a destination vertex based on the edge weights and vertex weights, thereby obtaining target vertex weights corresponding to the subgraph destination vertex set and the overlapping vertex set. In one embodiment, calculation module 540 may be configured to perform operation S204 described above and will not be further described herein.

[0086] The updating module 550 is configured to update the target vertex weight to the corresponding storage unit using the calculation module. In one embodiment, the updating module 550 can be configured to execute the operation S205 described above, which will not be described in detail here.

[0087] According to the parallel graph computing processing method, apparatus, electronic device, storage medium, and program product provided by the present disclosure, by grouping the graph data set of the processing unit array based on preset grouping rules, the data in the graph data set can be accessed sequentially based on the grouping results during traversal, so that the access process does not involve random address access, thereby speeding up access. During the data processing process, the vertex weights of the destination vertex and the source vertex are updated to more quickly determine the single-source shortest path, and by setting the overlapping vertex set, the weights of the overlapping vertices can be further determined during the update process, thereby speeding up the graph calculation.

[0088] According to an embodiment of the present disclosure, the grouping module 510 includes a destination grouping submodule and a source grouping submodule.

[0089] The destination grouping submodule is used to evenly group the destination vertices based on the vertex numbers of multiple destination vertices in the graph data set using hash grouping rules to obtain the destination vertex set of each row of processing units in the processing unit array.

[0090] The source grouping submodule is used to evenly group multiple source vertices corresponding to each destination vertex set based on the total in-degree of the destination vertices in each destination vertex set and a preset number of groups to obtain multiple source vertex sets.

[0091] According to an embodiment of the present disclosure, the source grouping submodule includes a computing unit and a source grouping unit.

[0092] The calculation unit is configured to calculate the target out-degree of each source vertex set based on the total in-degree and the preset grouping.

[0093] The source grouping unit is used to evenly group multiple source vertices based on the target out-degree to obtain multiple source vertex sets.

[0094] According to an embodiment of the present disclosure, the storage module 520 includes a storage sub-module.

[0095] The storage submodule is used to use the distribution scheduler to store the subgraph destination vertex set into the on-chip static destination storage unit, store the subgraph source vertex set into the on-chip static source storage unit, and store the overlap vertex set into the off-chip static overlap storage unit.

[0096] According to an embodiment of the present disclosure, the update module 550 includes a first update submodule and a second update submodule.

[0097] The first updating submodule is used to update the target vertex weights corresponding to the subgraph destination vertex set to the on-chip static destination storage unit using the calculation module.

[0098] The second updating submodule is used to update the target vertex weights corresponding to the overlapping vertex set to the off-chip static overlap storage unit using the calculation module when the target vertex weights corresponding to the overlapping vertices in the overlapping vertex set are less than the initial vertex weights.

[0099] According to an embodiment of the present disclosure, the parallel graph computing processing device 500 further includes an off-chip storage module, an iteration module, and a result determination module.

[0100] The off-chip storage module is used to select the fastest converging processing unit in each row of processing units when each processing unit completes the vertex weight update, and store the iteration destination vertex set and iteration source vertex set corresponding to the fastest converging processing unit in the off-chip storage unit.

[0101] The iteration module is used to iteratively process each iteration destination vertex set and iteration source vertex set in the off-chip storage unit using the calculation module to obtain a new iteration destination vertex set and a new iteration source vertex set.

[0102] The result determination module is used to determine the new iteration destination vertex set and the new iteration source vertex set corresponding to the i-th iteration as the graph calculation result when the difference in vertex weights of the corresponding vertices between the new iteration destination vertex set and the new iteration source vertex set generated in the i-th and i+1-th iterations respectively is less than a preset value.

[0103] According to an embodiment of the present disclosure, any multiple modules among the grouping module 510, the storage module 520, the reading module 530, the computing module 540, and the updating module 550 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the grouping module 510, the storage module 520, the reading module 530, the computing module 540, and the updating module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the grouping module 510 , the storage module 520 , the reading module 530 , the calculation module 540 , and the updating module 550 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0104] Figure 6 A block diagram of an electronic device suitable for implementing a parallel graph computing processing method according to an embodiment of the present disclosure is schematically shown.

[0105] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for executing different actions of the method flow according to an embodiment of the present disclosure.

[0106] Various programs and data required for the operation of the electronic device 600 are stored in the RAM 603. The processor 601, ROM 602, and RAM 603 are connected to each other via a bus 604. The processor 601 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than the ROM 602 and RAM 603. The processor 601 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0107] According to an embodiment of the present disclosure, electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to bus 604. Electronic device 600 may further include one or more of the following components connected to I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. Communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in drive 610 as needed, so that computer programs read from the removable media can be installed into storage section 608 as needed.

[0108] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0109] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0110] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the parallel graph computing processing method provided by the embodiments of the present disclosure.

[0111] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 601 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0112] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0113] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from a removable medium 611. When the computer program is executed by the processor 601, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0114] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0117] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A parallel graph computing method, comprising: Grouping the graph data set of the processing unit array based on a preset grouping rule to obtain a destination vertex set for each row of processing units in the processing unit array and a plurality of source vertex sets corresponding to each destination vertex set; For each processing unit in each row of processing units, using the distribution scheduler in the processing unit, the subgraph destination vertex set, the subgraph source vertex set, and the coincident vertex set corresponding to the processing unit are stored in the corresponding storage unit, wherein the coincident vertex set is a set of vertices that coincide with the subgraph destination vertex set and the subgraph source vertex set; Using a computing module in the processing unit, the edge data corresponding to the processing unit in the storage unit and the graph data set are read respectively to obtain initial vertex weights corresponding to source vertices and destination vertices of each vertex set in the storage unit, as well as edge weights of the edge data; Based on the edge weights and the vertex weights, using the calculation module to determine a single-source shortest path between the source vertex and the destination vertex, and obtain target vertex weights corresponding to the subgraph destination vertex set and the coincident vertex set; The calculation module is used to update the target vertex weight to the corresponding storage unit.

2. The method according to claim 1, wherein The method of grouping the graph data set of the processing unit array based on a preset grouping rule to obtain a destination vertex set for each row of processing units in the processing unit array and a plurality of source vertex sets corresponding to each destination vertex set includes: Based on vertex numbers of multiple destination vertices in the graph data set, the destination vertices are evenly grouped using a hash grouping rule to obtain a destination vertex set for each row of processing units in the processing unit array; Based on the total in-degree of the destination vertices in each of the destination vertex sets and a preset number of groups, multiple source vertices corresponding to the destination vertex set are evenly grouped to obtain multiple source vertex sets.

3. The method according to claim 2, wherein: The step of uniformly grouping the multiple source vertices corresponding to the destination vertex set based on the total in-degree of the destination vertices in each destination vertex set and the preset number of groups to obtain the multiple source vertex sets includes: Calculate the target out-degree of each source vertex set based on the total in-degree and the preset grouping; The plurality of source vertices are evenly grouped based on the target out-degree to obtain a plurality of source vertex sets.

4. The method according to claim 1, wherein The utilizing the distribution scheduler in the processing unit to store the subgraph destination vertex set, the subgraph source vertex set, and the coincident vertex set corresponding to the processing unit into the corresponding storage unit includes: The distribution scheduler is used to store the subgraph destination vertex set in an on-chip static destination storage unit, store the subgraph source vertex set in an on-chip static source storage unit, and store the coincident vertex set in an off-chip static coincidence storage unit.

5. The method according to claim 4, wherein The updating of the target vertex weight to the corresponding storage unit by using the calculation module includes: Using the calculation module, the target vertex weights corresponding to the subgraph destination vertex set are updated to the on-chip static destination storage unit; When the target vertex weights corresponding to the coincident vertices in the coincident vertex set are less than the initial vertex weights, the target vertex weights corresponding to the coincident vertex set are updated to the off-chip static coincidence storage unit by using the calculation module.

6. The method according to claim 1, further comprising: For each row of processing units, when each processing unit completes the vertex weight update, select the fastest converging processing unit in the row of processing units, and store the iteration destination vertex set and iteration source vertex set corresponding to the fastest converging processing unit in an off-chip storage unit; For each iteration destination vertex set and iteration source vertex set in the off-chip storage unit, using the calculation module, iteratively process the iteration destination vertex set and the iteration source vertex set to obtain a new iteration destination vertex set and a new iteration source vertex set; When the difference in vertex weights of corresponding vertices between the new iteration destination vertex set and the new iteration source vertex set generated in the i-th and i+1-th iterations respectively is less than a preset value, the new iteration destination vertex set and the new iteration source vertex set corresponding to the i-th iteration are determined as the graph calculation results.

7. A parallel graph computing and processing device, comprising: A grouping module, configured to group the graph data set of the processing unit array based on a preset grouping rule, to obtain a destination vertex set for each row of processing units in the processing unit array, and a plurality of source vertex sets corresponding to each destination vertex set; a storage module configured to store, for each processing unit in each row of processing units, a subgraph destination vertex set, a subgraph source vertex set, and a coincident vertex set corresponding to the processing unit in a corresponding storage unit using a distribution scheduler in the processing unit, wherein the coincident vertex set is a set of vertices that coincide with the subgraph destination vertex set and the subgraph source vertex set; a reading module, configured to use the calculation module in the processing unit to read the edge data corresponding to the processing unit in the storage unit and the graph data set respectively, to obtain the initial vertex weights corresponding to the source vertex and the destination vertex of each vertex set in the storage unit, and the edge weights of the edge data; a calculation module, configured to determine, based on the edge weights and the vertex weights, a single-source shortest path between the source vertex and the destination vertex using the calculation module, and obtain target vertex weights corresponding to the subgraph destination vertex set and the coincident vertex set; An updating module is used to update the target vertex weight to a corresponding storage unit using the calculation module.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images