Incremental calculation optimization method for dynamic graph point pair analysis and hardware accelerator

Abstract

The invention provides an incremental calculation optimization method for dynamic graph point pair analysis. The method comprises the following steps: executing edge adding and edge deleting operations through a preset hardware accelerator; the hardware accelerator comprises a prefetching module, an identification module, a scheduling module and a propagation module, and adopts an assembly line technology to process multi-batch dynamic changes in parallel. During edge adding operation, whether a source point can reach an edge adding starting point or not is judged, only edge topology changes are recorded if the source point cannot reach the edge adding starting point, and state calculation is not conducted; if yes, recording the topology change and updating the reachable vertex state of the edge-adding end point topology; during edge deletion operation, judging whether the edge deletion end point state depends on the edge deletion starting point state, if not, only recording edge deletion, and not triggering state update; and if yes, recording the change of the edge topology, and updating the edge deletion end point topology to reach a vertex state. The invention further provides a corresponding hardware accelerator. Therefore, the number of activated (traversed) vertexes can be effectively reduced, so that the calculation overhead is reduced to a great extent; and the processing efficiency can be further improved based on the preset hardware accelerator.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic graph analysis, and particularly to an incremental calculation optimization method and a hardware accelerator for dynamic graph point pair analysis. Background Art

[0002] A dynamic graph refers to a graph that changes over time. The changes in the graph come from the changes in the edges between vertices. Therefore, each time it is the edges in the graph that change. A point pair refers to setting a source point and a sink point in an entire graph, which will be represented by S (source) and D (destination) respectively in this article; for a directed graph, each edge has a starting point and an ending point, which can be represented by u and v respectively, and the state of a vertex u is called stat(u). Taking the SSSP (Single-Source Shortest Path) algorithm as an example, stat(u) represents the shortest path distance from u to S.

[0003] Currently, the acceleration of dynamic graph updates mainly uses the method of incremental calculation. For each change, the state of vertices is updated in the current snapshot of the graph. By making the number of activated vertices only related to the change, unnecessary state propagation and calculation can be reduced, and the time spent on each update can be reduced.

[0004] For the changes in a dynamic graph, generally, they refer to the operations of adding edges and deleting edges. In the calculation process of the existing incremental calculation method, it is necessary to recalculate the states of all nodes affected by the topology. For adding an edge <u, v>, the naive incremental calculation starts from vertex v, calculates stat(v), and at the same time calculates stat(*) for all vertices affected by v; for deleting an edge, any vertex that is topologically reachable from the vertex v pointed to by the deleted edge needs to start from stat(v) and recalculate their states. Moreover, when calculating the state stat(*) of each vertex, it is necessary to traverse the states of all vertices pointing to it. This process is extremely costly. The reason for this problem is that the dependency relationship of vertices is not screened more refinedly, resulting in the propagation of some unnecessary states. Therefore, how to reduce unnecessary calculations and thus reduce the overhead is the key to improving the execution efficiency of dynamic graph analysis algorithms.

[0005] Since the existing incremental calculation method for processing dynamic graphs will activate many redundant vertices during the calculation process, and a large number of invalid state values will be propagated according to these vertices. According to the experimental results of multiple dynamic graph analysis algorithms, on average, 60% of the calculations are redundant during the calculation process, and this proportion is relatively large compared to the total running load.

[0006] In summary, it is obvious that the existing technology has inconveniences and defects in actual use, so it is necessary to improve it. Summary of the Invention

[0007] In view of the above defects, the purpose of the present invention is to provide an incremental calculation optimization method and a hardware accelerator for dynamic graph point pair analysis, which are used to solve the problem of how to reduce redundant calculations in the incremental calculation process of dynamic graphs.

[0008] To solve the above technical problems, on the one hand, the present invention provides an incremental calculation optimization method for dynamic graph point pair analysis, including the steps of:

[0009] Executing edge addition operations and edge deletion operations through a preset hardware accelerator; the hardware accelerator includes a prefetch module, an identification and scheduling module, and a propagation module, and the hardware accelerator uses pipeline technology to parallelly process multiple dynamically changing batches;

[0010] During the execution of the edge addition operation, determine whether the source point is reachable to the start point of the edge addition in the edge addition operation; if not reachable, only record the topological change of the edge in the edge addition operation without performing state calculation; if reachable, record the topological change of the edge in the edge addition operation and update the state of the vertices that are topologically reachable to the end point of the edge addition;

[0011] During the execution of the edge deletion operation, determine whether the state of the end point of the edge deletion depends on the state of the start point of the edge deletion; if not, only record the deletion of the edge in the edge deletion operation without triggering state update; if so, record the topological change of the edge in the edge deletion operation and update the state of the vertices that are topologically reachable to the end point of the edge deletion;

[0012] Among them, the state is represented as the shortest path distance from the source point to the point.

[0013] Furthermore, the prefetch module is used to dynamically load graph data and change batches; the identification and scheduling module classifies dynamic changes based on the reachability and dependency relationships of vertices; the propagation module performs state update calculations according to the classification results.

[0014] Furthermore, the determination of whether the state of the end point of the edge deletion depends on the state of the start point of the edge deletion includes:

[0015] Determine whether the optimal value of stat(v) of the end point of the edge deletion directly depends on stat(u) of the start point of the edge deletion. If so, it is determined that the state of the end point of the edge deletion depends on the state of the start point of the edge deletion; where stat(v) represents the shortest path distance from the source point to the end point v of the edge deletion, and stat(u) represents the shortest path distance from the source point to the start point u of the edge deletion.

[0016] Furthermore, the pipeline technology includes allocating the edge addition operation and the edge deletion operation to different processing units respectively to achieve parallel processing of multiple batches of dynamic changes.

[0017] On the other hand, the present invention also provides a hardware accelerator, which is applied to implement the incremental calculation optimization method for dynamic graph point-to-point analysis as described in any one of the above; the hardware accelerator includes:

[0018] A prefetch module for preloading dynamic graph data and change batches;

[0019] An identification and scheduling module for classifying dynamic changes according to vertex reachability and critical upstream dependency relationships and allocating calculation tasks;

[0020] A propagation module for performing state update calculations; a pipelined processing unit for parallel processing of multiple dynamic change batches.

[0021] The present invention performs edge addition and edge deletion operations through a preset hardware accelerator; the hardware accelerator includes prefetch, identification and scheduling, and propagation modules, and uses pipeline technology to parallel process multiple batches of dynamic changes; during the edge addition operation, it is judged whether the source point is reachable to the edge addition start point. If not, only the edge topology change is recorded and no state calculation is performed; if reachable, the topology change is recorded and the topology reachable vertex state of the edge addition end point is updated; during the edge deletion operation, it is judged whether the state of the edge deletion end point depends on the state of the edge deletion start point. If not, only the edge deletion is recorded and no state update is triggered; if so, the edge topology change is recorded and the topology reachable vertex state of the edge deletion end point is updated. The present invention also provides a corresponding hardware accelerator. Thereby, the present invention can effectively reduce the number of activated (traversed) vertices, thereby greatly reducing the computational overhead; and based on the preset hardware accelerator, the processing efficiency can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic flowchart of the incremental calculation optimization method for dynamic graph point-to-point analysis provided by an embodiment of the present invention;

[0023] Figure 2 It is a schematic structural diagram of the hardware accelerator provided by an embodiment of the present invention;

[0024] Figure 3 It is a schematic diagram of the speedup ratio of the incremental calculation optimization method for dynamic graph point-to-point analysis provided by an embodiment of the present invention and the traditional ColdStart method;

[0025] Figure 4 It is a schematic diagram of the ratio of the number of activated vertices of the incremental calculation optimization method for dynamic graph point-to-point analysis provided by an embodiment of the present invention and the traditional ColdStart method. DETAILED DESCRIPTION OF THE INVENTION

[0026] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0027] It should be noted that the references to "one embodiment", "embodiment", "exemplary embodiment", etc. in this specification mean that the described embodiment may include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not refer to the same embodiment. Further, when combining embodiments to describe specific features, structures or characteristics, whether or not there is an explicit description, it has been shown that it is within the knowledge of those skilled in the art to combine such features, structures or characteristics into other embodiments.

[0028] In addition, in the specification and subsequent claims, certain terms are used to refer to specific components or parts. Those of ordinary skill in the art should understand that manufacturers may use different nouns or terms to refer to the same component or part. The specification and subsequent claims do not use the difference in name as a way to distinguish components or parts, but use the difference in function of components or parts as the criterion for distinction. The terms "comprising" and "including" mentioned throughout the specification and subsequent claims are open-ended terms, so they should be interpreted as "including but not limited to". In addition, the term "connected" herein includes any direct and indirect electrical connection means. Indirect electrical connection means include connection through other devices.

[0029] Before describing the embodiments of the present application in detail, first briefly describe the technical concept of the present application: In the border addition operation, by determining whether the source point S is reachable from the starting point u of the newly added edge, only the changes that may affect the path from S to the sink point D trigger status updates, avoiding invalid calculations; in the edge deletion operation, based on the "critical upstream" dependency relationship between vertices, only the part of the vertices directly affected by the deleted edge is updated, reducing redundant propagation; at the same time, an accelerator composed of a prefetch module (data loading), an identification and scheduling module (change classification), and a propagation module (status calculation) is designed to support pipeline technology and achieve parallel processing of multiple batches of dynamic changes, breaking through the performance bottleneck of general-purpose CPU computing.

[0030] Next, the specific principle of the incremental calculation optimization method for dynamic graph point-to-point analysis of the present application will be described in combination with specific embodiments; for the convenience of understanding, in a changed edge, the vertex that emits this edge is called the starting point, and the vertex that is pointed to is called the end point; and in the entire dynamic graph analysis process, only one point pair is set globally, and what is obtained is the path from the source point to the sink point.

[0031] Figure 1The incremental calculation optimization method for dynamic graph point pair analysis provided by an embodiment of the present invention is shown, and the steps are as follows:

[0032] S101: Execute the edge addition operation and the edge deletion operation through a preset hardware accelerator; the hardware accelerator includes a prefetch module, an identification and scheduling module, and a propagation module, and the hardware accelerator uses pipeline technology to parallel process multiple dynamically changing batches. That is, in this embodiment, the preset hardware accelerator is used to improve the batch processing speed; since the CPU itself does not have specific optimizations for graph data processing, this hardware accelerator belongs to an accelerator for processing dynamic graph analysis; it is composed of a prefetch module, an identification and scheduling module, and a propagation module, and pipeline technology is introduced to process multiple changes simultaneously.

[0033] Among them, the prefetch module is used to dynamically load graph data and change batches; the identification and scheduling module classifies dynamic changes based on the reachability and dependency relationships of vertices; the propagation module performs state update calculations according to the classification results.

[0034] Furthermore, the pipeline technology includes allocating the edge addition operation and the edge deletion operation to different processing units respectively to achieve parallel processing of multiple batches of dynamic changes. The pipeline technology enables multiple batches of dynamic changes to be "flowed" and executed in the accelerator through phased parallel processing. Each batch sequentially experiences stages such as prefetching, analysis, and updating, and the tasks of different batches overlap in time, thereby maximizing hardware utilization and processing speed;

[0035] S102: During the execution of the edge addition operation, determine whether the source point is reachable to the edge addition start point in the edge addition operation; if not reachable, only record the topological changes of the edges in the edge addition operation without performing state calculations; if reachable, record the topological changes of the edges in the edge addition operation and update the states of the vertices that are topologically reachable to the edge addition end point.

[0036] In specific implementation, for the edge addition operation <u, v>, where u is the starting point of the edge addition in the edge addition operation and v is the ending point of the edge addition in the edge addition operation, it is determined whether the source point S is reachable to the starting point u of the edge addition; if not reachable, only record the topological change of the edge <u, v> and do not perform state calculation; if reachable, activate the added vertex v and its topologically reachable vertices for state update; S is a globally fixed vertex: S is the preset source point in the entire dynamic graph analysis, is the starting point of the point pair analysis (from S to the sink point D), and is independent of the specific edge addition operation. In this embodiment, the changes are classified according to stat(u) of the vertex u where the edge is added; due to the randomness of the changes, in many added edges <u, v>, S is not reachable to u, so the addition of <u, v> cannot affect stat(v); therefore, for a vertex, if it is not reachable from S to this vertex u, then record the change of this edge topologically in the graph, but do not perform any calculation; otherwise, not only record the change, but also perform state calculation on the topologically reachable vertices; the obtained execution effect is that after the judgment and screening of the added edges, the number of vertices traversed when processing a batch of added edges is greatly reduced.

[0037] S103: In the execution of the edge deletion operation, determine whether the state of the ending point of the edge deletion depends on the state of the starting point of the edge deletion; if not, only record the deletion of the edge in the edge deletion operation and do not trigger state update; if so, record the topological change of the edge in the edge deletion operation and perform state update on the vertices topologically reachable to the ending point of the edge deletion; where the state is represented as the shortest path distance from the source point to the point.

[0038] In specific implementation, for the edge deletion operation <u, v>, it is determined whether the status stat(v) of vertex v depends on the status stat(u) of vertex u; if not, only the deletion of the edge <u, v> is recorded without triggering status update; if so, vertex v and its topologically reachable vertices are activated for status update; in this embodiment, according to the dependency relationship between u and v of the deleted edge, it is determined whether status update is required. For a given pair of adjacent vertices <u, v>, if there is no direct relationship between stat(v) and stat(u) when obtaining the optimal stat(v), then u is called not the key upstream vertex of v, so even if <u, v> is deleted, it will not have any impact on stat(v). Since during the convergence process of all nodes in the graph, the status stat(v) of node v is only related to at most one in-degree vertex, for the edge pointed to by a non-key upstream vertex, the deletion process does not affect the value of stat(v). Based on this, it can be clear that during the process of processing a deleted edge, as long as the traversed u is not the key upstream vertex of v, then changing the stat of u will not have any impact on v; the obtained execution effect is that by adopting the judgment based on the key upstream vertex, in this embodiment, during the process of processing a batch of deleted edges, the number of traversed vertices will be greatly reduced, and moreover, it is not necessary to recalculate the stat of all vertices that are topologically reachable from u in the deleted edge <u, v>, which greatly reduces the computational overhead.

[0039] This embodiment determines whether the status of the end point of the deleted edge depends on the status of the start point of the deleted edge, including:

[0040] Determine whether the stat(v) of the end point of the deleted edge directly depends on the optimal value of the stat(u) of the start point of the deleted edge. If so, it is determined that the status of the end point of the deleted edge depends on the status of the start point of the deleted edge; where stat(v) represents the shortest path distance from the source point to the end point v of the deleted edge, and stat(u) represents the shortest path distance from the source point to the start point u of the deleted edge.

[0041] Compared with the existing incremental calculation method, this embodiment can greatly reduce the number of activated (traversed) vertices by using the optimized algorithm, thereby greatly reducing the computational overhead; at the same time, due to the design of a dedicated accelerator, it can further reduce the running time of the optimized algorithm compared to running on the CPU and further increase the efficiency.

[0042] Participate Figures 3 to 4。The incremental calculation optimization method of this embodiment is specifically experimented on algorithms of directed graphs such as SSSP (Single-Source Shortest Path), SSNP (Single-Source Narrowest Path), SSWP (Single-Source Widest Path), Viterbi, and Reach. It can be seen from the comparison of the speedup ratio and the number of activated vertices with existing methods that the method provided in this embodiment has a speedup ratio several times that of the existing ColdStart method, and the number of vertices of the optimized method is also significantly less than that of the existing ColdStart method. Among them, the ColdStart method refers to performing a complete calculation after the change in the previous moment's state for each batch of changes (for example, re-doing SSSP once) to obtain timely results.

[0043] Figure 2 Figure 4 shows a hardware accelerator provided by an embodiment of the present invention. The hardware accelerator is applied to implement the incremental calculation optimization method for dynamic graph point-to-point analysis as described in any one of the above. The hardware accelerator includes a prefetch module 10, an identification and scheduling module 20, and a propagation module 30, where:

[0044] The prefetch module 10 is used to preload dynamic graph data and change batches. The identification and scheduling module 20 is used to classify dynamic changes according to vertex reachability and critical upstream dependency relationships and allocate calculation tasks. The propagation module 30 is used to perform state update calculations. The pipeline processing unit is used to parallel process multiple dynamic change batches. Through the dedicated accelerator provided in this embodiment, the optimization algorithm can be further reduced in running time and increased in efficiency compared to running on a CPU.

[0045] In summary, the present invention performs edge addition and edge deletion operations through a preset hardware accelerator. The hardware accelerator includes a prefetch, identification and scheduling, and propagation module, and uses pipeline technology to parallel process multiple batches of dynamic changes. During the edge addition operation, it is judged whether the source point is reachable to the edge addition start point. If not, only the edge topology change is recorded and no state calculation is performed. If reachable, the topology change is recorded and the topology reachable vertex state of the edge addition start point is updated. During the edge deletion operation, it is judged whether the state of the edge deletion end point depends on the state of the edge deletion start point. If not, only the edge deletion is recorded and no state update is triggered. If so, the edge topology change is recorded and the topology reachable vertex state of the edge deletion start point is updated. The present invention also provides a corresponding hardware accelerator. Thereby, the present invention can effectively reduce the number of activated (traversed) vertices, thereby greatly reducing the computational overhead. And based on the preset hardware accelerator, the processing efficiency can be further improved.

[0046] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware. For example, it can be implemented using an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, or a floppy disk and the like. Additionally, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with the processor to execute each step or function.

[0047] The present invention can be implemented on a computer as a computer-implemented method, or in dedicated hardware, or in a combination of both. The executable code or portions thereof for the method according to the present invention can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Optionally, the computer program product includes non-transitory program code components stored on a computer-readable medium for executing the method according to the present invention when the program product is executed on a computer.

[0048] In an alternative embodiment, the computer program includes computer program code components suitable for executing all steps of the method according to the present invention when the computer program is run on a computer. Optionally, the computer program is embodied on a computer-readable medium.

[0049] It should be noted that in this document, the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising such element. In addition, it should be pointed out that the scope of the method and device in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described method can be performed in an order different from that described, and various steps can be added, omitted, or combined. Additionally, the features described with reference to certain examples can be combined in other examples.

[0050] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and deformations according to the present invention. However, these corresponding changes and deformations should all fall within the protection scope of the appended claims of the present invention.

Claims

1. An incremental calculation optimization method for dynamic graph point pair analysis, characterized in that: Includes steps: The edge addition operation and the edge deletion operation are performed by a preset hardware accelerator; the hardware accelerator includes a pre-fetch module, an identification and scheduling module, and a propagation module, and the hardware accelerator uses pipeline technology to process multiple dynamically changing batches in parallel; When performing the edge adding operation, determine whether the source point is reachable to the edge adding starting point in the edge adding operation; if not, only record the topological change of the edge in the edge adding operation without performing state calculation; if reachable, record the topological change of the edge in the edge adding operation and update the state of the vertex that is topologically reachable from the edge adding starting point; When executing the edge deletion operation, determine whether the state of the edge deletion end point depends on the state of the edge deletion start point; if not, only record the edge deletion in the edge deletion operation without triggering a state update; if yes, record the topological change of the edge in the edge deletion operation and update the state of the vertices that are topologically reachable from the edge deletion end point; The state is represented by the shortest path distance from the source point to the point.

2. The incremental calculation optimization method for dynamic graph point pair analysis according to claim 1 is characterized in that: The pre-fetch module is used to dynamically load graph data and change batches; the identification and scheduling module classifies dynamic changes based on vertex reachability and dependency relationships; The propagation module performs state update calculations according to the classification results.

3. The incremental calculation optimization method for dynamic graph point pair analysis according to claim 1 is characterized in that: The step of determining whether the state of the edge deletion end point depends on the state of the edge deletion start point comprises: Determine whether the optimal value of stat(v) of the edge deletion end point directly depends on stat(u) of the edge deletion start point. If so, determine that the state of the edge deletion end point depends on the state of the edge deletion start point; wherein stat(v) represents the shortest path distance from the source point to the edge deletion end point v, and stat(u) represents the shortest path distance from the source point to the edge deletion start point u.

4. The incremental calculation optimization method for dynamic graph point pair analysis according to claim 1 is characterized in that: The pipeline technology includes allocating the edge adding operation and the edge deleting operation to different processing units respectively, so as to realize parallel processing of multiple batches with dynamic changes.

5. A hardware accelerator, characterized in that: The hardware accelerator is used to implement the incremental calculation optimization method for dynamic graph point pair analysis according to any one of claims 1 to 4; the hardware accelerator includes: Prefetch module, used to preload dynamic graph data and change batches; Identification and scheduling module, which is used to classify dynamic changes and allocate computing tasks according to vertex reachability and key upstream dependencies; The propagation module, which performs state update calculations; Pipeline processing unit for processing multiple dynamically changing batches in parallel.

Images