Incremental computation optimization method and hardware accelerator for dynamic graph point pair analysis

Abstract

The application provides a dynamic graph point pair analysis incremental calculation optimization method, which performs edge adding and edge deleting operations through a preset hardware accelerator; the hardware accelerator contains pre-fetching, identification and scheduling, and propagation modules, adopts a pipeline technology to process multiple batches of dynamic changes in parallel; when the edge adding operation is performed, whether the source point is reachable to the edge adding starting point is judged, the edge topology change is recorded only if the source point is not reachable, and no state calculation is performed; the topology change is recorded and the edge adding ending point topology reachable vertex state is updated if the source point is reachable; when the edge deleting operation is performed, whether the edge deleting ending point state depends on the edge deleting starting point state is judged, the edge is deleted only if the edge deleting ending point state does not depend on the edge deleting starting point state, and the state update is not triggered; the edge topology change is recorded and the edge deleting ending point topology reachable vertex state is updated if the edge deleting ending point state depends on the edge deleting starting point state. The application also provides a corresponding hardware accelerator. Therefore, the application can effectively reduce the number of activated (traversed) vertices, thereby greatly reducing the calculation cost; and the preset hardware accelerator can further improve the processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of dynamic graph analysis technology, and in particular to an incremental computation optimization method and hardware accelerator for dynamic graph point pair analysis. Background Technology

[0002] A dynamic graph is a graph that changes over time. The changes in the graph originate from the changes in the edges between vertices; therefore, each time, it is the edges in the graph that change. A vertex pair refers to a graph with a source vertex and a sink vertex, which will be represented by S (source) and D (destination) in this paper. For a directed graph, each edge has a starting point and a ending point, which can be represented by u and v, respectively. The state of a vertex u is called stat(u). Taking the SSSP (Single Source Shortest Path) algorithm as an example, stat(u) represents the shortest path distance from u to S.

[0003] Currently, the main method for accelerating updates to dynamic graphs is incremental computation. For each change, the state of the vertices is updated in the current graph snapshot. By making the number of active vertices only related to the change, unnecessary state propagation in computation is reduced, which can reduce the time spent on each update.

[0004] Changes to a dynamic graph generally refer to adding and deleting edges. In existing incremental computation methods, it's essential to recalculate the states of all nodes affected by the topology. For adding edges...<u,v> In contrast, naive incremental computation starts from vertex v, calculates stat(v), and simultaneously calculates stat(*) for all vertices affected by v. For edge deletion, any vertex topologically reachable from the deleted edge to vertex v needs to start from stat(v) and recalculate its state. Furthermore, calculating the state stat(*) of each vertex requires traversing the states of all vertices pointing to it, a process that is extremely costly. This problem arises because there is no more refined filtering of vertex dependencies, leading to the propagation of unnecessary states. Therefore, reducing unnecessary computation and lowering overhead is key to improving the execution efficiency of dynamic graph analysis algorithms.

[0005] Existing incremental computation methods for processing dynamic graphs generate the activation of many redundant vertices during the computation process, leading to the propagation of a large number of invalid state values. Experimental results from multiple dynamic graph analysis algorithms show that, on average, 60% of the computation is redundant, a significant proportion relative to the total runtime load.

[0006] In conclusion, the existing technology obviously has inconveniences and defects in practical use, so it is necessary to improve it. Summary of the Invention

[0007] To address the aforementioned shortcomings, the present invention aims to provide an incremental computation optimization method and hardware accelerator for dynamic graph point pair analysis, which solves the problem of reducing redundant computation during incremental computation in dynamic graphs.

[0008] To address the aforementioned technical problems, this invention provides, on the one hand, an incremental calculation optimization method for dynamic graph point pair analysis, comprising the following steps:

[0009] Edge addition and edge deletion operations are performed through a preset hardware accelerator; the hardware accelerator includes a prefetching module, an identification and scheduling module, and a propagation module, and the hardware accelerator uses pipeline technology to process multiple dynamically changing batches in parallel.

[0010] During the edge-adding operation, it is determined whether the source vertex is reachable from the starting point of the edge-adding operation. If it is not reachable, only the topological change of the edge in the edge-adding operation is recorded without performing state calculation. If it is reachable, the topological change of the edge in the edge-adding operation is recorded and the state of the vertices that are topologically reachable from the endpoint of the edge-adding operation is updated.

[0011] During the edge deletion operation, it is determined whether the state of the edge deletion endpoint depends on the state of the edge deletion starting point. If not, only the deletion of the edge in the edge deletion operation is recorded without triggering a state update. If so, the topological changes of the edge in the edge deletion operation are recorded and the state of the topologically reachable vertices of the edge deletion endpoint is updated.

[0012] The state is represented by the shortest path distance from the source point to the point.

[0013] Furthermore, the prefetching module is used to dynamically load graph data and change batches; the identification and scheduling module classifies dynamic changes based on vertex reachability and dependencies; and the propagation module performs state update calculations based on the classification results.

[0014] Furthermore, determining whether the state of the edge deletion endpoint depends on the state of the edge deletion starting point includes:

[0015] Determine whether the optimal value of stat(v) of the edge deletion endpoint directly depends on stat(u) of the edge deletion starting point. If so, determine that the state of the edge deletion endpoint depends on the state of the edge deletion starting point. Here, stat(v) represents the shortest path distance from the source point to the edge deletion endpoint v, and stat(u) represents the shortest path distance from the source point to the edge deletion starting point u.

[0016] Furthermore, the pipeline technology includes assigning the edge-adding operation and the edge-removing operation to different processing units to achieve parallel processing of multiple batches of dynamically changing operations.

[0017] On the other hand, the present invention also provides a hardware accelerator, which is applied to implement the incremental computation optimization method for dynamic graph point pair analysis as described in any of the above claims; the hardware accelerator includes:

[0018] The prefetch module is used to preload dynamic graph data and change batches;

[0019] The identification and scheduling module is used to classify dynamic changes based on vertex reachability and critical upstream dependencies, and to allocate computational tasks.

[0020] The propagation module is used to perform state update calculations; the pipeline processing unit is used to process multiple dynamically changing batches in parallel.

[0021] This invention performs edge addition and deletion operations using a pre-defined hardware accelerator. This hardware accelerator includes prefetching, identification, scheduling, and propagation modules, employing pipeline technology to process multiple batches of dynamically changing vertices in parallel. During edge addition, it determines whether the source vertex is reachable from the edge addition starting point. If not, it only records the edge topology change without performing state calculations; if reachable, it records the topology change and updates the reachable vertex state of the edge addition endpoint. During edge deletion, it determines whether the edge deletion endpoint state depends on the edge deletion starting point state. If not, it only records the edge deletion without triggering a state update; if so, it records the edge topology change and updates the reachable vertex state of the edge deletion endpoint. This invention also provides a corresponding hardware accelerator. Therefore, this invention can effectively reduce the number of activated (traversed) vertices, thereby significantly reducing computational overhead; and based on the pre-defined hardware accelerator, it can further improve processing efficiency. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an incremental calculation optimization method for dynamic graph point pair analysis according to an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the hardware accelerator provided in one embodiment of the present invention;

[0024] Figure 3 This diagram illustrates the speedup ratio between the incremental calculation optimization method for dynamic graph point pair analysis provided in this embodiment of the invention and the traditional ColdStart method.

[0025] Figure 4 This diagram illustrates the ratio of the incremental calculation optimization method for dynamic graph point pair analysis provided in this embodiment of the invention to the traditional ColdStart method in terms of the number of activated vertices. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0027] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics, but not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0028] Furthermore, certain terms are used in the specification and subsequent claims to refer to specific components or parts. Those skilled in the art will understand that manufacturers may use different names or terms to refer to the same component or part. This specification and subsequent claims do not distinguish components or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Additionally, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections made through other means.

[0029] Before describing the embodiments of this application in detail, the technical concept of this application is briefly described first: In the edge addition operation, by determining whether the source vertex S can reach the starting point u of the newly added edge, the state update is triggered only for changes that may affect the path from S to the sink vertex D, thus avoiding invalid calculations; In the edge deletion operation, based on the "key upstream" dependency relationship between vertices, only the vertices directly affected by the deleted edge are updated, reducing redundant propagation; At the same time, an accelerator composed of a prefetch module (data loading), an identification and scheduling module (change classification), and a propagation module (state calculation) is designed to support pipeline technology, realize parallel processing of multiple batches of dynamic changes, and break through the performance bottleneck of general-purpose CPU computing.

[0030] The specific principle of the incremental calculation optimization method for dynamic graph point pair analysis of this application will be described below with reference to specific embodiments. For ease of understanding, in a changed edge, the vertex that sends out the edge is called the starting point, and the vertex that is pointed to is called the ending point. In the entire dynamic graph analysis process, only one point pair is set globally, and the result is the distance from the source point to the sink point.

[0031] Figure 1This invention illustrates an incremental calculation optimization method for dynamic graph point pair analysis, comprising the following steps:

[0032] S101: Edge addition and deletion operations are performed using a pre-defined hardware accelerator. The hardware accelerator includes a prefetch module, an identification and scheduling module, and a propagation module, and employs pipeline technology to process multiple dynamically changing batches in parallel. In this embodiment, the pre-defined hardware accelerator improves batch processing speed. Since the CPU itself does not have specific optimizations for graph data processing, this hardware accelerator is designed for dynamic graph analysis. It consists of a prefetch module, an identification and scheduling module, and a propagation module, and incorporates pipeline technology to process multiple changes simultaneously.

[0033] The prefetching module is used to dynamically load graph data and change batches; the identification and scheduling module classifies dynamic changes based on vertex reachability and dependencies; and the propagation module performs state update calculations based on the classification results.

[0034] Furthermore, the pipeline technology includes allocating the edge-adding and edge-removing operations to different processing units to achieve parallel processing of multiple batches of dynamically changing data. Through staged parallel processing, the pipeline technology allows multiple dynamically changing batches to "flow" through the accelerator, with each batch sequentially undergoing prefetching, analysis, and updating stages. The tasks of different batches overlap in time, thereby maximizing hardware utilization and processing speed.

[0035] S102: During the edge addition operation, determine whether the source vertex is reachable from the starting point of the edge addition operation; if it is not reachable, only record the topological changes of the edge in the edge addition operation without performing state calculation; if it is reachable, record the topological changes of the edge in the edge addition operation and update the state of the vertices that are topologically reachable from the endpoint of the edge addition operation.

[0036] In practice, regarding the edge-adding operation<u,v> Let u be the starting point of the edge addition operation, and v be the ending point of the edge addition operation. Determine whether the source vertex S is reachable from the starting point u; if not, only record the edge.<u,v> For topological changes, no state calculation is performed; if reachable, the added vertex v and its topologically reachable vertices are activated for state updates; S is a globally fixed vertex: S is the preset source vertex in the entire dynamic graph analysis, and is the starting point of the vertex-to-sink analysis (S to D), independent of the specific edge addition operation. In this embodiment, changes are classified according to the stat(u) of the added vertex u; since the changes are random, many added edges...<u,v> In the middle, S is unreachable from u, therefore<u,v> The addition of edge u does not affect stat(v); therefore, for a vertex, if it is unreachable from S to this vertex u, the change of edge u is recorded on the graph topology, but no calculation is performed; otherwise, not only is the change recorded, but the state of the topologically reachable vertices is also calculated. The execution effect obtained is that by using the judgment and filtering of edge addition, the number of vertices traversed when processing a batch of edge additions is greatly reduced.

[0037] S103: During the edge deletion operation, determine whether the state of the edge deletion endpoint depends on the state of the edge deletion starting point; if not, only record the deletion of the edge in the edge deletion operation without triggering a state update; if so, record the topological changes of the edge in the edge deletion operation and update the state of the topologically reachable vertices of the edge deletion endpoint; wherein, the state is represented by the shortest path distance from the source point to the point.

[0038] In practice, regarding the edge deletion operation<u,v> Determine whether the state stat(v) of vertex v depends on the state stat(u) of vertex u; if not, only record the edges.<u,v> Deleting an edge does not trigger a state update; if so, vertex v and its topologically reachable vertices are activated for a state update. This embodiment determines whether a state update is needed based on the dependency relationship between the deleted edge u and v. For a given pair of adjacent vertices...<u,v> If finding the optimal stat(v) is not directly related to stat(u), then u is not considered a critical upstream neighbor of v, and therefore can be removed.<u,v> Furthermore, it will not affect stat(v). Since the state stat(v) of node v is only related to at most one in-degree vertex during the convergence of all nodes in the graph, the deletion process does not affect the value of stat(v) for edges pointed to by non-critical upstream vertices. Based on this, it can be clearly stated that during the processing of an edge deletion, as long as the traversed u is not a critical upstream vertex of v, changing the stat of u will not have any impact on v. The resulting execution effect is that by adopting the judgment based on the critical upstream vertex, this embodiment greatly reduces the number of vertices traversed during the processing of a batch of edge deletions, and does not require recalculating the edge deletion.<u,v> In this context, the stat of all topologically reachable vertices is obtained, which greatly reduces computational overhead.

[0039] In this embodiment, determining whether the state of the edge deletion endpoint depends on the state of the edge deletion starting point includes:

[0040] Determine whether the stat(v) of the edge deletion endpoint directly depends on the optimal value of the stat(u) of the edge deletion starting point. If so, determine that the state of the edge deletion endpoint depends on the state of the edge deletion starting point. Here, stat(v) represents the shortest path distance from the source point to the edge deletion endpoint v, and stat(u) represents the shortest path distance from the source point to the edge deletion starting point u.

[0041] Compared to existing incremental calculation methods, the optimized algorithm used in this embodiment can greatly reduce the number of activated (traversed) vertices, thereby significantly reducing computational overhead. At the same time, due to the design of a dedicated accelerator, the optimized algorithm can be further reduced in time compared to running on a CPU, thus further increasing efficiency.

[0042] join Figures 3-4The incremental computation optimization method of this embodiment was specifically tested on directed graph algorithms such as SSSP (Single Source Shortest Path), SSNP (Single Source Narrowest Path), SSWP (Single Source Widest Path), Viterbi, and Reach (Single Source Reachable Path). Comparisons with existing methods in terms of speedup and number of activated vertices show that the method provided in this embodiment has a speedup many times greater than the existing ColdStart method, and the optimized method also has significantly fewer vertices than the existing ColdStart method. The ColdStart method refers to performing a complete computation after the change in state at the previous time step (e.g., redoing SSSP) for each batch of changes to obtain timely results.

[0043] Figure 2 This invention illustrates a hardware accelerator provided in an embodiment of the present invention. The hardware accelerator is applied to implement an incremental computation optimization method for dynamic graph point pair analysis as described above. The hardware accelerator includes a prefetch module 10, an identification and scheduling module 20, and a propagation module 30, wherein:

[0044] The prefetch module 10 is used to preload dynamic graph data and change batches; the identification and scheduling module 20 is used to classify dynamic changes based on vertex reachability and critical upstream dependencies, and allocate computational tasks; the propagation module 30 is used to perform state update calculations; and the pipeline processing unit is used to process multiple dynamic change batches in parallel. The dedicated accelerator provided in this embodiment further reduces the time required for the optimization algorithm to run on the CPU, thus further increasing its efficiency.

[0045] In summary, this invention performs edge addition and deletion operations using a pre-defined hardware accelerator. This hardware accelerator includes prefetching, identification, scheduling, and propagation modules, employing pipeline technology to process multiple batches of dynamically changing data in parallel. During edge addition, it determines whether the source vertex is reachable from the edge addition starting point. If not, it only records the edge topology change without performing state calculations; if reachable, it records the topology change and updates the topology reachable vertex state of the edge addition starting point. During edge deletion, it determines whether the edge deletion endpoint state depends on the edge deletion starting point state. If not, it only records the edge deletion without triggering a state update; if so, it records the edge topology change and updates the topology reachable vertex state of the edge deletion starting point. This invention also provides a corresponding hardware accelerator. Therefore, this invention can effectively reduce the number of activated (traversed) vertices, thereby significantly reducing computational overhead; and based on the pre-defined hardware accelerator, it can further improve processing efficiency.

[0046] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including associated data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of the present invention can be implemented in hardware, for example, as circuitry that works with a processor to perform the various steps or functions.

[0047] This invention can be implemented on a computer as a computer-based method, or in dedicated hardware, or a combination of both. Executable code or portions thereof for the method according to the invention can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Optionally, the computer program product includes non-transitory program code components stored on a computer-readable medium so as to execute the method according to the invention when the program product is executed on a computer.

[0048] In an optional embodiment, the computer program includes computer program code components adapted to perform all the steps of the method according to the invention when the computer program is run on a computer. Optionally, the computer program is embodied on a computer-readable medium.

[0049] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0050] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. An incremental calculation optimization method for dynamic graph point pair analysis, characterized in that, Including the following steps: Edge addition and edge deletion operations are performed through a preset hardware accelerator; the hardware accelerator includes a prefetching module, an identification and scheduling module, and a propagation module, and the hardware accelerator uses pipeline technology to process multiple dynamically changing batches in parallel. When performing the edge-adding operation, it is determined whether the source vertex is reachable from the edge-adding starting point in the edge-adding operation; if it is not reachable, only the topological change of the edge in the edge-adding operation is recorded without performing state calculation; if it is reachable, the topological change of the edge in the edge-adding operation is recorded and the state of the vertices that are topologically reachable from the edge-adding starting point is updated. When performing the edge deletion operation, it is determined whether the state of the edge deletion endpoint depends on the state of the edge deletion starting point; if not, only the deletion of the edge in the edge deletion operation is recorded without triggering a state update; if so, the topological changes of the edge in the edge deletion operation are recorded and the state of the topologically reachable vertices of the edge deletion endpoint is updated. The state is represented by the shortest path distance from the source point to the point.

2. The incremental calculation optimization method for dynamic graph point pair analysis according to claim 1, characterized in that, The prefetching module is used to dynamically load graph data and change batches; the identification and scheduling module classifies dynamic changes based on vertex reachability and dependency relationships. The propagation module performs state update calculations based on the classification results.

3. The incremental calculation optimization method for dynamic graph point pair analysis according to claim 1, characterized in that, The determination of whether the state of the edge deletion endpoint depends on the state of the edge deletion starting point includes: Determine whether the optimal value of stat(v) of the edge deletion endpoint directly depends on stat(u) of the edge deletion starting point. If so, determine that the state of the edge deletion endpoint depends on the state of the edge deletion starting point. Here, stat(v) represents the shortest path distance from the source point to the edge deletion endpoint v, and stat(u) represents the shortest path distance from the source point to the edge deletion starting point u.

4. The incremental calculation optimization method for dynamic graph point pair analysis according to claim 1, characterized in that, The pipeline technology includes assigning the edge-adding operation and the edge-removing operation to different processing units to achieve parallel processing of multiple batches of dynamically changing operations.

5. A hardware accelerator, characterized in that, The hardware accelerator is used to implement the incremental computation optimization method for dynamic graph point pair analysis as described in any one of claims 1 to 4; the hardware accelerator includes: The prefetch module is used to preload dynamic graph data and change batches; The identification and scheduling module is used to classify dynamic changes based on vertex reachability and critical upstream dependencies, and to allocate computational tasks. The propagation module is used to perform state update calculations; The pipeline processing unit is used to process multiple dynamically changing batches in parallel.

Images