An inter-process data flow analysis method and device based on distributed graph computation

Abstract

The application discloses a method and device for inter-process data flow analysis based on distributed graph computing, which first receives an inter-process control flow graph of a program as input, and the node attribute of the control flow graph is program statement information; a user reserves an interface according to a specific data flow analysis; at the initial time of calculation, all nodes in the control flow graph are active nodes, and output data flow facts are initialized; a distributed graph computing system processes the active nodes in a parallel mode, iteratively analyzes according to the interface instance implemented by the user in a three-time message transmission mode until there is no active node in the system or the iteration times reach a user-specified threshold; and the distributed data flow analysis device based on the method is developed on an existing distributed graph computing engine, the device can run inter-process data flow analysis applications for large program codes on a large-scale cluster, and by providing a distributed processing framework for data flow analysis, the development threshold of using distributed technology and data flow analysis technology is reduced.

Description

Technical Field

[0001] This invention relates to distributed computing and data flow analysis technology, specifically designing an inter-process data flow analysis method and apparatus based on distributed graph computing. Background Technology

[0002] Data flow analysis originated from compiler optimization and is a technique for collecting program information at each point in the program execution flow. Besides forming the backbone of compiler optimization, data flow analysis is used in many other important applications, such as error detection, security vulnerability discovery, privacy analysis, and program understanding. In data flow analysis, each program point in the control flow graph maintains a separate data flow fact. Based on the function of each statement, a transfer function is applied along the control flow graph to transform the data flow fact accordingly. The transformation process is iteratively executed using a working set algorithm until a fixed point is reached, meaning that the data flow facts of all nodes in the graph no longer change.

[0003] While data flow analysis is important, it presents challenges for data flow analysis during the execution of code in large-scale systems. First, because modern programs are typically large-scale, maintaining a data flow solution for all program points within limited memory is not scalable. Furthermore, for some analyses, the data flow solution maintained at each program point is itself highly space-intensive. Existing work attempts to address this issue using sparse representations of program control flow graphs, but the enormous memory consumption still severely limits the scalability of the analysis. Second, flow-sensitive analysis requires executing transfer functions at every statement in the control flow graph to update the data flow facts. Considering that the number of transfer function executions is at least linearly related to the number of program statements and that the computation of each transfer function can be expensive, executing transfer functions for modern large-scale software is highly computationally intensive. Summary of the Invention

[0004] For inter-process data flow analysis and other scalable instances that can be instantiated as data flow analysis, this invention proposes an inter-process data flow analysis method and apparatus based on distributed graph computing. By leveraging the high efficiency, scalability, and fault tolerance advantages of distributed graph processing systems, the computational performance and scalability of inter-process data flow analysis are improved, while the development difficulty of static data flow analysis is reduced. Users only need to implement interface instances on the apparatus according to specific data flow analysis, and issues such as scalability and high performance are managed by the underlying distributed system.

[0005] To solve the above problems, the present invention is achieved through the following technical solution:

[0006] This invention discloses a method for inter-process data flow analysis based on distributed graph computing, characterized in that the method includes the following steps:

[0007] S1: Receives the inter-process control flow graph of the program to be analyzed as input. The node attributes in the control flow graph are the statement information at that node.

[0008] S2: Implement reserved interfaces based on specific data stream analysis;

[0009] S3: At the start of the computation, all nodes in the control flow graph are active nodes, and the output data flow is initialized.

[0010] S4: The distributed graph computing system processes active nodes in parallel, and performs iterative analysis based on the user-implemented interface instances according to a three-step message passing method, until there are no active nodes in the system or the number of iterations reaches the user-specified threshold.

[0011] The above-described method for inter-process data flow analysis based on distributed graph computing is characterized in that step S4 further includes the following steps:

[0012] S41: Distributed working set algorithm implemented using a distributed graph computing framework;

[0013] S42: If it is the first iteration, process the entry node in the control flow graph; otherwise, process only the active node in the current round.

[0014] S43: In each round of iteration, the node performs analysis by passing three messages, and at the same time determines whether to update and pass the output data stream facts based on the passing conditions.

[0015] S44: Repeat steps S42 to S43 until all nodes are inactive, i.e., there is no data flow in the system.

[0016] The above-described method for inter-process data flow analysis based on distributed graph computing is characterized in that the distributed working set algorithm implemented using the distributed graph computing framework in step S41 is based on a mapping of two algorithms: the vertex-centric computing model in the field of distributed graph computing and the working set algorithm in the field of program analysis.

[0017] The above-described method for inter-process data flow analysis based on distributed graph computing is characterized in that the distributed working set algorithm implemented using the distributed graph computing framework in step S41 is further optimized by a three-way message passing method.

[0018] The above-described method for inter-process data flow analysis based on distributed graph computing is characterized in that step S43 includes the following steps:

[0019] S431: In this round of iteration calculation, during the first message transmission, the active node that receives the message becomes the node to be analyzed in this round. The active node sends node identifiers to all predecessors to obtain the output data stream facts of all predecessors and then becomes an inactive node.

[0020] S432: During the second message transmission, the node that receives the subsequent message is an active node. Based on the information containing the successor node's identifier, it sends its output data stream facts to the successor and becomes an inactive node.

[0021] S433: During the third message transmission, the active node that receives the message becomes the node to be analyzed again. Based on the output data flow facts of all predecessors in the message, the analysis is performed according to the interface instance implemented by the user. If the transmission conditions are met, the output data flow facts of this node are updated and the node identifier is sent to the successor node to be analyzed in the next iteration. Then it becomes an inactive vertex.

[0022] The present invention also discloses an apparatus for inter-process data flow analysis based on distributed graph computing, characterized in that the apparatus comprises the following modules:

[0023] Input receiving module M1: Used to receive the inter-process control flow graph of the program to be analyzed as input, where the node attributes in the control flow graph are the statement information at that node;

[0024] Analysis Reserved Module M2: Used for: implementing reserved interfaces based on specific data flow analysis;

[0025] Analysis Loading Module M3: Used to: ensure all nodes in the control flow graph are active at the start of computation, and initialize the output data flow facts;

[0026] Analysis and computation module M4: Used by the distributed graph computing system to process active nodes in parallel, and to perform iterative analysis based on the user-implemented interface instance according to the three-way message passing method, until there are no active nodes in the system or the number of iterations reaches the user-specified threshold.

[0027] The aforementioned apparatus for inter-process data flow analysis based on distributed graph computing is characterized in that the analysis and computing module M4 further includes the following modules:

[0028] Computational Algorithm Module M41: Used for: Distributed working set algorithms implemented using a distributed graph computing framework;

[0029] Node processing module M42: Used to process the entry nodes in the control flow graph if it is the first iteration; otherwise, it only processes the active nodes in the current round.

[0030] Node analysis module M43: Used for: performing analysis on nodes in each iteration by passing three messages, and determining whether to pass data stream facts based on the passing conditions;

[0031] Repeat Execution Module M44: Used to repeatedly execute the steps of Node Processing Module M42 to Node Analysis Module M43 until all nodes are inactive, i.e., there is no data flow in the system.

[0032] The aforementioned apparatus for inter-process data flow analysis based on distributed graph computing is characterized in that the distributed working set algorithm implemented using the distributed graph computing framework in the computation table algorithm module M41 is based on a mapping of two algorithms: the vertex-centric computation model in the field of distributed graph computing and the working set algorithm in the field of program analysis.

[0033] The aforementioned apparatus for inter-process data flow analysis based on distributed graph computing is characterized in that the distributed working set algorithm implemented using a distributed graph computing framework in the computing algorithm module M41 is further optimized using a three-way message passing method.

[0034] The aforementioned apparatus for inter-process data flow analysis based on distributed graph computing is characterized in that the data flow judgment module M43 includes the following modules:

[0035] The first message passing module M431: In this round of iterative calculation, during the first message passing, the active node that receives the message becomes the node to be analyzed in this round. The active node sends node identifiers to all predecessors to obtain the output data stream facts of all predecessors and then becomes an inactive node.

[0036] Second message passing module M432: During the second message passing, the node that receives the subsequent message is an active node. Based on the information containing the successor node's identifier, it sends its output data stream to the successor and becomes an inactive node.

[0037] The third message passing module M433: During the third message passing, the active node that receives the message becomes the node to be analyzed again. Based on the output data flow facts of all predecessors in the message, the analysis is performed according to the interface instance implemented by the user. If the passing conditions are met, the output data flow facts of this node are updated and the node identifier is sent to the successor node to be analyzed in the next iteration. Then it becomes an inactive vertex.

[0038] By adopting the above technical solutions, the present invention can achieve the following beneficial effects:

[0039] 1. This invention, through the examination and combination of algorithms from different fields, proposes a novel parallel distributed working set algorithm tailored for inter-process data flow analysis, which leverages the advantages of distributed graph computing to make it possible to perform complex data flow analysis on large-scale program code.

[0040] 2. This invention proposes a three-step message passing method, which further optimizes the parallel distributed working list algorithm and further improves the scalability of the system.

[0041] 3. This invention implements the proposed new algorithm on an existing distributed graph computing engine, successfully solving the scalability problem faced by large-scale system code analysis, and improving the execution speed of data flow analysis while reducing memory consumption.

[0042] 4. This invention provides developers with a distributed processing framework for data flow analysis. By providing advanced interfaces, it lowers the development threshold for using distributed technology and data flow analysis technology, allowing general developers to enjoy the benefits brought by distributed data flow analysis technology. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the overall process of the method for inter-process data flow analysis based on distributed graph computing according to an embodiment of the present invention.

[0044] Figure 2 This is a flowchart illustrating the implementation of the three-way message passing method in an embodiment of the present invention.

[0045] Figure 3 This is an illustrative diagram illustrating an example of implementing a three-way message passing method according to an embodiment of the present invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] This embodiment relates to a distributed inter-process data flow analysis framework. This framework provides users with analysis interfaces and, through user-defined implementations, fully leverages the advantages of distributed graph processing systems. While ensuring analysis accuracy, it accelerates the analysis performance of static data flow analysis. Users only need to implement interface instances according to specific data flow analysis requirements; issues such as scalability and high performance are managed by the underlying system.

[0048] like Figure 1 As shown, Figure 1 This is a schematic diagram of the overall process of the method for inter-process data flow analysis based on distributed graph computing according to an embodiment of the present invention. The method includes the following steps:

[0049] S1: Receives the inter-process control flow graph of the program to be analyzed as input. The node attributes in the control flow graph are the statement information at that node.

[0050] S2: Implement reserved interfaces based on specific data stream analysis;

[0051] S3: At the start of the computation, all nodes in the control flow graph are active nodes, and the output data flow is initialized.

[0052] S4: The distributed graph computing system processes active nodes in parallel, and performs iterative analysis based on the user-implemented interface instances according to a three-step message passing method, until there are no active nodes in the system or the number of iterations reaches the user-specified threshold.

[0053] In step S1, the inter-process control flow graph of the program code is the input of this invention, generated based on the specific data flow analysis to be executed by the user. Depending on the input method of the actual distributed graph computing engine, personalized designs can be made, including loading the control flow graph as a set of nodes and edges, and inputting corresponding configuration files, etc.

[0054] In step S2, this invention draws inspiration from vertex-centric graph processing models, designing and implementing high-level interfaces developed based on a distributed framework. These high-level interfaces are tailored for data flow analysis during large-scale code execution. Similar to existing general-purpose graph systems, the user-friendly interfaces provided by this invention include the expression of key data structures in data flow analysis, such as data flow facts and statement information at nodes; as well as operations performed by nodes during data flow analysis, such as composition functions, transfer functions, and transfer condition judgment functions. Users can easily implement their own client-side analysis based on these interfaces without worrying about the underlying system management implementation.

[0055] In step S3, during distributed graph computation, the termination of the computation process is determined based on the activity status of the nodes in the graph. At the initial stage of the analysis, all nodes in the inter-process control flow graph are active nodes, and the output data flow facts need to be initialized for subsequent computations.

[0056] Step S4 is an iterative process, which specifically includes the following steps:

[0057] S41: Implement the distributed working set algorithm using a distributed graph computing framework;

[0058] S42: If it is the first iteration, process the entry points in the control flow graph; otherwise, only process the active nodes in the current round.

[0059] S43: In each iteration, the node performs analysis by passing three messages, and at the same time determines whether to pass the data stream fact based on the passing conditions.

[0060] S44: Repeat steps S42 to S43 until all nodes are inactive, meaning there is no data flow in the system.

[0061] Step S41 represents the mapping relationship of the algorithm and is a crucial support point for applying distributed graph computing technology to inter-process data flow analysis. The working set algorithm shares strong similarities with the vertex-centric computing model, providing a possibility for algorithm migration. The node statement information and output data information in the working set algorithm can correspond to the node attributes in the vertex-centric computing model; the nodes to be analyzed in the current working set can be the active nodes to be processed in graph computing, placed into the active node set stored during distributed graph computing; the iterative computation process of the former can be represented as the concept of each round step (i.e., superstep) in the iterative computation of the latter; the data flow method of the former can be implemented using the message transmission method of the latter; the computational logic of the former in solving data flow facts can be implemented by the execution of user-defined computation functions by nodes in the latter; the aggregation processing of data flow facts at the intersection of program paths in the former can also be completed by the aggregation and synchronization operation of multiple messages in the latter. In summary, we can establish a correspondence between the various parts of the two algorithms. Since the distributed graph computing framework is based on the vertex-centric computing model, we can implement the working set algorithm on a distributed graph computing system.

[0062] Step S42 will distinguish whether the active node to be processed is an entry node of the control flow graph. If it is an entry node, since there is no predecessor node, the transfer function can be directly applied to the initialized data flow fact to calculate the output data flow fact; otherwise, the active vertex will be processed in a more complex manner as described later.

[0063] In step S43, to improve performance without affecting correctness, the original vertex-centric model assumes that only updated vertices need to propagate messages. However, this assumption does not hold in the context of dataflow analysis, whose computation requires the output dataflow facts of all predecessor nodes. The most direct way to obtain all the necessary information to get the input dataflow facts is to maintain the output dataflow facts of all predecessor nodes locally on each node; however, this would lead to a large amount of redundant storage, severely impacting the system's scalability.

[0064] To address the memory redundancy issue in local storage of data stream facts, this invention designs a three-step message passing method, and further designs an optimized distributed working list algorithm based on active pull. Instead of maintaining redundant data stream facts locally at each vertex, it actively extracts data stream facts from all predecessor nodes when needed.

[0065] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the implementation of the three-message passing method in an embodiment of the present invention. Step S43 describes the steps involved in performing data flow analysis according to the three-message passing method in each iteration calculation of the node in the present invention, specifically including the following steps:

[0066] S431: During the first message passing in this round of iteration, the active node that receives the message becomes the node to be analyzed in this round. The active node sends its node identifier to all predecessors to obtain the output data stream facts of all predecessors, and then becomes an inactive node;

[0067] S432: During the second message transmission, the node that receives the subsequent message becomes the active node. The active node then sends its output data stream to the designated successor based on the message, and subsequently becomes an inactive node.

[0068] S433: During the third message transmission, the active node that receives the message is again the node to be analyzed. Based on the output data stream facts of all predecessors in the message, the user-implemented interface instance is executed for analysis; if the transmission conditions are met, the output data stream facts of this node are updated and the node identifier is sent to the successor node to be analyzed in the next iteration.

[0069] The following reference Figure 3 The calculation process of step S43 in this embodiment is given with reference to a specific example.

[0070] In step S431, the active node 3 in this iteration, acting as the node to be analyzed, receives the message sent by its predecessor 1 in the previous iteration, i.e., step S4331; this message is used to notify node 3 that the output data stream facts of node 1 have changed. In step S4332, node 3 retrieves the output data stream facts of all its predecessors 1 and 2 by sending its node identifier to them, i.e., the first message transmission, and at the same time, nodes 1 and 2 become active nodes.

[0071] In step S4321, the predecessor nodes 1 and 2 that receive the message are active nodes, and the message they receive contains the identifier of the successor 3 that requests the output data stream fact from them. In step S4322, nodes 1 and 2 push their output data stream fact to 3 based on the message, i.e., the second message transmission, and then their successor 3 becomes an active node again.

[0072] Steps S431 and S432 essentially accomplish one task: for the active vertex to be analyzed, it successfully retrieves all predecessor data stream facts from its predecessor node only when analysis is needed, thus avoiding redundant local storage of predecessor data stream facts. The overhead introduced by these two message passes is the transmission of node identifiers, which is efficient compared to the expensive overhead of locally storing data stream facts.

[0073] In step S433, the active node 3 that received the message is again the node to be analyzed. At this time, the message received by node 3 contains the output data stream facts of all predecessors 1 and 2. In step S4331, the interface instance of 3 uses it as input and performs merging, transmission, and propagation operations in sequence; then, based on the judgment of the data stream transmission condition by the propagation function, it decides whether to transmit its node identifier to successor nodes 4 and 5. Assuming that the transmission condition is met, that is, the output data stream facts of 3 have changed, as shown in step S4332, 3 updates its output data stream facts and sends the node identifier to successors 4 and 5. As a result, nodes 4 and 5 will jump to step S42 for judgment and then execute step S43.

[0074] The three message passing processes in step S43 above are considered as one round of analysis steps in the logical processing of this invention. For a distributed graph computing system, it is impossible to distinguish the differences between the three message sending processes. Therefore, we obtain the overstep count during the computation of the distributed graph computing system, and use the remainder after dividing by three to determine the current round of message passing. When a node executes its computation logic, it first determines the current round of message passing to decide which processing logic to execute.

[0075] In the specific implementation of step S431, after a node receives a message, it calculates through a counter that it belongs to the first message delivery node in the current processing stage. At this time, the received message comes from the predecessor node that has updated the data stream facts. Then, the currently active node will send its node identifier to all predecessors to request their output data stream facts. Therefore, the message is the current node identifier. Subsequently, it becomes an inactive node by calling the stop function.

[0076] In the specific implementation of step S432, the activated node is the predecessor node of the node to be analyzed, and a second message transmission is required. The message value at this stage is the identifier of the successor node that made the request. Subsequently, the active vertex will send its output data stream facts to these successors based on the message. Therefore, the message is its output data stream facts. Users can implement various message value types that support optimization options by implementing the message value interface.

[0077] In the specific implementation of step S433, the activated node is again the node to be analyzed. Through the calculation of the overstep count, it is determined that the current stage is the third message passing phase. The message value in this phase contains the data flow facts of all predecessor nodes. By specifying the user-implemented interface type, the corresponding merge function, pass function, and propagation function for specific data flow analysis will be executed. When the propagation function makes a judgment, if the data flow passing conditions are met, such as if the output data flow facts have changed, the newly calculated data flow facts are used to update the data flow, and the node identifier is sent to the successor node; at this time, the message value is its node identifier.

[0078] Therefore, this invention divides an analysis phase into three message passing processes, thereby eliminating the need to store large amounts of redundant information, significantly reducing computational memory consumption, and improving the analysis speed. Furthermore, the two additional message passing processes require almost no complex calculations, and the information contained in the transmitted messages is merely the node's identification number; the message communication overhead is extremely low, thus having virtually no significant impact on system performance.

[0079] The above description is merely a preferred embodiment of the present invention, but the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Any simple modifications, equivalent changes, and alterations made by those skilled in the art to the above embodiments based on the technical essence of the present invention without departing from the principles and spirit of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for inter-process dataflow analysis based on distributed graph computation, characterized in that, The method includes the following steps: S1: Receives the inter-process control flow graph of the program to be analyzed as input. The node attributes in the control flow graph are the statement information at that node. S2: Implement reserved interfaces based on specific data stream analysis; S3: At the start of the computation, all nodes in the control flow graph are active nodes, and the output data flow facts are initialized. S4: The distributed graph computing system processes active nodes in parallel, performing iterative analysis based on user-implemented interface instances using a three-step message passing method, until there are no active nodes in the system or the number of iterations reaches a user-specified threshold; the three-step message passing method includes: First message passing: The active node sends its node identifier to all predecessor nodes to obtain the output data stream facts of all predecessor nodes, and then becomes an inactive node; Second message passing: Upon receiving the request, the predecessor node sends its output data stream facts to the successor node based on the node identifier, and then becomes an inactive node; The third message passing: The node to be analyzed receives the output data stream facts of all predecessor nodes, executes the user-implemented interface instance for analysis, and if the passing conditions are met, updates the output data stream facts of this node and sends the node identifier to the successor node to be analyzed in the next iteration, and then becomes an inactive node.

2. The method for inter-process data flow analysis based on distributed graph computing according to claim 1, characterized in that, Step S4 further includes the following steps: S41: Distributed working set algorithm implemented using a distributed graph computing framework; S42: If it is the first iteration, process the entry node in the control flow graph; otherwise, process only the active node in this round. S43: In each round of iteration, the node performs analysis by passing three messages, and at the same time determines whether to update and pass the output data stream facts based on the passing conditions. S44: Repeat steps S42 to S43 until all nodes are inactive, i.e., there is no data flow in the system.

3. The method for inter-process data flow analysis based on distributed graph computing according to claim 2, characterized in that, The distributed working set algorithm implemented using the distributed graph computing framework in step S41 is based on a mapping of two algorithms: the vertex-centric computing model in the field of distributed graph computing and the working set algorithm in the field of program analysis.

4. The method for inter-process data flow analysis based on distributed graph computing according to claim 2, characterized in that, The distributed working set algorithm implemented using the distributed graph computing framework in step S41 is further optimized by employing a three-step message passing method.

5. An apparatus for inter-process data flow analysis based on distributed graph computing, characterized in that, The device includes the following modules: Input receiving module M1: Used to receive the inter-process control flow graph of the program to be analyzed as input, where the node attributes in the control flow graph are the statement information at that node; Analysis Reserved Module M2: Used for: implementing reserved interfaces based on specific data flow analysis; Analysis Loading Module M3: Used to: ensure all nodes in the control flow graph are active at the start of computation, and initialize the output data flow facts; Analysis and computation module M4: Used for: processing active nodes in parallel within the distributed graph computing system, performing iterative analysis based on user-implemented interface instances using a three-step message passing method, until there are no active nodes in the system or the number of iterations reaches a user-specified threshold; the three-step message passing method includes: First message passing: The active node sends its node identifier to all predecessor nodes to obtain the output data stream facts of all predecessor nodes, and then becomes an inactive node; Second message passing: Upon receiving the request, the predecessor node sends its output data stream facts to the successor node based on the node identifier, and then becomes an inactive node; The third message passing: The node to be analyzed receives the output data stream facts of all predecessor nodes, executes the user-implemented interface instance for analysis, and if the passing conditions are met, updates the output data stream facts of this node and sends the node identifier to the successor node to be analyzed in the next iteration, and then becomes an inactive node.

6. The apparatus for inter-process data flow analysis based on distributed graph computing according to claim 5, characterized in that, The analysis and calculation module M4 also includes the following modules: Computational Algorithm Module M41: Used for: Distributed working set algorithms implemented using a distributed graph computing framework; Node processing module M42: Used to process the entry nodes in the control flow graph if it is the first iteration; otherwise, it only processes the active nodes in the current round. Node analysis module M43: Used for: performing analysis on nodes in each iteration by passing messages three times, and determining whether to pass data stream facts based on the passing conditions; Repeat Execution Module M44: Used to repeatedly execute the steps of Node Processing Module M42 to Node Analysis Module M43 until all nodes are inactive, i.e., there is no data flow in the system.

7. The apparatus for inter-process data flow analysis based on distributed graph computing according to claim 6, characterized in that, The distributed working set algorithm implemented in the computation table algorithm module M41 using a distributed graph computation framework is based on a mapping of two algorithms: the vertex-centric computation model in the field of distributed graph computation and the working set algorithm in the field of program analysis.

8. The apparatus for inter-process data flow analysis based on distributed graph computing according to claim 7, characterized in that, The computation algorithm module M41 uses a distributed working set algorithm implemented with a distributed graph computation framework, which is further optimized by a three-way message passing method.

Images