A method for automatically extracting hot code of parallel applications

Abstract

The application discloses a method for automatically extracting hot code of a parallel application, and has the characteristics that the source code of the parallel application to be tested is obtained by static analysis on the parallel application to be tested, the source code is divided into multiple code segments according to the resource features consumed by the parallel application to be tested during running, the running states are marked in the order of time according to the running states of the parallel application to be tested, the running states are marked as vertices, the running process from one running state to another running state is marked as the edges between the vertices, a program running graph is constructed, all the code segments are attached to all the vertices and edges in the program running graph, the code segments with the resource features greater than a set threshold are marked, the code segments after the feature marking are taken as the hot code and are extracted, and the method has the advantages that the hot code can be extracted conveniently and quickly, and the accuracy of extracting the hot code is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and particularly relates to a method for automatically extracting hot code of a parallel application. BACKGROUND

[0002] A parallel application is an application that can be executed simultaneously on multiple processors or multiple computing nodes. In a parallel application, tasks are decomposed into multiple subtasks, which can be executed simultaneously on different processors or computing nodes independently. Communication and synchronization mechanisms are used among the subtasks to coordinate their execution and share data.

[0003] Hot code identification and extraction is a performance analysis and optimization technique, which is mainly used to identify and extract code segments with high execution frequency, long execution time and large impact on overall performance, and source code segments that can represent the running behavior of an application. In the process of performance analysis and optimization, developers identify hot code to find code segments that become performance bottlenecks of the system, and optimize them to reduce execution time and resource consumption, thereby improving system performance.

[0004] For hot code identification and extraction in ordinary workloads, developers usually use dynamic performance analysis tools or static performance analysis tools, but they still have the following defects:

[0005] 1. Existing dynamic performance analysis tools introduce large additional overhead when collecting running data, which may affect the actual execution speed and behavior of the program, which means that the collected data may be different from the case when it is not monitored, which may distort the analysis results, especially in time-sensitive applications and fine-grained parallel programs;

[0006] 2. Static performance analysis tools analyze code without executing it, resulting in a lack of runtime context and the inability to consider dynamic behaviors such as actual input data, runtime conditions or dynamic interactions between threads, which are crucial for identifying real hot code in parallel programs;

[0007] 3. Many existing dynamic or static performance analysis tools have the problem of data granularity: too coarse granularity may miss specific bottlenecks, while too fine granularity may result in a large amount of data and difficult analysis.

[0008] Hot code identification and extraction of parallel applications are more complex than hot code identification and extraction of ordinary workloads, as they need to consider inter-process communication, computing resources, memory resources and load balancing, etc. Existing tools and methods are not efficient and accurate when dealing with large-scale parallel applications. SUMMARY

[0009] The technical problem solved by the present application is to provide a method for automatically extracting hot code of a parallel application, which can extract hot code conveniently and quickly and improve the accuracy of extracting hot code.

[0010] The technical solution adopted by the present application to solve the above technical problem is as follows: a method for automatically extracting hot code of a parallel application, comprising the following steps:

[0011] Step 1: obtaining source code of the to-be-tested parallel application by performing static analysis on the to-be-tested parallel application, and dividing the source code into multiple code segments according to resource features consumed by the to-be-tested parallel application during running;

[0012] Step 2: marking the running states in the order of time according to the running states of the to-be-tested parallel application during running, marking the running states as vertices, marking the running process from one running state to another running state as edges between vertices, and constructing a program running graph;

[0013] Step 3: attaching all the code segments to corresponding vertices and edges in the program running graph, performing feature identification on the code segments whose consumed resource features are greater than a set threshold, and extracting the code segments after feature identification as hot code.

[0014] Compared with the prior art, the present application has the advantages that the source code is divided into multiple code segments according to the resource features consumed by the to-be-tested parallel application during running, which helps to accurately evaluate the amount of resources consumed by each code segment, thereby helping to realize the identification and extraction of hot code; the construction of the program running graph helps developers to clearly understand the running flow of the entire parallel application and the synchronization or asynchronous process between threads; the feature identification of the code segments according to the consumed resource features, the extraction of the code segments after feature identification as hot code, and the improvement of the accuracy of hot code extraction also facilitate the subsequent developers to select the corresponding hot code for running in the simulator according to the needs, for example, if the developer only wants to explore the computing capability, only the code segments containing C identification need to be selected for running, which saves more time for the developers.

[0015] Further, the resource features include inter-thread communication resources, computing resources and input / output resources.

[0016] Further, in Step 3, the specific operation process of performing feature identification on the code segments whose consumed resource features are greater than a set threshold and extracting the code segments after feature identification as hot code is as follows:

[0017] Step ③-1, a feature vector V is constructed for each code snippet, V = (M, C, IO), wherein M represents the inter-thread communication resource consumed by the code snippet during the running of the parallel application, C represents the computing resource consumed by the code snippet during the running of the parallel application, and IO represents the input / output resource consumed by the code snippet during the running of the parallel application;

[0018] Step ③-2, each dimension of the feature vector of each code snippet is normalized to obtain a normalized feature vector, wherein the normalized feature vector corresponding to the i-th code snippet is denoted as V i , n represents the total number of code snippets, M i represents the inter-thread communication resource consumed by the i-th code snippet, C i represents the computing resource consumed by the i-th code snippet, and IO i represents the input / output resource consumed by the i-th code snippet.

[0019] Step ③-3, it is judged whether the total number of code snippets on each vertex or each edge in the program running graph is single or multiple, if it is single, step ③-4 is executed, and if it is multiple, step ③-5 is executed.

[0020] Step ③-4, each dimension of the normalized feature vector corresponding to the code snippet on each vertex or each edge in the program running graph is traversed, if the value of a certain dimension is greater than a preset first threshold value, it is considered that the code snippet has the corresponding consumed resource feature, and the code snippet after feature marking is regarded as a hotspot code and is extracted, and if the value of each dimension is less than or equal to the preset first threshold value, the code snippet is not marked with a feature.

[0021] Step ③-5, all code snippets on each vertex or each edge in the program running graph are clustered to obtain a plurality of code clusters after clustering, wherein the specific operation of obtaining a code cluster is as follows: if the Euclidean distance between the normalized feature vectors corresponding to any two code snippets is less than a preset second threshold value, the two code snippets are classified into one code cluster.

[0022] Step ③-6, in each code cluster, a code fragment with the minimum total distance of feature vectors from other code fragments is selected, and each dimension of the normalized feature vector corresponding to the code fragment is traversed, if the value of a dimension is greater than a preset first threshold value, it is considered that the code fragment has a corresponding consumed resource feature, all code fragments in the code cluster where the code fragment is located are marked with a feature identifier, and the code fragment after the feature identifier is extracted as a hotspot code; if the value of each dimension is less than or equal to the preset first threshold value, all code fragments in the code cluster where the code fragment is located are not marked with a feature identifier. Because the resource features consumed by all code fragments in each code cluster are similar, the code fragment with the minimum total distance of feature vectors from other code fragments in each code cluster is selected as a representative point, and according to the resource feature consumed by the representative point in a code cluster, the resource feature consumed by all code fragments in the code cluster can be determined, without calculating all code fragments in the code cluster one by one, thereby reducing the calculation amount and accelerating the calculation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The figure is a schematic diagram of the overall flow of the present application.

[0024] Figure 2 The figure is a schematic diagram of the program running in the present application. DETAILED DESCRIPTION

[0025] The present application will be further described in detail below with reference to the embodiments of the drawings.

[0026] As shown in the figure, a method for automatically extracting hotspot codes of a parallel application program includes the following steps: Figure 1

[0027] Step ①, by performing static analysis on the to-be-tested parallel application program, the source code of the to-be-tested parallel application program is obtained, and the source code is divided into multiple code fragments according to the resource features consumed by the to-be-tested parallel application program during running; wherein, static analysis refers to detecting the source code of the application program when the application program is not running; the resource features include inter-thread communication resources, computing resources and input-output resources; the multiple code fragments include multiple code fragments with consumed resource features of inter-thread communication resources, multiple code fragments with consumed resource features of computing resources and multiple code fragments with consumed resource features of input-output resources;

[0028] The starting position and the ending position of the MPI (Message Passing Interface, Chinese name: information transmission interface) call in the source code are recorded, and are used as the code fragment with consumed resource features of inter-thread communication resources, and the time of the MPI call is used as an index for measuring the consumed inter-thread communication resources;

[0029] ​Identify and monitor the loop structure in the source code, and as the consumed resource feature for the code segment of the computing resource, intercept the program execution flow using dynamic binary translation technology, calculate the number of instructions consumed by the loop execution, and take the number of instructions consumed by the loop as the index of the consumed computing resource;

[0030] For example: identify the loop structure in the source code, find the syntax features of loop structures such as for, while, do-while, etc.

[0031] Identify the input and output program segment in the source code, and as the consumed resource feature for the code segment of the input and output resource, take the time consumed by the input and output in the input and output program segment as the index of the consumed input and output resource;

[0032] Examples of identifying input and output program segments in source code: find the function parameters that are resource identifiers (such as file paths, database connections, etc.) and data containers (such as buffer pointers, etc.), and find the function body that calls library functions related to input and output (such as fopen, fread, etc.);

[0033] Step 2: According to the time when the running state of the to-be-tested parallel application appears, mark the running state as a vertex and the running process from one running state to another running state as an edge between vertices, and construct a program running graph; The edge between the vertices represents the running process from one running state to another running state, such as from one running state to another running state through inter-thread communication, or from one loop iteration to another loop iteration; Wherein, the running state includes the memory state and the value of the register;

[0034] Step 3: Attach all code segments to the corresponding vertices and edges in the program running graph, identify the code segments whose consumed resource features are greater than the set threshold, and extract the code segments after feature identification as hot code;

[0035] The vertices of the program running graph come from the call stack information of external function calls, and the edges represent the conversion between these function calls; Connecting the divided code segments to the corresponding vertices and edges in the program running graph helps to organize and visualize the program execution flow;

[0036] Among them, the specific operation process of identifying the code segments whose consumed resource features are greater than the set threshold, and extracting the code segments after feature identification as hot code is as follows:

[0037] Step ③-1, a feature vector V is constructed for each code snippet, V = (M, C, IO), wherein M represents the inter-thread communication resource consumed by the code snippet during the running of the parallel application, C represents the computing resource consumed by the code snippet during the running of the parallel application, and IO represents the input / output resource consumed by the code snippet during the running of the parallel application;

[0038] Step ③-2, each dimension of the feature vector of each code snippet is normalized to obtain a normalized feature vector, wherein the normalized feature vector corresponding to the i-th code snippet is denoted as V i , n represents the total number of code snippets, M i represents the inter-thread communication resource consumed by the i-th code snippet, C i represents the computing resource consumed by the i-th code snippet, and IO i represents the input / output resource consumed by the i-th code snippet.

[0039] Step ③-3, it is judged whether the total number of code snippets on each vertex or each edge in the program running graph is single or multiple, if it is single, step ③-4 is executed, and if it is multiple, step ③-5 is executed.

[0040] Step ③-4, each dimension of the normalized feature vector corresponding to the code snippet on each vertex or each edge in the program running graph is traversed, if the value of a certain dimension is greater than a preset first threshold value, it is considered that the code snippet has the corresponding consumed resource feature, and a feature identifier is marked thereon, and the code snippet after the feature identifier is taken as a hotspot code and is extracted, and if the value of each dimension is less than or equal to the preset first threshold value, the code snippet is not marked with a feature identifier, and the code snippet is not a hotspot code; the preset first threshold value is determined by a user, for example, if the user hopes to screen out a code snippet whose consumed resource feature proportion is greater than 15%, the first threshold value is set to 15%.

[0041] Step ③-5, all code snippets on each vertex or each edge in the program running graph are clustered to obtain a plurality of code clusters after clustering, wherein the specific operation of obtaining a code cluster is as follows: if the Euclidean distance between the normalized feature vectors corresponding to any two code snippets is less than a preset second threshold value, the two code snippets are classified into a code cluster; the preset second threshold value is defined by a user, for example, if the user hopes that the resource feature similarity of two code snippets is greater than 90%, the second threshold value is set to 10%.

[0042] Step ③-6, in each code cluster, the distance between the feature vector of each code snippet and the feature vector of other code snippets is obtained, the code snippet with the minimum total distance to the feature vectors of other code snippets is selected, and each dimension in the normalized feature vector corresponding to the code snippet is traversed, if the value of a certain dimension is greater than a preset first threshold value, it is considered that the code snippet has the corresponding consumed resource feature, all code snippets in the code cluster where the code snippet is located are marked with a feature identifier, and the code snippet after feature identification is extracted as a hot code snippet; if the value of each dimension is less than or equal to the preset first threshold value, all code snippets in the code cluster where the code snippet is located are not marked with a feature identifier;

[0043] For example: all code snippets on vertex X are clustered to obtain three code clusters, code cluster a, code cluster b and code cluster c, and each code cluster has some code snippets, then the code snippet with the minimum total distance to the feature vectors of other code snippets in code cluster a is selected, and the dimensions where the code snippet M is located, the dimensions where the code snippet C is located, and the dimensions where the code snippet IO is located are traversed to determine whether they are greater than a preset first threshold value; code cluster b and code cluster c are similarly processed;

[0044] For example: the set first threshold value is 15%, and the normalized feature vector corresponding to a code snippet is (0.16, 0.13, 0.155), it is considered that the code snippet is both an MPI snippet and an IO snippet, and it is marked with a feature identifier (M, IO).

Claims

1. A method for automatically extracting hotspot code from parallel applications, characterized in that... Includes the following steps: Step ①: Obtain the source code of the parallel application under test by performing static analysis, and divide the source code into multiple code segments according to the resource consumption characteristics of the parallel application under test during runtime. Step 2: Mark the running states of the parallel application under test in chronological order according to the time of their occurrence. Mark the running states as vertices and the process of transitioning from one running state to another as edges between vertices to construct the program running graph. Step 3: Attach all code snippets to the corresponding vertices and edges in the program execution graph, mark the code snippets whose resource consumption characteristics exceed the set threshold, and extract the marked code snippets as hotspot code. In step ③, the code segments whose resource consumption characteristics exceed a set threshold are characterized by features, and the code segments with the identified features are extracted as hotspot code. The specific operation process is as follows: Step ③-1: Construct a feature vector V for each code segment, V=(M,C,IO), where M represents the inter-thread communication resources consumed by the code segment when the parallel application is running, C represents the computational resources consumed by the code segment when the parallel application is running, and IO represents the input and output resources consumed by the code segment when the parallel application is running. Step ③-2: Normalize each dimension of the feature vector of each code segment to obtain the normalized feature vector. The normalized feature vector corresponding to the i-th code segment is denoted as... , n represents the total number of code snippets. This represents the inter-thread communication resources consumed by the i-th code snippet. This represents the computational resources consumed by the i-th code snippet. This represents the input / output resources consumed by the i-th code snippet; Step ③-3: Determine whether the total number of code segments on each vertex or edge in the program execution graph is single or multiple. If it is single, proceed to step ③-4; if it is multiple, proceed to step ③-5. Steps ③-4: For each vertex or edge of the program execution graph, traverse each dimension of the normalized feature vector corresponding to the code segment. If the value of a certain dimension is greater than the preset first threshold, the code segment is considered to have the corresponding resource consumption feature, and a feature label is added to it. The code segment with the feature label is then extracted as hot code. If the value of each dimension is less than or equal to the preset first threshold, the code segment is not labeled. Step ③-5: Cluster all code segments on each vertex or edge in the program execution graph to obtain multiple code clusters. The specific operation to obtain a code cluster is as follows: if the Euclidean distance between the normalized feature vectors corresponding to any two code segments is less than the preset second threshold, then the two code segments are classified into one code cluster. Steps ③-6: In each code cluster, select the code segment with the smallest total distance to the feature vectors of other code segments, and iterate through each dimension of the normalized feature vector corresponding to the code segment. If the value of a certain dimension is greater than a preset first threshold, the code segment is considered to have the corresponding resource consumption feature. Feature labels are then applied to all code segments in the code cluster to which the code segment belongs, and the feature-labeled code segments are extracted as hotspot code. If the value of each dimension is less than or equal to the preset first threshold, feature labels are not applied to all code segments in the code cluster to which the code segment belongs.

2. The method for automatically extracting hotspot code from parallel applications according to claim 1, characterized in that... The resource characteristics mentioned include inter-thread communication resources, computing resources, and input / output resources.

Images