Supercomputing architecture-oriented parallel program performance data monitoring and collecting method

Abstract

The invention discloses a supercomputing architecture-oriented parallel program performance data monitoring and collecting method. The method comprises the following steps of: designing a unified performance trajectory data format for three performance domains, namely a general computing domain, an acceleration computing domain and a communication domain, according to different supercomputing heterogeneous architectures; packaging PMU performance data monitoring interfaces according to different supercomputing architectures; a unified performance trajectory data acquisition process is designed for different supercomputing heterogeneous architectures; designing and realizing a performance trajectory data acquisition method based on multi-granularity instrumentation and sampling; performance trajectory data acquisition processes of the three performance domains are unified into synchronous and asynchronous acquisition processes. The invention provides a method for carrying out lightweight trajectory data acquisition on a heterogeneous architecture supercomputing parallel program in combination with a synchronous and asynchronous mechanism, an upper-layer performance analysis function and a specific hardware architecture are decoupled, and the development cost of a performance analysis tool and the performance data acquisition overhead are reduced.

Description

Technical Field

[0001] The present invention belongs to the field of program performance analysis, and specifically relates to a method for monitoring and collecting parallel program performance data for a supercomputing architecture. Background Art

[0002] In heterogeneous many-core systems, the execution of high-performance computing programs is greatly affected by parallelism, communication overhead, and resource competition. Locking mechanisms, serialization, blocking, load imbalance, etc. often occur in multi-process or multi-threaded synchronous operations, which have become important issues that must be considered in the development of high-performance computing programs. At the same time, due to the complexity of the supercomputer system architecture and the diversity of application computing behaviors, it is difficult to sustain the scalability bottleneck of high-performance computing programs by manually determining them. Performance analysis tools are needed to automatically analyze the performance of large-scale parallel applications.

[0003] On the new generation of supercomputers, parallel applications may face an environment with a scale of more than 10 million cores. The performance trace collection of general computing domains, acceleration domains, and communication domains at this new scale is very sensitive to time and space overhead. The research on performance analysis tools began many years ago. For decades, performance analysis tools have been an indispensable part of the high-performance computing software stack, and many tools have emerged that can be used for various analyses. These tools can be classified according to the information collection and organization principles they are based on. For example, the hpcrun component in the HPCToolkit based on performance statistics collects relevant performance data of the call context through sampling, and then performs binary analysis, code tracing, etc. through the hpcstruct component and the hpcprof component. SCALANA uses static analysis technology to achieve comprehensive performance statistics with an overhead similar to performance statistics. At runtime, lightweight technology is used to collect performance data based on the structure diagram and generate program performance graphs. Based on the program performance graph, a new root cause detection method can be used to effectively and automatically detect the root cause of scalability loss. The TAU toolkit based on performance tracing, combined with tools such as the PAPI interface and the Score-P framework, supports many performance statistics and performance tracing modes, as well as a large number of instrumentation methods. Some work based on modeling analysis also attempts to strike a balance between high overhead and high accuracy. However, these existing works are basically unable to truly balance low overhead and accuracy in the performance data collection of large-scale parallel programs.

[0004] Furthermore, neither the sampling method nor the instrumentation method has been used to collect performance data, and no testing and operation has been done at a new scale of more than 10 million cores. At the scale of 10 million cores, it is necessary to compare and select data collection methods, and balance between overhead and accuracy. It is necessary to combine instrumentation and sampling technology with efficient asynchronous processing mechanisms to achieve lightweight and accurate performance trajectory collection. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a parallel program performance data monitoring and collection method for supercomputing architecture, which is a multi-angle parallel program performance data monitoring and collection method based on the combination of synchronous and asynchronous methods and sampling and plugging methods.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A method for monitoring and collecting performance data of parallel programs for supercomputing architecture, comprising:

[0008] Step S1: extracting the running characteristics of parallel programs on different supercomputer heterogeneous architectures in the general computing domain, the accelerated computing domain, and the communication domain, and designing a unified data format, wherein the unified data format is used for the same performance domain when storing performance trajectory data on different supercomputer heterogeneous architectures;

[0009] Step S2: encapsulate the PMU performance data monitoring interface for different supercomputer heterogeneous architectures, align the performance event types on the CPUs and accelerators of different supercomputer heterogeneous architectures, and unify the performance monitoring unit reading process based on performance event counting on different supercomputer heterogeneous architectures;

[0010] Step S3: unify the performance trace data collected by different supercomputer heterogeneous architectures in the general computing domain of the parallel program into the data format designed in step S1, and store them in the general computing trace file; unify the performance trace data collected by different supercomputer heterogeneous architectures in the accelerated computing domain of the parallel program into the data format designed in step S1, and store them in the accelerated computing trace file; unify the performance trace data collected by different supercomputer heterogeneous architectures in the communication domain of the parallel program into the data format designed in step S1, and store them in the communication trace file; organize the trace files of different performance domains through calling relationships and timing relationships;

[0011] Step S4: different collection methods are used for the parts of the parallel program in the three performance domains, and the performance trace data is collected in accordance with the unified data format specified in step S1 by combining multi-granularity instrumentation and sampling methods, and the collected data results are written into the corresponding files in the storage and organization form specified in step S3;

[0012] Step S5: Design and implement two types of interfaces, synchronous and asynchronous, for collectors to use, and unify the performance trace data collection process of the three performance domains into a callback-based synchronous collection process and an additional collection thread-based asynchronous collection process.

[0013] Furthermore, the unified data format is designed in step S1, specifically including:

[0014] Step S11: designing a first performance trace data structure for a heterogeneous programming model composed of a CPU and a GPU, wherein the first performance trace data structure covers synchronous and asynchronous APIs including memory allocation and data transmission;

[0015] Step S12: Design a second performance trace data structure aligned with the first performance trace data structure for the heterogeneous programming model composed of CPU and DSP, record the interaction process between CPU and different types of accelerators under different supercomputer heterogeneous architectures in a consistent format, and align the storage space occupancy of the first performance trace data structure and the second performance trace data structure.

[0016] Furthermore, in step S2, the types of performance events on the CPU and accelerator of the heterogeneous architecture are aligned, and the performance monitoring unit reading process based on performance event counts on different supercomputer heterogeneous architectures is unified, including:

[0017] Step S21: Verify whether the existing PMU performance data monitoring interface can support performance data monitoring on the supercomputing heterogeneous architecture;

[0018] Step S22: If partially supported, unify the performance monitoring unit reading process based on performance event counting on different supercomputer heterogeneous architectures, and finally unify the name and code of the encapsulated performance event.

[0019] Furthermore, the step S3 includes:

[0020] Step S31: The trajectory data collected by the specified collection program is organized in the first or second performance trajectory data structure in step S1;

[0021] Step S32: the host-side performance data collection program obtains the first or second performance trace data structure in S1 from the collection interface callback functions in the three performance domains;

[0022] Step S33: The files storing the trajectory data are uniformly written into the data buffer structure on the host side, including three types of trajectory files, namely, the host side MPI communication trajectory file, the host side general computing trajectory file and the device side accelerated computing trajectory file, and their corresponding metadata files describing and guiding the analysis.

[0023] Further, the step S4 includes:

[0024] Step S41: using the dynamic library interception and stub method in the CPU-side program under the supercomputer heterogeneous architecture, wrapping the original function with the acquisition function, and obtaining the communication and calculation information between process threads, including the communicator and the amount of data transmitted;

[0025] Step S42: Design a source code instrumentation method for the accelerator-side program under the supercomputing heterogeneous architecture, use llvm and clang to find the corresponding function call in the AST generated after the source code lexical and grammatical analysis, perform code insertion and data collection and transmission, and combine the source code instrumentation with the signal interrupt mechanism between the CPU and DSP to realize various sampling and collection methods.

[0026] Furthermore, the step S5 comprises:

[0027] Step S51: design two types of acquisition interfaces, synchronous and asynchronous, for users to call, to respectively collect synchronous interface call information and asynchronous interface call information of the entire parallel computing program in the process of MPI communication, general CPU computing and accelerator computing;

[0028] Step S52: Implementing a collection thread and a data buffer structure for the asynchronous collection interface.

[0029] The beneficial effects of the present invention are:

[0030] The present invention unifies the heterogeneous architecture of the new generation supercomputer, provides a unified performance trajectory data collection process and interface to the upper-level analysis function, and supports performance analysis work on the supercomputer. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the fields included in the acceleration domain synchronization acquisition method in the first performance trace data structure designed for the present invention;

[0032] Figure 2 A schematic diagram of the fields included in the acceleration domain synchronization acquisition method in the second performance trace data structure designed for the present invention;

[0033] Figure 3 This is a schematic diagram of the classification of sampling and instrumentation collection techniques;

[0034] Figure 4 A diagram of the compilation process for C language programs using llvm and clang;

[0035] Figure 5 Schematic diagram of the effect of the insertion position on the accuracy of trajectory data collection in the present invention;

[0036] Figure 6 This is a schematic diagram of a recording structure of a trajectory asynchronous acquisition method designed for two different heterogeneous architectures in the present invention;

[0037] Figure 7 Schematic diagram of the overall interface and software architecture. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0039] The present invention provides a method for monitoring and collecting performance data of parallel programs for supercomputer architecture, designs a series of trace data structure formats that align the differences of two supercomputer heterogeneous architectures and a collection method that combines synchronization and asynchronousness, thereby reducing the collection time and space overhead of parallel programs on them. Specifically, it includes:

[0040] Step S1: extracting the running characteristics of parallel programs on different supercomputer heterogeneous architectures in the general computing domain, the accelerated computing domain, and the communication domain, and designing a unified data format, wherein the unified data format is used for the same performance domain when storing performance trajectory data on different supercomputer heterogeneous architectures;

[0041] Step S2: encapsulate the PMU performance data monitoring interface for different supercomputer heterogeneous architectures, align the performance event types on the CPUs and accelerators of different supercomputer heterogeneous architectures, and unify the performance monitoring unit reading process based on performance event counting on different supercomputer heterogeneous architectures;

[0042] Step S3: unify the performance trace data collected by different supercomputer heterogeneous architectures in the general computing domain of the parallel program into the data format designed in step S1, and store them in the general computing trace file; unify the performance trace data collected by different supercomputer heterogeneous architectures in the accelerated computing domain of the parallel program into the data format designed in step S1, and store them in the accelerated computing trace file; unify the performance trace data collected by different supercomputer heterogeneous architectures in the communication domain of the parallel program into the data format designed in step S1, and store them in the communication trace file; organize the trace files of different performance domains through calling relationships and timing relationships;

[0043] Step S4: different collection methods are used for the parts of the parallel program in the three performance domains, and the performance trace data is collected in accordance with the unified data format specified in step S1 by combining multi-granularity instrumentation and sampling methods, and the collected data results are written into the corresponding files in the storage and organization form specified in step S3;

[0044] Step S5: Design and implement two types of interfaces, synchronous and asynchronous, for collectors to use, and unify the performance trace data collection process of the three performance domains into a callback-based synchronous collection process and an additional collection thread-based asynchronous collection process.

[0045] Example

[0046] Take the synchronous trajectory data structure of the accelerated computing domain as an example, Figure 1As shown, a method for monitoring and collecting performance data of parallel programs for supercomputing architecture is provided in an embodiment of the present invention. The trajectory data structure of the general computing domain and the communication domain is similar to the structure in the figure. Taking two supercomputing heterogeneous architectures of CPU+GPU and CPU+DSP as examples, the architecture is unified based on the method of the present invention, including the following steps:

[0047] Step S1: Extract the running characteristics of programs in the general computing domain, accelerated computing domain, and communication domain on different supercomputer heterogeneous architectures, design a data structure that can reflect the running status of the program in the three performance domains, and ensure that the same data format can be used to store performance traces in the same performance domain on different supercomputer heterogeneous architectures GPU;

[0048] Step S2: encapsulate the PMU performance data monitoring interface for different supercomputer heterogeneous architectures, and unify the performance monitoring unit reading process based on performance event counting on different supercomputer heterogeneous architectures;

[0049] Step S3: The performance information that can be collected by different supercomputer heterogeneous architectures in the general computing domain of the parallel program is unified into the trajectory data format mentioned in S1 and stored in the general computing trajectory file. The information collected by different supercomputer heterogeneous architectures in the acceleration computing domain and communication domain is similarly stored in the trajectory files of their respective categories. The trajectory data files of different performance domains are organized through calling relationships and timing relationships;

[0050] Step S4: different collection methods are used for the parts of the parallel program in the three performance domains, and the performance trace data is collected in accordance with the format specified in S1 by combining multi-granularity instrumentation and sampling methods, and the collected data results are written into a file in the storage and organization form specified in S3;

[0051] Step S5: Implement a method combining synchronization and asynchronous to reduce overhead while ensuring acquisition accuracy. Combining the advantages of the two acquisition methods of sampling and plugging in S4, and considering the characteristics of the two methods, a sampling method focusing on reducing overhead and a plugging method with specific accuracy are implemented on the device side, and synchronous callback-based acquisition and asynchronous interrupt-based acquisition are combined to implement a high-performance asynchronous processing method of performance trajectory data on the host side to ensure accuracy and reduce overhead, and the performance trajectory data acquisition process of the three performance domains of general computing domain, accelerated computing domain and communication domain is unified into a callback-based synchronous acquisition interface and an asynchronous acquisition interface based on an additional acquisition thread. Users can use the same interface to monitor and collect program execution in the general computing domain, accelerated computing domain and communication domain.

[0052] In one embodiment, the above step S1: designs a unified performance trajectory data format for different domestic supercomputer heterogeneous architectures. Specifically, a unified performance trajectory data structure is designed for the heterogeneous architectures of CPU+GPU and CPU+DSP, extracts the similarities between the two different heterogeneous architectures, and aligns the space occupancy of the trajectory data structure while ensuring that the information contained is the same, specifically including:

[0053] First, to run a program on the GPU, it is necessary to use the CPU to apply for GPU memory space and then asynchronously start the GPU program running through kernellaunch. The memory structure of the GPU is roughly composed of global memory, shared memory within the CU, and register files. After the CPU applies for the GPU global memory space, it copies the content in the CPU memory to the GPU and specifies the required shared memory size. These memory spaces can then be used in programs running on the GPU. Therefore, in the CPU+GPU supercomputing heterogeneous architecture, the GPU interface functions that require special attention are kernel launch, synchronous and asynchronous memory copying. For kernel launch, special information such as thread dimension and shared memory size is recorded. For memory copying, the sending and receiving addresses are recorded. For other things such as memory application, synchronization, and communication between GPUs, only the most basic timestamps, call types, and correlation IDs used for timeline alignment are temporarily recorded. For example Figure 1 As shown in the diagram of the data structure of the first performance trace, the left side is the record structure of kernel function startup under the CPU+GPU supercomputing heterogeneous architecture, and the right side is the record structure of memory copy transmission. There are some differences in the data items in the record structures of kernel function startup and memory copy operations. Memory copy distinguishes between synchronous and asynchronous versions. Asynchronous memory copy operations need to record the stream that carries the asynchronous operation.

[0054] Secondly, in terms of memory structure, there will be some differences between CPU+DSP. The CPU and DSP will share a part of the memory space. After applying for the space occupied by the data that needs to be passed into the DSP program, there is no need to copy the host-side memory to the device-side memory like the GPU, and the initialization work can be performed directly in it. The program running on the DSP side needs to be processed using the matching DSP-side compiler, assembler and connector, and finally packaged into an image file and passed to the DSP chip. The operation of the DSP program on it is based on the DSP cluster and thread group. The interface similar to the kernel function startup requires the introduction of the description information of the required parameters. The memory distribution and operation under the two supercomputer heterogeneous architectures are very different, so a new acquisition data structure is designed for the thread group function execution and various DMA transfers on the DSP side. Such as Figure 2As shown in the schematic diagram of the data structure of the second performance trace, the left side is the kernel function startup record structure under the CPU+DSP supercomputing heterogeneous architecture, and the right side is the record structure of memory replication. Due to the different hardware environments, the kernel function startup record structure corresponds to the thread block and grid dimensions in the CPU+GPU supercomputing heterogeneous architecture, which are replaced by the thread dimension information of the DSP cluster; asynchronous memory replication is completed by directly specifying DMA operations. In order to distinguish the various types of memory of the CPU and DSP, the data items of the transmission type are added to the record structure.

[0055] At the same time, in order to ensure that the space occupied by the record structures under the two architectures is not too large, union is used to store different types of structures in an overlapping manner.

[0056] In one embodiment, the step S2: aligning the performance event types on the CPU and accelerator of the heterogeneous architecture, and combining the reading process and interface of the performance event registers on the two heterogeneous architectures for unified packaging, specifically includes:

[0057] A unified performance data monitoring interface similar to PAPI (Performance API), namely, a hardware performance event interface for general computing domain and acceleration domain hardware, as well as a performance monitoring method based on application internal behavior semantic information, hardware performance event semantics and code correlation extraction. Ultimately, the unified interface will be able to collect performance data from multiple levels such as hardware, system software, and parallel applications.

[0058] It is implemented in two types of hardware: general computing domain and accelerated computing domain. For the general computing domain, the PAPI interface can collect some performance events supported by the processor. For the other performance events not supported by the PAPI interface, the interface tool is expanded and rewritten. In addition, targeted event design is carried out considering the types and numbers of performance counters under different supercomputer general computing domain hardware structures.

[0059] For the acceleration domain, following the architecture of PAPI, the management and performance data collection interface of the accelerator is called, and the native performance counter reading interface provided on the two types of supercomputing heterogeneous architectures is aligned with the additional performance event collection function to align the usage of the PMU collection interface on the two types of supercomputing heterogeneous architectures. Finally, the common performance events under different architectures are extracted for the CPU and accelerator respectively, and a unified PMU performance data monitoring interface that can be used in multiple different supercomputing architectures is formed after unifying the name and encoding. The collected performance counter readings can be directly stored in the performance trajectory record structure described in S1.

[0060] In one embodiment, step S3: unify the performance trace data collection processes of different supercomputing architectures in the three performance domains of parallel program general computing domain, accelerated computing domain, and communication domain, respectively, to form three different performance trace files with multi-granularity instrumentation as the underlying collection means, and organize the trace files using the correlation ID contained in the structure in S1, specifically including:

[0061] The data structure used to store the trace contains the readings of the PMC on the hardware. These readings are initialized by the PMU acquisition interface described in S2 before running the acquired program, and are read dynamically during the acquisition process. They are all int64_t types, representing the number of occurrences of a specific event. Since the performance events of interest may be different during each acquisition process, the length of this data structure should be indefinite, and additional metadata is required to describe the corresponding events when storing them.

[0062] The trajectory data of the general domain will be stored in the tr trajectory file, and the trajectory data of the acceleration domain will be stored in the etr file. Since the communication domain is also a host-side function called by the host, the trajectory data can be written into the tr file together with the general domain trajectory data, or it can be written into a separate file. Since the functions on the accelerator and the functions on the CPU side in the two architectures run asynchronously, a correlation ID is added to the data structure to identify the accelerator function and the main process (main thread) that starts it. Even if these different trajectory data bodies are arranged in a chaotic manner on the timeline due to the possibility of multi-accelerator devices and multi-core functions in parallel, the correlation ID recorded in the trajectory metadata is needed to match the acceleration domain trajectory stored in the etr file with the general domain trajectory in the tr file, so as to facilitate the subsequent analysis of the program execution process. Finally, the same structure under the two types of supercomputer heterogeneous architectures will occupy the same amount of space, and the fields in the header will be used to distinguish each other so that the field information can be correctly obtained when analyzing access.

[0063] Although the function prototypes and underlying implementations of the two types of supercomputer heterogeneous architectures are quite different, some commonalities can be found, that is, most of the interactions between the host and device sides and the execution process of the device-side functions are consistent. The host and the device are asynchronously and loosely linked, without strict timing connection, but with a logical order relationship. Therefore, it is possible to design unified monitoring interfaces and trajectory collection interfaces on different architectures:

[0064] (1) Performance data monitoring interface based on PMU. Processors of different architectures are equipped with a monitoring unit PMU with a performance monitoring counter (PMC). By using its performance event-based mechanism, the PMC count can be read at any stage of program execution to obtain the running status of a certain part of the program, forming a unified performance data monitoring interface under the multi-supercomputing architecture.

[0065] (2) Synchronous and asynchronous acquisition interfaces in the acceleration domain. The commonality of the two types of supercomputer heterogeneous architectures makes their execution processes for functions on the accelerator essentially the same. Both require marking the device-side program as a function that can be called directly or indirectly by the host. Therefore, two types of acquisition interfaces based on instrumentation, synchronous and asynchronous, are designed for the device-side programs under the two architectures. The mechanism is to insert callback functions before and after the device-side function call, first perform data acquisition, then run the original function, and finally perform data acquisition. The synchronous acquisition interface corresponds to the synchronous function, and the asynchronous interface corresponds to the asynchronous function. In this way, accurate acquisition of the device-side program can be achieved, forming two types of acquisition interfaces, synchronous and asynchronous, in the acceleration domain.

[0066] (3) General domain acquisition interface. In the general domain, functions such as memory operations malloc and memcpy can be collected by writing wrappers through dynamic library interception. Computing libraries in the general domain, such as the parallel programming library OpenMP, have their own OMPT interface to implement mechanisms such as instrumentation and trajectory data acquisition at the bottom implementation level. The pthread library can use a similar source code instrumentation or dynamic library interception method to implement the acquisition interface. Ultimately, a synchronous acquisition interface based on callback functions will also be formed in the general domain.

[0067] (4) Unified acquisition interface for the communication domain. The communication domain focuses on inter-MPI process communication. It uses the PMPI tool interface of MPI to write a wrapper to dynamically intercept MPI function calls. It can also form synchronous and asynchronous acquisition interfaces for the synchronous and asynchronous communication processes in MPI. The final collected trajectory data can be stored in the same tr file as the trajectory of the general computing domain, forming two loosely coupled trajectories with the etr file of the acceleration domain.

[0068] like Figure 3 As shown, in one embodiment, the above step S4: designs and implements performance trace data collection based on instrumentation and sampling for the new self-developed heterogeneous architecture software and hardware programming model. According to the characteristics of the programming models of the two heterogeneous architectures, different collection methods are used for different types of programs on the CPU and accelerator, and a combination of multiple instrumentation and sampling methods is used to realize the collection of performance trace data, specifically including:

[0069] The sampling method periodically records the current execution status of the program. Here, periodicity can refer to interrupt signals sent by timers implemented by software and hardware at certain time intervals or interrupts caused by overflow of some registers, or some asynchronous event mechanisms defined by users at the software level. The sampling method uses the processing function of this interrupt to collect data and capture a snapshot of the program running at a certain moment. If the hardware has the ability to monitor precise events, the collection of this data will be even more accurate. Since the sampling frequency of this method is variable, it generally has little effect on the operation of the collected program itself, but it is essentially based on statistics and probability. In trajectory analysis, the number of samples falling in a certain interval can reflect the path and execution time of the program. Using a lower sampling frequency will cause many short-lived events to go unnoticed, which is not conducive to generating accurate performance trajectories.

[0070] The instrumentation rule is implemented by inserting the collected code into the program before it runs using a variety of methods. The mainstream instrumentation methods can be roughly divided into source code instrumentation, compiler instrumentation, binary rewriting and dynamic library interception. In essence, they all insert code at a certain stage in the process of compiling the user source code into machine code. The CPU+GPU supercomputing heterogeneous architecture has a wider choice of instrumentation methods because the accelerator is the GPU. The current llvm compilation framework and the clang compiler in it do not have an existing compiler backend implemented for the CPU+DSP supercomputing heterogeneous architecture. It is impossible to use the compiler to instrument the IR (Intermediate Representation) compiled by the user program, and static binary rewriting is too cumbersome. Dynamic binary instrumentation requires its underlying code and is too costly. Considering the instrumentation location and data collection accuracy, the source code instrumentation form was finally chosen, converting the source code into AST (Abstract Syntax Tree) - a structured representation of the source code, and then using clang's rewriter to identify and instrument specific functions or code segments on it, and further converting it back to the source code after the code is inserted. Figure 4 The compilation process of llvm and clang and the position of AST in it. A source code is parsed into tokens after preprocessing and lexical analysis, and then the target AST of source code conversion is obtained through syntax analysis. After static analysis, AST generates IR, and after optional multi-layer optimization, it enters the assembler and linker to obtain the final executable program. AST is in the first half of the whole process, which is more intuitive than the compilation stub that operates on IR.

[0071] If the instrumentation method is used throughout the entire acquisition process, some short functions may cause a significant increase in the program time and space overhead after instrumentation, so it must be used in conjunction with the sampling method. The interrupt signal transmission mechanism between the CPU and DSP can be used to define an interrupt processing function, and track data can be collected in the interrupt processing function, which largely avoids the "uncertainty" problem in performance analysis tools, that is, the higher the acquisition accuracy, the more serious the distortion of program execution information;

[0072] In one embodiment, the above step S5: designing and implementing two types of interfaces, synchronous and asynchronous, exposed to users, unifying the performance trace data collection process of the three performance domains, and reducing the overhead while ensuring the collection accuracy, specifically includes:

[0073] Performance data monitoring interface based on PMU. This interface call can be embedded in the performance data collection code or calculation code of the three performance domains in the form of a stub. Processors of different architectures have monitoring units PMU equipped with performance monitoring counters (PMC). Using its performance event-based mechanism, PMC counts can be read at any stage of program execution to obtain the running status of a certain part of the program.

[0074] To achieve lightweight performance trace collection based on a unified interface, it is necessary to make full use of asynchronous processing mechanisms and data buffering mechanisms. From the perspective of the location of the stub, if the stub is only performed before and after the function on the CPU side (including various memory operation interfaces and kernel function startup interfaces, etc.) is called, some functions are processed synchronously, so the calling statement will not return until the called function returns. However, some asynchronously implemented function running information will be partially lost, such as Figure 5 As shown, for a synchronously processed function (such as memory allocation in the figure), the time point corresponding to the start of execution and return of the function matches the statement that starts it. As long as stubs are inserted at the first position 1 and the second position 2 in the figure, accurate information of the synchronous function can be collected; but for an asynchronous function (such as the kernel function startup in the figure), the calling function will return before the kernel function returns, that is to say, the information of the kernel function itself cannot be obtained by inserting stubs at the third position 3 and the fourth position 4 of the host-side API, and collection is required at the fifth position 5 and the sixth position 6 to obtain the precise time when the kernel function runs.

[0075] For the common problem of the two types of architectures, that is, the lack of accuracy in the collection results of asynchronous events, we can refer to the method adopted by CUPTI in CUDA. CUPTI (CUDA Profiling Tool Interface) is a series of performance analysis and performance tracking interfaces designed for CUDA programs. The performance tracking interfaces are divided into Activity API and Callback API. The former will asynchronously record the activities of CUDA programs and use background threads to monitor the call and return events of CUDA APIs, kernel functions, etc.; the latter will insert callback functions before and after specific CUDA APIs, so that users can customize callback functions for data collection. It is a synchronous implementation mechanism. Under the CPU+DSP architecture, different codes can be inserted before and after the host program calls the API or even in the device-side function by using source code instrumentation to achieve collection. For the synchronous collection mechanism, the callback function is inserted before and after the API call, and the logic is subdivided according to the type of API and the code insertion position. The parameters, PMC readings, timestamps, call stacks and some other information passed to the API are packaged and handed over to the callback function for specific processing such as writing the trajectory file. For the asynchronous acquisition mechanism, it is necessary to combine the buffer mechanism such as memory pool to insert code before and after the user calls the API and in the kernel function written by the user. Instead of calling the callback function, the simple function running information is passed to a global data buffer structure and then returned to the original place to continue execution. This structure needs to be able to greatly reduce the dynamic memory allocation overhead and have the ability to asynchronously process the stored trajectory data. When the space usage of the data buffer structure reaches a certain level or certain specific events occur, a disk flush is performed to write all the current data in the buffer to the file.

[0076] The acquisition accuracy and performance impact of the acquired program based on synchronous and asynchronous processing are different. Therefore, the data format and content acquired by the two methods are also different. The synchronous callback function will distinguish the API type acquisition call parameters, PMC readings, etc., and perform stack expansion in the callback function, and store the call stack information in the trajectory data structure. Asynchronous acquisition may only store simple information such as timestamp, process thread ID, function name, etc. in the structure, and will not perform stack expansion, which is a more expensive process. Figure 6 It is a data item in the asynchronous acquisition trajectory record structure shared by the two types of architectures. The correlation ID is used to organize the function calls on the host side and the function calls on the device side. The start and end timestamps are used for subsequent analysis of the program execution path. The union contains three methods to uniquely identify a device-side thread. Threads can be distinguished by device-operation queue method, main process-child thread method or other methods. Finally, it also contains text information that can be empty, which is used to provide additional description or marking of the record structure.

[0077] In addition to the above-mentioned synchronous and asynchronous acquisition interfaces for the acceleration domain, the synchronous and asynchronous function calls of the general computing domain and the communication domain are also organized in the two major categories of acquisition interfaces. When using them, the collector does not need to consider which device the collected function or event occurs on. The integrated synchronous and asynchronous acquisition interfaces can be used for CPU functions, accelerator functions, kernel functions, and inter-process communication functions, which greatly reduces the coupling and difficulty of use between the upper-level analysis functions and the underlying data acquisition functions, and improves the scalability of the system. The final overall interface architecture is as follows: Figure 7 As shown, the level of abstraction of the three-layer interface increases from bottom to top, making it more convenient to use.

[0078] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for monitoring and collecting performance data of parallel programs for supercomputing architecture, characterized in that: include: Step S1: extracting the running characteristics of parallel programs on different supercomputer heterogeneous architectures in the general computing domain, the accelerated computing domain, and the communication domain, and designing a unified data format, wherein the unified data format is used for the same performance domain when storing performance trajectory data on different supercomputer heterogeneous architectures; Step S2: encapsulate the PMU performance data monitoring interface for different supercomputer heterogeneous architectures, align the performance event types on the CPUs and accelerators of different supercomputer heterogeneous architectures, and unify the performance monitoring unit reading process based on performance event counting on different supercomputer heterogeneous architectures; Step S3: unify the performance trace data collected by different supercomputer heterogeneous architectures in the common computing domain of the parallel program into the data format designed in step S1, and store them in the common computing trace file; The performance trajectory data collected by different supercomputer heterogeneous architectures in the accelerated computing domain of the parallel program are unified into the data format designed in step S1, and stored in the accelerated computing trajectory file; Unify the performance trace data collected by different supercomputer heterogeneous architectures in the communication domain of the parallel program into the data format designed in step S1, and store them in the communication trace file; Trajectory files of different performance domains are organized through call relationships and timing relationships; Step S4: different collection methods are used for the parts of the parallel program in the three performance domains, and the performance trace data is collected in accordance with the unified data format specified in step S1 by combining multi-granularity instrumentation and sampling methods, and the collected data results are written into the corresponding files in the storage and organization form specified in step S3; Step S5: Design and implement two types of interfaces, synchronous and asynchronous, for collectors to use, and unify the performance trace data collection process of the three performance domains into a callback-based synchronous collection process and an additional collection thread-based asynchronous collection process.

2. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: The unified data format is designed in step S1, specifically including: Step S11: designing a first performance trace data structure for a heterogeneous programming model composed of a CPU and a GPU, wherein the first performance trace data structure covers synchronous and asynchronous APIs including memory allocation and data transmission; Step S12: Design a second performance trace data structure aligned with the first performance trace data structure for the heterogeneous programming model composed of CPU and DSP, record the interaction process between CPU and different types of accelerators under different supercomputer heterogeneous architectures in a consistent format, and align the storage space occupancy of the first performance trace data structure and the second performance trace data structure.

3. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: In step S2, the types of performance events on the CPU and accelerator of the heterogeneous architecture are aligned, and the performance monitoring unit reading process based on performance event counts on different supercomputer heterogeneous architectures is unified, including: Step S21: Verify whether the existing PMU performance data monitoring interface can support performance data monitoring on the supercomputing heterogeneous architecture; Step S22: If partially supported, unify the performance monitoring unit reading process based on performance event counting on different supercomputer heterogeneous architectures, and finally unify the name and code of the encapsulated performance event.

4. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: The step S3 comprises: Step S31: The trajectory data collected by the specified collection program is organized in the first or second performance trajectory data structure in step S1; Step S32: the host-side performance data collection program obtains the first or second performance trace data structure in S1 from the collection interface callback functions in the three performance domains; Step S33: The files storing the trajectory data are uniformly written into the data buffer structure on the host side, including three types of trajectory files, namely, the host side MPI communication trajectory file, the host side general computing trajectory file and the device side accelerated computing trajectory file, and their corresponding metadata files describing and guiding the analysis.

5. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: The step S4 comprises: Step S41: using the dynamic library interception and stub method in the CPU-side program under the supercomputer heterogeneous architecture, wrapping the original function with the acquisition function, and obtaining the communication and calculation information between process threads, including the communicator and the amount of data transmitted; Step S42: Design a source code instrumentation method for the accelerator-side program under the supercomputing heterogeneous architecture, use llvm and clang to find the corresponding function call in the AST generated after the source code lexical and grammatical analysis, perform code insertion and data collection and transmission, and combine the source code instrumentation with the signal interrupt mechanism between the CPU and DSP to realize various sampling and collection methods.

6. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: The step S5 comprises: Step S51: design two types of acquisition interfaces, synchronous and asynchronous, for users to call, to respectively collect synchronous interface call information and asynchronous interface call information of the entire parallel computing program in the process of MPI communication, general CPU computing and accelerator computing; Step S52: Implementing a collection thread and a data buffer structure for the asynchronous collection interface.

Images