A parallel program performance data monitoring and collecting method for supercomputer architecture

Abstract

The application discloses a kind of parallel program performance data monitoring and acquisition methods for supercomputing architecture, its method includes: for different supercomputing heterogeneous architecture, three performance domains of general computing domain, acceleration computing domain, communication domain are designed unified performance trajectory data format;For different supercomputing architecture, the encapsulation of PMU performance data monitoring interface is carried out;For different supercomputing heterogeneous architecture, unified performance trajectory data acquisition process is designed;Design and realize the performance trajectory data acquisition method based on multi-granularity insertion and sampling;The performance trajectory data acquisition process of three performance domains is unified into two major categories of synchronous and asynchronous acquisition process.The application provides a kind of method for the parallel program on heterogeneous architecture supercomputer to combine synchronous and asynchronous mechanism and carry out light trajectory data acquisition, decouples the upper performance analysis function and specific hardware architecture, reduces the performance analysis tool development cost and performance data acquisition overhead.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of program performance analysis, and particularly relates to a parallel program performance data monitoring and collection method for supercomputer architecture. BACKGROUND

[0002] In a heterogeneous many-core system, the execution of a high-performance computing program is greatly affected by parallelism, communication overhead, and resource contention. Lock mechanisms, serialization, blocking, and load imbalance often occur in multi-process or multi-thread synchronization operations, which have become important issues that must be considered in the development of high-performance computing programs. At the same time, due to the complexity of the supercomputer system architecture and the diversity of application computing behavior, it is difficult to continuously determine the scalability bottleneck of a high-performance computing program by manual means, and a performance analysis tool is needed to automatically analyze the performance of a large-scale parallel application program.

[0003] On a new generation of supercomputer, a parallel application may face an environment with more than ten million cores. The collection of performance traces of the general computing domain, acceleration domain, and communication domain under this new scale is very sensitive to time and space overhead. The research on performance analysis tools has been started for many years. For decades, performance analysis tools have been an indispensable part of the high-performance computing software stack, and many tools have appeared that can be used for various analyses. These tools can be classified according to the information collection and organization principles on which they are based. For example, the hpcrun component in the HPCToolkit based on performance statistics collects relevant performance data of the call context by sampling, and then performs binary analysis, code tracing, etc. through the hpcstruct component and the hpcprof component. SCALANA uses static analysis techniques to achieve the comprehensiveness of performance statistics with similar performance statistics overhead. At runtime, a lightweight technique is used to collect performance data according to the structure graph and generate a program performance graph. Based on the program performance graph, a new root cause detection method can effectively and automatically detect the root cause of scalability loss. The TAU toolkit based on performance tracking, in combination with tools such as the PAPI interface and the Score-P framework, supports many performance statistics and performance tracking modes, as well as a large number of instrumentation methods. Some modeling analysis-based work also tries to balance between high overhead and high accuracy. However, these existing works basically cannot truly balance low overhead and accuracy in the performance data collection of large-scale parallel programs.

[0004] Further, neither the use of a sampling method nor the use of an instrumentation method has been tested and run under the new scale of more than ten million cores. Under the scale of ten million cores, the data collection method needs to be compared and selected, and a balance needs to be struck between overhead and accuracy. It is necessary to combine instrumentation and sampling techniques and cooperate with an efficient asynchronous processing mechanism to achieve lightweight and accurate performance trace collection. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a parallel program performance data monitoring and collection method for supercomputing architecture, which is a multi-angle parallel program performance data monitoring and collection method based on the combination of synchronous and asynchronous methods and sampling and instrumentation methods.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A method for monitoring and collecting performance data of parallel programs for supercomputing architecture, comprising:

[0008] Step S1: Extract the running characteristics of parallel programs on different supercomputer heterogeneous architectures in the general computing domain, the accelerated computing domain, and the communication domain, and design a unified data format. The unified data format is used for the same data format when storing performance trace data on different supercomputer heterogeneous architectures for the same performance domain.

[0009] Step S2: Encapsulate the PMU performance data monitoring interface for different supercomputing heterogeneous architectures, align the performance event types on the CPUs and accelerators of different supercomputing heterogeneous architectures, and unify the performance monitoring unit reading process based on performance event counting on different supercomputing heterogeneous architectures;

[0010] Step S3: The performance trace data collected by different supercomputer heterogeneous architectures in the general computing domain of the parallel program are unified into the data format designed in step S1 and stored in the general computing trace file; the performance trace data collected by different supercomputer heterogeneous architectures in the accelerated computing domain of the parallel program are unified into the data format designed in step S1 and stored in the accelerated computing trace file; the performance trace data collected by different supercomputer heterogeneous architectures in the communication domain of the parallel program are unified into the data format designed in step S1 and stored in the communication trace file; the trace files of different performance domains are organized by calling relationships and timing relationships;

[0011] Step S4: Using different collection methods for the parts of the parallel program within the three performance domains, combining multi-granularity instrumentation and sampling methods to collect performance trace data in the unified data format specified in step S1, and writing the collected data results into the corresponding files according to the storage and organization format specified in step S3;

[0012] Step S5: Design and implement two types of interfaces, synchronous and asynchronous, for collectors to use, and unify the performance trace data collection process of the three performance domains into a callback-based synchronous collection process and an additional collection thread-based asynchronous collection process.

[0013] Furthermore, the unified data format is designed in step S1, specifically including:

[0014] Step S11: design a first performance trajectory data structure for a heterogeneous programming model composed of a CPU and a GPU, the first performance trajectory data structure covering synchronous and asynchronous APIs including memory allocation, data transmission;

[0015] Step S12: design a second performance trajectory data structure aligned with the first performance trajectory data structure for a heterogeneous programming model composed of a CPU and a DSP, record the interaction process between the CPU and different types of accelerators under different supercomputing heterogeneous architectures in a consistent format, and align the storage space occupation of the first performance trajectory data structure and the second performance trajectory data structure.

[0016] Further, the step S2 of aligning the performance event types on the CPU and the accelerator of the heterogeneous architecture unifies the performance monitoring unit reading process based on performance event counting on different supercomputing heterogeneous architectures.

[0017] Step S21: verify whether the existing PMU performance data monitoring interface can support performance data monitoring on supercomputing heterogeneous architectures;

[0018] Step S22: if partially supported, unify the performance monitoring unit reading process based on performance event counting on different supercomputing heterogeneous architectures, and finally unify the name and code of the performance event.

[0019] Further, the step S3 includes:

[0020] Step S31: organize the trajectory data collected by the collection program in the first or second performance trajectory data structure in step S1;

[0021] Step S32: the host performance data collection program obtains the first or second performance trajectory data structure in S1 from the collection interface callback function in the three performance domains;

[0022] Step S33: the file storing the trajectory data is uniformly written by the data buffer structure of the host, including three types of trajectory files: host MPI communication trajectory file, host general computing trajectory file, and device end acceleration computing trajectory file, and the metadata file corresponding to each trajectory file for guiding analysis.

[0023] Further, the step S4 includes:

[0024] Step S41: use a dynamic library interception method in the CPU program under the supercomputing heterogeneous architecture to wrap the original function with a collection function to obtain process thread communication and computing information, including communication sub, and transmission data volume;

[0025] Step S42: a method for inserting a probe into a source code of an accelerator-side program design under a supercomputer heterogeneous architecture, finding a corresponding function call in an AST generated by llvm and clang after a source code lexical syntax analysis, performing code insertion and data collection and returning work, and realizing various sampling and collecting methods by combining a signal interruption mechanism between a CPU and a DSP.

[0026] Further, the step S5 comprises:

[0027] Step S51: designing two types of collecting interfaces of synchronization and asynchronization called by a user, and collecting synchronization interface call information and asynchronization interface call information of an entire parallel computing program in an MPI communication, a general CPU calculation and an accelerator calculation process respectively.

[0028] Step S52: realizing a collecting thread and a data buffer structure for the asynchronization collecting interface.

[0029] The present application has the beneficial effects that:

[0030] The present application unifies a new generation of supercomputer heterogeneous architecture, provides a unified performance trajectory data collection process and interface for an upper layer analysis function, and supports performance analysis work on a supercomputer. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A schematic diagram of an acceleration domain synchronization collection method containing fields in a first performance trajectory data structure designed by the present application;

[0032] Figure 2 A schematic diagram of an acceleration domain synchronization collection method containing fields in a second performance trajectory data structure designed by the present application;

[0033] Figure 3 A classification schematic diagram of sampling and probe collection methods;

[0034] Figure 4 A schematic diagram of a compilation process of llvm and clang for a C language program;

[0035] Figure 5 A schematic diagram of an influence of a probe position on trajectory data collection accuracy in the present application;

[0036] Figure 6 A schematic diagram of a trajectory asynchronization collection method record structure designed by the present application for two different heterogeneous architectures;

[0037] Figure 7 A schematic diagram of an overall interface and software architecture. DETAILED DESCRIPTION

[0038] The present application will be further described below in combination with the drawings and examples.

[0039] The application provides a parallel program performance data monitoring and collection method for supercomputer architecture, designs a series of trajectory data structure formats that align the differences of two supercomputer heterogeneous architectures and synchronous and asynchronous combined collection methods, and reduces the collection time and space overhead of parallel programs thereon. Specifically, it includes:

[0040] Step S1: Extract the running characteristics of parallel programs on different supercomputer heterogeneous architectures in the general computing domain, the accelerated computing domain and the communication domain, design a unified data format, and the unified data format is used for the same performance domain to store performance trajectory data on different supercomputer heterogeneous architectures in the same data format;

[0041] Step S2: Package the PMU performance data monitoring interface for different supercomputer heterogeneous architectures, align the performance event types on the CPU and accelerator of different supercomputer heterogeneous architectures, and unify the performance monitoring unit reading process based on performance event counting on different supercomputer heterogeneous architectures;

[0042] Step S3: The performance trajectory data collected by different supercomputer heterogeneous architectures in the general computing domain of the parallel program is unified into the data format designed in step S1 and stored in the general computing trajectory file; the performance trajectory data collected by different supercomputer heterogeneous architectures in the accelerated computing domain of the parallel program is unified into the data format designed in step S1 and stored in the accelerated computing trajectory file; the performance trajectory data collected by different supercomputer heterogeneous architectures in the communication domain of the parallel program is unified into the data format designed in step S1 and stored in the communication trajectory file; the different performance domain trajectory files are organized through call relationship and time sequence relationship;

[0043] Step S4: Different collection methods are used for parts of the parallel program in the three performance domains, and the performance trajectory data is collected according to the unified data format specified in step S1 by combining multi-granularity instrumentation and sampling methods, and the collected data results are written into the corresponding files according to the storage and organization form specified in step S3;

[0044] Step S5: Design and implement two types of interfaces, synchronous and asynchronous, for collectors to use, and unify the performance trajectory data collection process of the three performance domains into a synchronous collection process based on callback and an asynchronous collection process based on additional collection threads.

[0045] Embodiment

[0046] Taking the synchronous trajectory data structure of the accelerated computing domain as an example, as shown in Figure 1As shown in FIG, an embodiment of the present invention provides a method for monitoring and collecting performance data of parallel programs for supercomputing architecture. The trajectory data structures of the general computing domain and the communication domain are similar to the structures in the figure. Taking two supercomputing heterogeneous architectures of CPU+GPU and CPU+DSP as examples, the architecture is unified based on the method of the present invention, including the following steps:

[0047] Step S1: Extract the running characteristics of programs in the general computing domain, accelerated computing domain, and communication domain on different supercomputer heterogeneous architectures, and design data structures that can reflect the running status of programs in the three performance domains. This ensures that the same data format can be used to store performance traces in the same performance domain on different supercomputer heterogeneous architectures.

[0048] Step S2: Encapsulate the PMU performance data monitoring interface for different supercomputer heterogeneous architectures, and unify the performance monitoring unit reading process based on performance event counting on different supercomputer heterogeneous architectures;

[0049] Step S3: The performance information collected by different heterogeneous supercomputer architectures in the general computing domain of the parallel program is unified into the trajectory data format mentioned in S1 and stored in the general computing trajectory file. Similarly, the information collected by different heterogeneous supercomputer architectures in the accelerated computing domain and communication domain is stored in their respective trajectory files. The trajectory data files of different performance domains are organized by call relationships and timing relationships.

[0050] Step S4: Use different collection methods for the parts of the parallel program within the three performance domains, combine multi-granularity instrumentation and sampling methods to collect performance trace data in the format specified in S1, and write the collected data results into a file according to the storage and organization format specified in S3;

[0051] Step S5: Implement a combined synchronous and asynchronous method to ensure collection accuracy while reducing overhead. Combining the advantages of the sampling and instrumentation collection methods in S4, and considering their characteristics, a sampling method focused on reducing overhead and an instrumentation method with specific accuracy are implemented on the device side. Synchronous callback-based collection and asynchronous interrupt-based collection are combined to achieve high-performance asynchronous processing of performance trace data on the host side, ensuring accuracy and reducing overhead. Furthermore, the performance trace data collection process for the three performance domains, namely the general computing domain, the accelerated computing domain, and the communication domain, is unified into a callback-based synchronous collection interface and an additional collection thread-based asynchronous collection interface. Users can use the same interface to monitor and collect program execution status in the general computing domain, the accelerated computing domain, and the communication domain.

[0052] In one embodiment, the step S1: a unified performance trajectory data format is designed for different domestic supercomputing heterogeneous architectures. Specifically, for the unified performance trajectory data structure of CPU+GPU and CPU+DSP heterogeneous architecture design, the same points under two different heterogeneous architectures are extracted, and the space occupation of the aligned trajectory data structure is ensured to contain the same information, which specifically includes:

[0053] First, running a program on the GPU requires using the CPU side to apply for GPU memory space, and then starting the GPU program running asynchronously through kernel launch. The memory structure of the GPU side is roughly composed of global memory, CU shared memory and register file. After the CPU side applies for GPU global memory space, the contents of the CPU side memory are copied to the GPU side and the required shared memory size is specified, so that these memory spaces can be used in the program running on the GPU side. Therefore, in the CPU+GPU supercomputing heterogeneous architecture, the GPU interface functions to be particularly concerned are kernel launch, synchronization and asynchronous memory copy. For kernel launch, information such as thread dimension and shared memory size is recorded. For memory copy, the sending and receiving addresses are recorded. For other functions such as memory application, synchronization and GPU intercommunication, only the most basic timestamp, call type and correlation ID for timeline alignment are recorded. For example, Figure 1 As shown in the first performance trajectory data structure diagram in the above, the left side is the record structure of kernel launch in the CPU+GPU supercomputing heterogeneous architecture, and the right side is the record structure of memory copy transmission. The data items in the record structure of kernel launch and memory copy operation are different. The memory copy distinguishes between synchronous and asynchronous versions. The asynchronous memory copy operation needs to record the stream carrying the asynchronous operation.

[0054] Secondly, in terms of memory structure, CPU+DSP will have some differences. CPU and DSP will share a part of memory space. After applying for the space occupied by the data needed to be transmitted into the DSP program, it is not necessary to copy the host memory to the device memory like GPU, and initialization work can be directly performed therein. The program running on the DSP side needs to be processed by the matching DSP side compiler, assembler and connector, and finally packaged into an image file and transmitted into the DSP chip. The running of the DSP program on it is based on DSP cluster and thread group. The interface similar to kernel launch needs to transmit the description information of the required parameters. The memory distribution and operation under the two supercomputing heterogeneous architectures are very different, so new data collection structures are designed for the thread group function execution and various DMA transmissions of the DSP side. For example, Figure 2As shown in the diagram of the data structure of the second performance trace, the left side shows the kernel function launch record structure in the CPU+DSP supercomputing heterogeneous architecture, and the right side shows the memory copy record structure. Due to the different hardware environment, the kernel function launch record structure replaces the thread block and grid dimensions corresponding to the CPU+GPU supercomputing heterogeneous architecture with the thread dimension information of the DSP cluster. Asynchronous memory copy is completed by directly specifying DMA operations. In order to distinguish the various types of memory of the CPU and DSP, a transfer type data item is added to the record structure.

[0055] At the same time, in order to ensure that the space occupied by the record structures under the two architectures is not too large, union is used to store different types of structures in an overlapping manner.

[0056] In one embodiment, step S2: aligning the performance event types on the CPU and accelerator of the heterogeneous architecture, and combining the reading process and interface of the performance event registers on the two heterogeneous architectures for unified packaging, specifically includes:

[0057] A unified performance data monitoring interface similar to the Performance API (PAPI) will include a hardware performance event interface for both general-purpose computing and acceleration domains. This unified performance monitoring interface will also include performance monitoring methods based on application-internal behavioral semantics, hardware performance event semantics, and code relevance extraction. Ultimately, this unified interface will be able to collect performance data from multiple layers, including hardware, system software, and parallel applications.

[0058] The hardware implementation is divided into two categories: general computing domain and accelerated computing domain. For the general computing domain, the PAPI interface can collect a subset of performance events supported by the processor. For other performance events not supported by the PAPI interface, the interface tool has been expanded and rewritten. Furthermore, targeted event design was implemented based on the type and number of performance counters in different supercomputer general computing domain hardware structures.

[0059] For the acceleration domain, following the architecture of PAPI, the management and performance data collection interface of the accelerator is called, and the native performance counter reading interface provided on the two types of supercomputing heterogeneous architectures is aligned with the additional performance event collection function to align the usage of the PMU collection interface on the two types of supercomputing heterogeneous architectures. Finally, the common performance events under different architectures are extracted for the CPU and accelerator respectively, and a unified PMU performance data monitoring interface is formed after unifying the name and encoding, which can be used in multiple different supercomputing architectures. The performance counter readings collected can be directly stored in the performance trace record structure described in S1.

[0060] In one embodiment, step S3 unifies the performance trace data collection processes of different supercomputing architectures in the three performance domains of parallel program general computing domain, accelerated computing domain, and communication domain, respectively, to form three different performance trace files using multi-granularity instrumentation as the underlying collection method. The trace files are organized using the correlation ID contained in the structure in S1, specifically including:

[0061] The data structure used to store the trace contains the PMC readings on the hardware. These readings are initialized by the PMU acquisition interface described in S2 before running the program being acquired and are read dynamically during the acquisition process. All readings are of type int64_t and represent the number of occurrences of a specific event. Because the performance events of interest may vary from acquisition to acquisition, the length of this data structure should be variable and additional metadata describing the corresponding events is required when storing it.

[0062] Trajectory data for the general domain is stored in the tr trace file, while trajectory data for the acceleration domain is stored in the etr file. Since the communication domain also functions on the host, its trajectory data can be written to the tr file along with the general domain trajectory data or separately. Because functions on the accelerator and CPU run asynchronously in both architectures, a correlation ID is added to the data structure to identify the accelerator function and the main process (main thread) that launched it. Even though the timeline layout of these different trajectory data volumes may be chaotic due to the possibility of multiple accelerator devices and multi-core functions running in parallel, the correlation ID recorded in the trajectory metadata is required to match the acceleration domain trajectory stored in the etr file with the general domain trajectory in the tr file, facilitating subsequent analysis of the program execution flow. Ultimately, similar structures in both heterogeneous supercomputer architectures occupy the same space, distinguished by fields in the header to ensure accurate access to field information during analysis.

[0063] While the function prototypes and underlying implementations differ significantly between the two heterogeneous supercomputer architectures, some commonalities can be found. Specifically, most interactions between the host and device, as well as the execution flow of device-side functions, are consistent. The host and device are asynchronously and loosely connected, without a strict timing relationship, but with a logical precedence relationship. Therefore, it is possible to design unified monitoring interfaces and trajectory collection interfaces for each architecture:

[0064] (1) Performance data monitoring interface based on PMU. Processors of different architectures are equipped with a monitoring unit (PMU) called a performance monitoring counter (PMC). By using its performance event-based mechanism, the PMC count can be read at any stage of program execution to obtain the running status of a certain part of the program, forming a unified performance data monitoring interface under the multi-supercomputing architecture.

[0065] (2) Synchronous and asynchronous acquisition interfaces in the acceleration domain. This commonality in the two types of supercomputing heterogeneous architectures makes their execution processes for functions on the accelerator essentially the same. Both require marking the device-side program as a function that can be called directly or indirectly by the host. Therefore, two types of acquisition interfaces based on instrumentation, synchronous and asynchronous, are designed for the device-side programs under the two architectures. The mechanism is to insert callback functions before and after the device-side function call, first perform data acquisition, then run the original function, and finally perform data acquisition again. The synchronous acquisition interface corresponds to the synchronous function, and the asynchronous interface corresponds to the asynchronous function. In this way, accurate acquisition of the device-side program can be achieved, forming two types of acquisition interfaces, synchronous and asynchronous, in the acceleration domain.

[0066] (3) General domain acquisition interface. In the general domain, memory operations such as malloc and memcpy can be acquired through dynamic library interception and wrapper instrumentation. Computing libraries in the general domain, such as the parallel programming library OpenMP, have built-in OMPT interfaces for implementing low-level instrumentation and trace data acquisition mechanisms. The pthread library can implement acquisition interfaces using methods similar to source code instrumentation or dynamic library interception. Ultimately, a synchronous acquisition interface based on callback functions will also be formed in the general domain.

[0067] (4) Unified interface for acquisition in the communication domain. The communication domain focuses on MPI inter-process communication. It uses the PMPI tool interface of MPI to write a wrapper to dynamically intercept MPI function calls. Synchronous and asynchronous acquisition interfaces can also be formed for the synchronous and asynchronous communication processes in MPI. The final collected trajectory data can be stored in the same tr file as the trajectory of the general computing domain, forming two loosely coupled trajectories with the etr file of the acceleration domain.

[0068] like Figure 3 As shown, in one embodiment, the above step S4: designs and implements instrumentation and sampling-based performance trace data collection for the new self-developed heterogeneous architecture software and hardware programming model. Based on the characteristics of the programming models of the two heterogeneous architectures, different collection methods are used for different types of programs on the CPU and accelerator, and a combination of multiple instrumentation and sampling methods is used to implement performance trace data collection, specifically including:

[0069] The sampling method periodically records the current execution state of a program. This "periodic" can refer to interrupt signals issued at regular intervals by hardware or software timers, interrupts caused by register overflows, or user-defined asynchronous event mechanisms at the software level. The sampling method uses these interrupt handlers to collect data, capturing a snapshot of program execution at a specific moment. If the hardware is capable of monitoring precise events, this data collection can be even more accurate. Because the sampling frequency is variable, this method generally has little impact on the execution of the program itself. However, its fundamental principle is based on statistics and probability. In trajectory analysis, the number of samples falling within a certain interval can reflect the program's path and execution time. Using a lower sampling frequency can cause many short-lived events to go unnoticed, hindering the generation of accurate performance traces.

[0070] Instrumentation is achieved by inserting acquisition code into a program before it runs using various methods. Mainstream instrumentation methods can be broadly categorized as source code instrumentation, compiler instrumentation, binary rewriting, and dynamic library interception. Essentially, all of these methods insert code at some stage during the compilation of user source code into machine code. CPU+GPU heterogeneous supercomputing architectures, with GPU accelerators, offer a wider range of instrumentation methods. The current LLVM compilation framework and its Clang compiler lack compiler backends specifically designed for CPU+DSP supercomputing architectures, making it impossible to instrument the IR (Intermediate Representation) compiled from user programs using the compiler. Furthermore, static binary rewriting is too cumbersome, while dynamic binary instrumentation requires underlying code and is costly. Due to considerations regarding instrumentation placement and data collection accuracy, source code instrumentation was ultimately chosen. This involves converting the source code into an AST (Abstract Syntax Tree), a structured representation of the source code. Clang's rewriter is then used to identify and instrument specific functions or code sections, and then converts the instrumented code back to the source code. Figure 4 The LLVM and Clang compilation process and the role of the AST within it illustrate this process. A source code undergoes preprocessing and lexical analysis, where it is parsed into tokens. Syntax analysis then generates the target AST for source code conversion. The AST undergoes static analysis to generate IR, which undergoes optional multi-layer optimization before entering the assembler and linker to produce the final executable program. The AST is the first half of the process and is more intuitive than compilation instrumentation, which operates on IR.

[0071] If the instrumentation method is used throughout the entire acquisition process, some short functions may cause a significant increase in the program's time and space overhead after instrumentation. Therefore, it should be used in conjunction with sampling techniques. The interrupt signal transmission mechanism between the CPU and DSP can be used to define an interrupt handler function, and trajectory data can be collected in the interrupt handler function. This largely avoids the "uncertainty" problem in performance analysis tools, that is, the higher the acquisition accuracy, the more severe the distortion of program execution information.

[0072] In one embodiment, step S5 above: designing and implementing two types of interfaces, synchronous and asynchronous, exposed to users, unifying the performance trace data collection process for the three performance domains, and reducing overhead while ensuring collection accuracy, specifically includes:

[0073] A performance data monitoring interface based on the PMU. This interface call can be instrumented into the performance data collection code or calculation code for the three performance domains. Processors of different architectures have monitoring units equipped with performance monitoring counters (PMCs). Using its performance event-based mechanism, PMU counts can be read at any stage of program execution to obtain the operating status of a specific part of the program.

[0074] To achieve lightweight performance trace collection based on a unified interface, it is necessary to make full use of the asynchronous processing mechanism and data buffering mechanism. From the perspective of the instrumentation location, if the instrumentation is only performed before and after the function call on the CPU side (including various memory operation interfaces and kernel function startup interfaces, etc.), some functions are processed synchronously, so the calling statement will not return until the called function returns. However, some asynchronous function running information will be partially lost, such as Figure 5 As shown, for a synchronously processed function (such as memory allocation in the figure), the time point corresponding to the start of execution and return of the function matches the statement that starts it. As long as stubs are inserted at the first position 1 and the second position 2 in the figure, accurate information of the synchronous function can be collected; but for an asynchronous function (such as the kernel function startup in the figure), the calling function will return before the kernel function returns, that is, the kernel function information itself cannot be obtained by inserting stubs at the third position 3 and the fourth position 4 of the host-side API, and the fifth position 5 and the sixth position 6 must also be collected to obtain the precise time when the kernel function runs.

[0075] To address the common issue between the two architectures, namely the lack of accurate data collection for asynchronous events, we can leverage the approach adopted by CUPTI in CUDA. CUPTI (CUDA Profiling Tool Interface) is a suite of performance profiling and tracing interfaces designed for CUDA programs. These tracing interfaces are divided into two types: the Activity API and the Callback API. The former asynchronously records CUDA program activity, utilizing background threads to monitor events such as the calls and returns of CUDA APIs and kernel functions. The latter inserts callback functions before and after specific CUDA APIs, allowing users to customize callback functions for data collection. This is a synchronous mechanism. In a CPU+DSP architecture, source code instrumentation can be used to insert different code before and after API calls in the host program, or even within device-side functions, to achieve data collection. For the synchronous collection mechanism, callback functions are inserted before and after API calls, logically segmented according to API type and code insertion location. The callback functions package incoming API parameters, PMC readings, timestamps, call stack traces, and other information, and then pass them to the callback functions for processing, such as writing trace files. Asynchronous data collection requires a buffering mechanism like a memory pool. Code is inserted before and after user API calls and within user-written kernel functions. Rather than calling callbacks, this code simply passes function execution information to a global data buffer structure and then returns to the original location for continued execution. This structure must significantly reduce dynamic memory allocation overhead while also enabling asynchronous processing of stored trajectory data. When the data buffer reaches a certain size or certain events occur, a flush is performed, writing all current data in the buffer to a file.

[0076] Synchronous and asynchronous acquisition methods differ in their ability to achieve different levels of accuracy and performance impact on the collected application. Consequently, the formats and content of the collected data differ between the two methods. Synchronous callback functions distinguish between API types, collect call parameters, PMC readings, and more, and perform stack unwinding within the callback function, storing the call stack information in the trajectory data structure. Asynchronous acquisition methods may only store simple information such as timestamps, process thread IDs, and function names in the structure, without performing the more expensive stack unwinding. Figure 6 This is a data item in the asynchronous acquisition trace record structure shared by both architectures. The correlation ID is used to organize host-side function calls with device-side function calls. The start and end timestamps are used for subsequent analysis of the program execution path. The union contains three methods for uniquely identifying a device-side thread: threads can be distinguished by device-operation queue, main process-child thread, or other methods. Finally, there is an optional text message for additional description or labeling of the record structure.

[0077] In addition to the aforementioned synchronous and asynchronous acquisition interfaces for the acceleration domain, the synchronous and asynchronous function calls for the general computing domain and the communication domain are also organized into two major categories of acquisition interfaces. When using them, the collector does not need to consider which device the collected function or event occurs on. The integrated synchronous and asynchronous acquisition interfaces can be used for CPU functions, accelerator functions, kernel functions, and inter-process communication functions. This greatly reduces the coupling and usage difficulty between the upper-level analysis functions and the underlying data acquisition functions, and improves the scalability of the system. The final overall interface architecture is as follows: Figure 7 As shown, the three-layer interface abstraction level increases from bottom to top, making it more convenient to use.

[0078] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for monitoring and collecting performance data of parallel programs for supercomputing architecture, characterized in that: include: Step S1: Extract the running characteristics of parallel programs on different supercomputer heterogeneous architectures in the general computing domain, the accelerated computing domain, and the communication domain, and design a unified data format. The unified data format is used for the same data format when storing performance trace data on different supercomputer heterogeneous architectures for the same performance domain. Step S2: Encapsulate the PMU performance data monitoring interface for different supercomputing heterogeneous architectures, align the performance event types on the CPUs and accelerators of different supercomputing heterogeneous architectures, and unify the performance monitoring unit reading process based on performance event counting on different supercomputing heterogeneous architectures; Step S3: The performance trace data collected from different supercomputer heterogeneous architectures in the common computing domain of the parallel program are unified into the data format designed in step S1 and stored in the common computing trace file; Unify the performance trace data collected by different supercomputer heterogeneous architectures in the accelerated computing domain of the parallel program into the data format designed in step S1 and store them in the accelerated computing trace file; Unify the performance trace data collected by different supercomputer heterogeneous architectures in the communication domain of the parallel program into the data format designed in step S1 and store it in the communication trace file; Trajectory files of different performance domains are organized through call relationships and timing relationships; Step S4: Using different collection methods for the parts of the parallel program within the three performance domains, combining multi-granularity instrumentation and sampling methods to collect performance trace data in the unified data format specified in step S1, and writing the collected data results into the corresponding files according to the storage and organization format specified in step S3; Step S5: Design and implement two types of interfaces, synchronous and asynchronous, for collectors to use, and unify the performance trace data collection process of the three performance domains into a callback-based synchronous collection process and an additional collection thread-based asynchronous collection process.

2. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: The unified data format is designed in step S1, specifically including: Step S11: designing a first performance trace data structure for a heterogeneous programming model composed of a CPU and a GPU, wherein the first performance trace data structure covers synchronous and asynchronous APIs including memory allocation and data transmission; Step S12: Design a second performance trace data structure aligned with the first performance trace data structure for the heterogeneous programming model composed of CPU and DSP. Record the interaction process between the CPU and different types of accelerators in different supercomputer heterogeneous architectures in a consistent format, and align the storage space occupied by the first and second performance trace data structures.

3. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: In step S2, the performance event types on the CPUs and accelerators of the heterogeneous architectures are aligned, and the performance monitoring unit reading process based on performance event counts on different supercomputing heterogeneous architectures is unified, including: Step S21: Verify whether the existing PMU performance data monitoring interface can support performance data monitoring on the supercomputing heterogeneous architecture; Step S22: If partially supported, unify the performance monitoring unit reading process based on performance event counting on different supercomputing heterogeneous architectures, and finally unify the name and code of the encapsulated performance event.

4. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: The step S3 comprises: Step S31: The trajectory data collected by the prescribed collection program is organized in the first or second performance trajectory data structure in step S1; Step S32: the host-side performance data collection program obtains the first or second performance trace data structure in S1 from the collection interface callback functions in the three performance domains; Step S33: The files storing the trajectory data are uniformly written to the data buffer structure on the host side, including three types of trajectory files: host side MPI communication trajectory files, host side general computing trajectory files, and device side accelerated computing trajectory files, and their corresponding metadata files describing and guiding the analysis.

5. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: The step S4 comprises: Step S41: using a dynamic library interception and instrumentation method in the CPU-side program under the supercomputer heterogeneous architecture, wrapping the original function with the acquisition function to obtain the communication and calculation information between process threads, including the communicator and the amount of transmitted data; Step S42: Design a source code instrumentation method for the accelerator-side program under the supercomputing heterogeneous architecture, use llvm and clang to find the corresponding function call in the AST generated after the source code lexical and grammatical analysis, perform code insertion and data collection and return work, and combine the source code instrumentation with the signal interrupt mechanism between the CPU and DSP to implement various sampling and collection methods.

6. The method for monitoring and collecting performance data of parallel programs for supercomputing architecture according to claim 1, characterized in that: The step S5 comprises: Step S51: Design two types of acquisition interfaces, synchronous and asynchronous, for users to call, to respectively collect synchronous interface call information and asynchronous interface call information of the entire parallel computing program during MPI communication, general CPU computing, and accelerator computing; Step S52: Implementing a collection thread and data buffer structure for the asynchronous collection interface.

Images