A method for fast locating abnormal processes based on program performance counting

Abstract

The present invention discloses a method for quickly locating an abnormal process based on program performance counting, comprising the following steps: S1, a user submits an application job; S2, a job scheduler initializes a preset hardware performance counter on a processor corresponding to each task while starting the application job; S3, if the program is executed normally until the end, the performance counting based on the performance counter does not interfere with the program; S4, if the program execution is wrong, the error location processing flow is started; S45, an abnormal task is reported for further accurate error cause location. The present invention effectively improves the debugging and development efficiency of developers and improves the utilization rate of system resources.

Description

Technical Field

[0001] The invention relates to a method for quickly locating an abnormal process based on program performance counting, and belongs to the technical field of program debugging. Background Art

[0002] Program debugging is an important part of program development. Due to resource limitations, developers usually only carry out application programming and testing in small and medium-scale parallel environments, use debuggers to track and debug programs to solve most errors, and carry out large-scale production execution after the test is correct.

[0003] Scientific application simulations with larger problem scales and higher precision require larger program parallel scales. Under ultra-large-scale parallelism, programs may have expansion errors, that is, they may run correctly in small and medium-scale parallel environments, but fail to run correctly when expanded to large-scale production execution or even supercomputer scale. In addition, heterogeneous computing makes programming models more complex and the difficulty of debugging programs increases dramatically. Debugging large-scale parallel programs will take up a lot of computing resources and require rich debugging experience.

[0004] The usual debugging technique is to first use a lightweight error location tool to preliminarily locate the abnormal process, and then use a heavyweight full-featured debugger to accurately locate the cause of the error. The lightweight debugging tool effectively achieves high scalability and can be expanded to hundreds of thousands of processes or even higher parallel scales. It obtains part of the program's state information (such as process stack path, MPI call log, etc.) when the program is correct, models and analyzes to discover the program's rules, and detects violations of the rules in production execution to achieve abnormal process detection. However, the method of obtaining information by inserting piles in production execution will interfere with program behavior, and some non-deterministic errors will not be captured. At the same time, it will bring program overhead and reduce performance. Some users who are sensitive to computing performance will be reluctant to use it.

[0005] Supercomputer systems with tens of millions of computing cores are widely used in various fields. Errors that occur during the execution of large-scale parallel applications in production will seriously affect the progress of the project, and debugging large-scale parallel programs is extremely challenging. Developers are increasingly demanding E-level, highly scalable automated debugging and analysis technologies, and being able to quickly locate abnormal processes is extremely important for subsequent in-depth tracking, debugging and analysis.

[0006] Parallel debuggers such as DDT and TotalView are well designed and can be extended to a certain parallel scale to debug parallel programs. However, parallel debuggers are no longer effective when facing applications with a parallel scale of more than 10 million cores. On the one hand, the centralized control and interactive mechanism of parallel debuggers make it difficult to effectively expand to ultra-large scales; on the other hand, the communication, synchronization and other inter-process interactions in the program cause errors to spread rapidly between processes. Developers often find it difficult to obtain effective information from the complex inter-process interaction information and massive status information in ultra-large-scale parallel programs. Summary of the invention

[0007] The purpose of the present invention is to provide a method for quickly locating abnormal processes based on program performance counting, which effectively improves the debugging and development efficiency of developers and improves the utilization rate of system resources.

[0008] To achieve the above object, the technical solution adopted by the present invention is to provide a method for quickly locating abnormal processes based on program performance counting, comprising the following steps:

[0009] S1. User submits application job;

[0010] S2. The job scheduler initializes a preset hardware performance counter on the processor corresponding to each task while starting the application job;

[0011] S3. If the program is executed normally until the end, the performance counting based on the performance counter does not interfere with the program;

[0012] S4. If the program execution error occurs, the error location processing flow is started:

[0013] S41, obtaining, through the job management system, job information of the application job submitted in step S1, wherein the job information includes information of each computing node where the parallel task is located, numbers of the parallel tasks, and mapping information between the parallel tasks and the computing nodes;

[0014] S42, calling the maintenance management system interface, according to the operation information obtained in step S41, grouping the computing nodes that need to collect data according to the middle board of the system, and sending the collection request of each group to the management service board of the corresponding middle board in parallel, and the management service board then sends the requests in the group to each BMC in parallel;

[0015] S43, connecting to the BMC of the computing node, obtaining the request initiated in step S42, bypassing the computing task and the operating system, directly communicating with the processor core through the maintenance interface of the processor, and collecting performance data of each processor core;

[0016] S44: Based on the performance counts obtained in step S43, hierarchical error location is performed:

[0017] Generate process features based on the main core process and slave core thread features, use process features to cluster process groups, and locate abnormal processes;

[0018] Cluster all threads in the abnormal process again to locate the abnormal thread;

[0019] S45: Report an exception task to further locate the precise cause of the error.

[0020] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:

[0021] The present invention meets the user's demand for non-interference of tools on correctly executed programs during production execution, and at the same time provides an E-level oriented, highly scalable automated debugging and analysis method, which effectively improves the debugging and development efficiency of developers and improves the utilization rate of system resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Attached Figure 1 The present invention is a schematic diagram of a method for quickly locating abnormal processes based on program performance counting. DETAILED DESCRIPTION

[0023] Embodiment: The present invention provides a method for quickly locating an abnormal process based on program performance counting, comprising the following steps:

[0024] S1. User submits application job;

[0025] S2. The job scheduler initializes a preset hardware performance counter on the processor corresponding to each task while starting the application job;

[0026] S3. If the program is executed normally until the end, the performance counting based on the performance counter does not interfere with the program;

[0027] S4. If the program execution error occurs, the error location processing flow is started:

[0028] S41, obtaining, through the job management system, job information of the application job submitted in step S1, wherein the job information includes information of each computing node where the parallel task is located, numbers of the parallel tasks, and mapping information between the parallel tasks and the computing nodes;

[0029] S42, call the maintenance management system interface, combine the system organizational structure, based on the hierarchical parallel data acquisition interface, communicate with the BMC connected to the computing node, and collect data quickly and scalably:

[0030] According to the job information obtained in step S41, the computing nodes that need to collect data are grouped according to the middle board of the system, and the collection requests of each group are sent to the management service board of the corresponding middle board in parallel. The management service board then sends the requests in the group to each BMC in parallel;

[0031] S43, connecting to the BMC of the computing node, obtaining the request initiated in step S42, bypassing the computing task and the operating system, directly communicating with the processor core through the maintenance interface of the processor, and collecting performance data of each processor core;

[0032] S44: Based on the performance counts obtained in step S43, hierarchical error location is performed:

[0033] Generate process features based on the main core process and slave core thread features, use process features to cluster process groups, and locate abnormal processes;

[0034] Cluster all threads in the abnormal process again to locate the abnormal thread;

[0035] S45: Report an exception task to further locate the precise cause of the error.

[0036] The further explanation of the above embodiment is as follows:

[0037] The present invention proposes a method for quickly locating abnormal processes based on performance counting of large-scale parallel programs. It sets a lightweight status probe in hardware, and efficiently collects performance data with timing characteristics through a performance counter that records the program execution status provided by a processor, so that the data collection process does not interfere with the program. At the same time, the idea of ​​hardware and software collaborative design enables the method to be effectively expanded to a very large scale, and a hierarchical error location method based on performance characteristics can realize rapid and automatic location of abnormal tasks in large-scale parallel programs. It meets the requirements of not interfering with program performance when the application is correctly executed, and being able to quickly locate erroneous tasks after an abnormality occurs in a large-scale parallel program, thereby realizing highly scalable and rapid error location, effectively improving the debugging and development efficiency of developers, and improving the utilization rate of system resources.

[0038] The specific process is as follows:

[0039] 1) After the user submits the job for execution, the job scheduler initializes the preset hardware performance counters on the processor corresponding to each task while starting the application job;

[0040] 2) If the program executes normally until the end, the hardware-based performance counting does not interfere with the program;

[0041] 3) If the program execution error occurs, the error location processing flow is started:

[0042] a) Obtain job information through the job scheduler, such as information about each computing node where the parallel task is located;

[0043] b) Start the BMC on the computing node, communicate with the processor core through the maintenance interface, bypass the computing tasks and operating system, and collect the performance data of each process and thread (such as the count of L1 ICache, DCache, DMA, etc.) to achieve non-interference with the program;

[0044] c) Through the maintenance management network, combined with the system organizational structure, based on the hierarchical parallel data acquisition interface, highly scalable and fast data collection can be achieved;

[0045] d) Perform hierarchical error location based on performance counts: first locate the abnormal thread based on process and thread characteristics; then locate the abnormal thread from the threads of the abnormal process;

[0046] e) Report abnormal tasks, and developers can further locate the precise cause of the error.

[0047] When the above-mentioned method for quickly locating abnormal processes based on program performance counting is adopted, it meets the user's demand for tools to not interfere with correctly executed programs during production execution, and at the same time provides an E-level, highly scalable automated debugging and analysis method, which effectively improves the debugging and development efficiency of developers and improves the utilization of system resources.

[0048] In order to facilitate a better understanding of the present invention, the terms used in this article are briefly explained below:

[0049] Very large-scale parallel machine: A system composed of many independent computing nodes connected by a high-speed interconnection network. Each computing node contains an independent CPU, memory and operating system. All computing nodes use the same storage by mounting distributed shared storage.

[0050] Tracking and debugging: Developers use debuggers to control programs, track code line execution, obtain intermediate states of program execution, analyze execution paths, and differences between variable values ​​and expectations, discover program anomalies, and locate the root cause of errors.

[0051] Performance counter: A module of the processor performance monitoring unit (PMU), provided by the processor, used to monitor and count program execution events.

[0052] Job scheduler: It is an important part of the job management system. Its function is to select the most suitable job from the jobs waiting to be run as fairly and efficiently as possible to schedule and run, so as to meet the real-time operation requirements of user projects and improve the utilization of system resources.

[0053] BMC (Baseboard Management Controller): refers to a special service processor on the motherboard of the supervised computing node. It does not rely on the processor, BIOS or operating system on the computing node to work. It can still perform operations such as power on and off and extract information when the operating system is not responding or not loaded, and communicate with the system administrator through an independent connection.

[0054] Maintenance and management network: refers to the network system used to maintain and manage supercomputers, which is generally built using Ethernet with low latency.

[0055] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for quickly locating abnormal processes based on program performance counting, characterized in that: The following steps are involved: S1. User submits application job; S2. The job scheduler initializes a preset hardware performance counter on the processor corresponding to each task while starting the application job; S3. If the program is executed normally until the end, the performance counting based on the performance counter does not interfere with the program; S4. If the program execution error occurs, the error location processing flow is started: S41, obtaining, through the job management system, job information of the application job submitted in step S1, wherein the job information includes information of each computing node where the parallel task is located, numbers of the parallel tasks, and mapping information between the parallel tasks and the computing nodes; S42, calling the maintenance management system interface, according to the operation information obtained in step S41, grouping the computing nodes that need to collect data according to the middle board of the system, and sending the collection request of each group to the management service board of the corresponding middle board in parallel, and the management service board then sends the requests in the group to each BMC in parallel; S43, connecting to the BMC of the computing node, obtaining the request initiated in step S42, bypassing the computing task and the operating system, directly communicating with the processor core through the maintenance interface of the processor, and collecting performance data of each processor core; S44: Based on the performance counts obtained in step S43, hierarchical error location is performed: Generate process features based on the main core process and slave core thread features, use process features to cluster process groups, and locate abnormal processes; Cluster all threads in the abnormal process again to locate the abnormal thread; S45: Report an exception task to further locate the precise cause of the error.

Images