Large-scale parallel program-oriented operation failure detection and positioning method

Abstract

The invention provides a running failure detection and positioning method for a large-scale parallel program. The method comprises the steps of performance key kernel function analysis, performance key data analysis, irregular calculation kernel function analysis, performance fluctuation detection, performance fluctuation diagnosis and performance modeling. According to the method, for a rule calculation kernel function, operation failure detection of performance fluctuation is carried out smoothly by using data based on a fixed workload, and then performance fluctuation diagnosis is carried out; according to the method, performance prediction is carried out through performance modeling aiming at an irregular calculation kernel function, then the predicted performance is compared with the actual measurement performance, so that the operation failure problem of performance fluctuation is detected, and performance fluctuation diagnosis is further carried out. According to the method provided by the invention, the common problems of difficult detection, difficult analysis and difficult positioning of operation failure of irregular calculation kernel function performance fluctuation in large-scale parallel application are solved, and guidance help and support are provided for debugging of more large-scale applications.

Description

Technical Field

[0001] The present invention relates to the field of large-scale computing technology, and in particular to a method for detecting and locating operation failures of large-scale parallel programs. Background Art

[0002] Performance fluctuation is a common runtime failure in large-scale parallel applications. It can lead to unexpected, difficult-to-reproduce performance degradation, increasing runtimes. It can also hinder researchers and developers' judgment of the effectiveness of code optimizations, increasing optimization and testing costs. Detecting and diagnosing performance fluctuations is therefore crucial for computer systems and parallel applications. However, the causes of performance fluctuations are complex, making it difficult to reproduce the fluctuations and locate the program segments that cause them. Furthermore, sparse matrix multiplication is a fundamental computational kernel function in many scientific computations (such as linear algebra solvers), data mining, and deep learning. However, due to the irregular memory access of sparse matrices, the performance of sparse matrix multiplication is not directly related to the matrix size or the number of nonzero elements. Existing methods are unable to effectively detect the performance variance of sparse matrix multiplication, making it difficult to comprehensively detect and locate runtime failures in large-scale parallel applications involving sparse matrix computations. Summary of the Invention

[0003] In order to solve the problems existing in the prior art, embodiments of the present application provide a method, apparatus, computing device, computer storage medium, and product containing a computer program for detecting and locating runtime failures of large-scale parallel programs.

[0004] In a first aspect, an embodiment of the present application provides a method for detecting and locating runtime failures of large-scale parallel programs, comprising: collecting features of sparse matrices based on array names obtained from irregular computing kernel functions; establishing a performance model for irregular computing kernel functions based on features closely related to sparse matrices and hardware architecture; using the performance model to predict the performance of irregular computing kernel functions, and comparing it with measured performance to detect performance fluctuations; collecting performance data, and using an analysis method based on performance and debugging data to gradually identify potential causes and output diagnostic information.

[0005] In some possible embodiments, before collecting the features of the sparse matrix based on the array name obtained according to the irregular calculation kernel function, it also includes: identifying performance-critical kernel functions and their inputs by analyzing source code, the kernel functions including regular kernel functions and irregular kernel functions; using an executable program compiled based on DWARF to collect debugging data and construct a function call path.

[0006] In some possible embodiments, for the rule kernel function, data smoothing based on a fixed workload is used to perform performance fluctuation detection.

[0007] In some possible embodiments, when a large number of performance fluctuations are detected, it is determined whether the program workload has changed. If the workload has changed, the performance model is re-modeled based on the changed workload before the performance fluctuation detection is performed.

[0008] In some possible embodiments, performance fluctuation diagnosis utilizes performance counters and debugging data for analysis to generate hardware diagnostic information and software diagnostic information.

[0009] In some possible embodiments, performance modeling extracts features related to sparse matrices and hardware architecture as input to the regression problem, trains the performance model using a ridge regression method, the training dataset consists of matrices from the fields of computational fluid dynamics and structural engineering from the SuiteSparse matrix collection, determines the ideal regularization parameter through a cross-validation method, and evaluates the prediction accuracy of the model.

[0010] In the second aspect, an embodiment of the present application provides a device for detecting and locating runtime failures of large-scale parallel programs, including: an acquisition module for collecting features of sparse matrices based on array names obtained by irregular computing kernel functions; a processing module for establishing a performance model for irregular computing kernel functions based on features closely related to sparse matrices and hardware architectures; the processing module is also used to use the performance model to predict the performance of the irregular computing kernel function, and compare it with the measured performance to detect performance fluctuations; the processing module is also used to collect performance data, and use an analysis method based on performance and debugging data to gradually identify potential causes and output diagnostic information.

[0011] In a third aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method as described in any one of the first aspects.

[0012] In a fourth aspect, an embodiment of the present application provides a computing device comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method as described in any one of the first aspects is executed.

[0013] In a fifth aspect, an embodiment of the present application provides a product comprising a computer program, which, when the computer program product runs on a processor, enables the processor to execute the method as described in any one of the first aspects.

[0014] This method solves the problem of difficult detection, analysis, and location of operational failures caused by irregular performance fluctuations of computing kernel functions that are common in large-scale parallel applications. It will provide guidance, assistance, and support for debugging more large-scale parallel applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is a flowchart of an embodiment of a method for detecting and locating operational failures provided in an embodiment of the present application;

[0017] Figure 2 This is a flow chart of performance modeling provided by an embodiment of the present application; DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0019] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0020] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0021] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0022] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0023] To facilitate understanding of the embodiments of the present application, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0024] A specific embodiment of the runtime failure detection and localization method for large-scale parallel programs provided by the present invention is implemented using Ptrace, a monitoring mechanism provided by the Linux system. A parent process can observe and control the execution of other processes and inspect their performance events and registers without root privileges. Switching between the sampling phase and the detection phase is achieved by attaching and detaching the parent process from another process.

[0025] In view of this, the present application embodiment provides a method for detecting and locating an operation failure, please refer to Figure 1 First, analyze the source code for key performance kernel functions. Then, analyze key performance data and irregular computing kernel functions. Then, perform performance modeling based on the results of the irregular computing kernel function analysis. For regular computing kernel functions, use data smoothing based on the same workload to detect performance fluctuations. For irregular computing kernel functions, compare predicted performance with measured performance based on the analysis results and performance model to complete performance fluctuation detection. If significant performance fluctuations are detected, determine whether the program workload has changed. If so, re-perform performance modeling before performing performance fluctuation detection. Finally, based on the detected performance fluctuation information and combined with debugging information, perform performance fluctuation diagnosis, determine the cause of the performance fluctuation, and output diagnostic information. The following steps are explained in detail.

[0026] A: Performance-critical kernel function analysis, identifying performance-critical functions in the program (especially GPU kernels) and distinguishing regular / irregular computing patterns.

[0027] The analysis of performance-critical kernel functions is mainly implemented through the LLVM compilation framework. First, the LLVM Intermediate Representation (LLVM IR) is obtained based on the source code. Then the LLVM built-in function calling convention interface is applied to identify the CUDA / HIP kernel function and construct a set of performance-critical kernel functions. Since the workload of the kernel function is related to its input parameters, parameter identification is applied when the kernel function is identified as a CUDA / HIP kernel function definition. This embodiment uses parameter analysis technology to identify and extract all input parameter lists of the kernel function from the source code based on the constructed performance-critical kernel function name. For example, if the code of an example is represented as "__global__void MatrixMulCUDA(float*C,float*A,float*B,int wA,int hB){xxx}", the function will be identified as a kernel function type, and its parameters (C, A, B, wA, hB) will be extracted.

[0028] If the number of non-zero elements in a matrix is small enough, the matrix can be called a sparse matrix. Since the non-zero elements in a sparse matrix row may not be continuous, its memory access mode is also correspondingly discontinuous compared to a dense matrix. Therefore, according to the memory access mode of the matrix, its corresponding calculation kernel function is divided into regular calculation kernel function and irregular calculation kernel function. When storing matrices, compressed storage formats such as Compressed Sparse Row (CSR) and Coordinate (COO) are generally used. Therefore, if the product of the number of rows and columns does not match the matrix size, the corresponding calculation kernel function will be identified as an irregular calculation kernel function. This embodiment uses an array to analyze whether the identified calculation kernel function is a regular calculation kernel function or an irregular calculation kernel function, which facilitates the subsequent selection of the corresponding method for performance testing.

[0029] B: Analyze key performance data, dynamically build function call paths at runtime, and locate the code location where performance events occur.

[0030] For the analysis of key performance data, debugging data, especially function call paths, is crucial for finding the causes of performance fluctuations. This embodiment uses single-step tracing and parsing debugging metadata to dynamically construct function call paths at runtime without the need to inspect source code.

[0031] First, the Linux Ptrace system call interface is used to allow the parent process to monitor the registers, memory and instruction stream of the child process to control the child process, implement operations such as pausing and resuming execution, reading registers and accessing memory, and obtain the program counter value (PC), which represents the address of the instruction currently being executed or just executed, and is the key entry point for determining the execution context.

[0032] Subsequently, this embodiment can extract all compilation units (CU) from the DWARF (Debugging With Attributed Record Formats) section. Each CU usually corresponds to a set of source files or functions, and defines the address range it covers in the binary file. This embodiment will determine whether the current PC belongs to a certain CU, thereby narrowing the parsing scope. After finding the relevant CU, traverse the function-level debugging entries recorded therein. DWARF encodes the definitions, boundaries, and attributes of all functions in a structured format. By traversing this information, it is determined whether the current instruction is within a specific function address range, thereby deducing the function name of the function where the current PC is located, and establishing an initial mapping from "address" to "function semantics."

[0033] Finally, the line number table associated with the CU is consulted, the current PC is matched with the address entry in the .debug\_line section, and the corresponding source code file name and line number are retrieved to accurately locate the source code line. Each executed instruction is interpreted as "the program is currently executing line X in function Y." For example, main.c:12 indicates that the program is located at line 12 of the main.c file, which facilitates subsequent performance fluctuation diagnosis.

[0034] C: Analysis of irregular computing kernel functions, for sparse matrix operations, dynamically extracting matrix features and predicting performance.

[0035] The analysis of irregular computing kernel functions mainly focuses on the performance analysis of irregular computing kernel functions such as sparse matrix multiplication. Since the performance of irregular computing kernel functions is closely related to the distribution of non-zero elements in the matrix, it is impossible to directly predict the computing time by the matrix size like regular computing kernel functions. Therefore, this embodiment needs to dynamically obtain the specific characteristics of the sparse matrix at runtime and predict its performance based on these characteristics. Modern compilers will generate rich metadata during the compilation process. These metadata are the basis for precise debugging. In DWARF, debugging information is organized into a hierarchy of debug information entries (DIEs), each DIE describing a specific program element, such as a function, variable, data type, or source code line.

[0036] First, the Ptrace system call interface is used to dynamically construct the function call path by single-step tracking program execution and parsing debugging metadata, thereby obtaining information about the currently executed function. Then, by traversing all DW_TAG_variable entries within the function scope, the target array is identified and matched, and the basic type of the array is parsed based on its DW_AT_type attribute. Next, the libunwind library is used to perform stack analysis, obtain the canonical frame address (CFA) of the current function frame, and use the DWARF expression evaluator to parse the variable's DW_AT_location attribute to determine the runtime address of the array. Finally, the PTRACE_PEEKDATA interface is used to read the array content and decode and format it based on its DW_AT_encoding attribute, ultimately obtaining the characteristics of the sparse matrix, including matrix size, non-zero element distribution, non-zero element locality, etc., for subsequent performance prediction. The characteristics are input into a pre-trained ridge regression model to predict kernel execution time. In this embodiment, there is no need to collect a complete data set in advance, and different sparse matrices can be adapted in real time.

[0037] D: Performance fluctuation detection, discovering abnormal performance fluctuations during program operation.

[0038] The main function of performance fluctuation detection is to detect whether there are performance fluctuations during the execution of a program. For rule-based computing workloads, this embodiment detects performance fluctuations by analyzing segments of a fixed workload. The host side regularly sends the execution time and workload-related parameters (such as matrix size) to the device side for analysis. The analysis mainly compares the execution time of the same workload. If the execution time of a process deviates significantly from other processes, it is considered that the process has performance fluctuations. In some possible embodiments, the kernel with the shortest running time can be used as a benchmark. If the execution time exceeds 15% of the benchmark time, it is determined that it has a performance problem.

[0039] For irregular computing workloads, this embodiment uses a trained performance model to predict its execution time and compares the predicted time with the actual measured time. If the difference between the predicted time and the measured time exceeds a preset threshold, it is considered that there is performance fluctuation. However, when a large amount of performance fluctuation is detected, this embodiment will determine whether the program workload has changed, such as the matrix size, the number of non-zero elements, and the sparsity of sparse matrices. If the workload has changed, it returns to the sampling phase, collects new input parameters, and rebuilds the training performance model to adapt to the workload change.

[0040] E: Performance fluctuation diagnosis, locate the root cause of performance fluctuations.

[0041] Performance fluctuation diagnosis primarily analyzes detected performance fluctuations and locates their potential causes. This embodiment utilizes performance counters and debug data for analysis. Performance counters provide accurate architecture-related information, such as occupancy, total global load / store requests, number of registers used in threads, DRAM read / write counts, and cache hit / miss counts. Debug data provides execution context information for program code, such as source code location, function call paths, and function stack information.

[0042] Specifically, the collected performance counters are normalized and subjected to Pearson correlation coefficient analysis with the execution time. The performance counters are sorted according to the correlation coefficient, and the overhead of each counter is tested to finally determine the list of performance counters used for diagnosis. Then, the data of these performance counters are collected, and hardware diagnostic information is generated based on their impact and duration. In addition, this embodiment also uses debugging information such as the function call stack obtained by performance key data analysis to generate software diagnostic information, including source code location, function call path, function stack information, and even array information, to help users further analyze the software causes of performance fluctuations.

[0043] This embodiment supports three levels of software diagnostic information collection, and users can choose according to their needs to balance the detail of the diagnostic information and performance overhead.

[0044] Based on the operational failure detection and location method, the present application also provides a performance modeling method to establish a high-precision performance prediction model for irregular kernels. Figure 2 This is a flow chart of a performance modeling process provided by an embodiment of the present application. Figure 2 As shown in the figure, performance modeling will build a performance model for irregular computing kernel functions (such as sparse matrix multiplication) to predict their execution time. Figure 2, this embodiment extracts a series of features related to sparse matrices and hardware architecture from the input matrix, including matrix size features (such as the number of rows, columns and non-zero elements), non-zero element distribution features (such as the distribution of non-zero elements on rows and columns), non-zero element locality features (such as the average distance between consecutive non-zero elements and the Gini coefficient of the number of non-zero elements between column blocks) and hardware features (such as the number of streaming multiprocessors, L1 and L2 cache sizes and memory bandwidth). The above features are used as input to the regression problem, and the performance model is trained using the ridge regression method to solve the collinearity problem in multiple linear regression. The training dataset consists of matrices from the fields of computational fluid dynamics and structural engineering from the SuiteSparse matrix collection. The ideal regularization parameter is determined by the cross-validation method, and the prediction accuracy of the model is evaluated. Experimental results show that the performance model can effectively predict the execution time of sparse matrix multiplication with an average prediction error of 4.73%.

[0045] The above is an introduction to the runtime failure detection and location method for large-scale parallel programs provided by the embodiment of the present application. By analyzing the source code, the performance-critical kernel functions and their inputs on the GPU are identified, including regular and irregular kernels. Then, the DWARF-compiled executable program is used to collect debug data and construct the function call path. Sparse matrices are collected for irregular computing kernels, and performance is predicted using a machine learning model at runtime. The operation of regular and irregular kernels is then monitored to detect performance fluctuations. Regular kernels are detected using performance normalization and data smoothing, and irregular kernels are detected using comparative prediction and measurement performance. Performance data is then collected and the cause of performance fluctuations is analyzed using performance and debug data. The performance model is a performance model established for irregular computing kernels based on sparse matrices and GPU architectural characteristics. This method is particularly suitable for the problem of difficult detection, analysis, and location of runtime failures caused by performance fluctuations of irregular computing kernel functions that are common in large-scale parallel applications. Compared with traditional solutions, its performance fluctuation detection sensitivity is greatly improved, the root cause location time is shortened, and reliable technical support is provided for the performance optimization of large-scale parallel programs.

[0046] It is understandable that the size of the sequence number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. All or part of any features of any embodiment of the present application can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of the present application.

[0047] Based on the methods in the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the methods in the above embodiments.

[0048] Based on the methods in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the methods in the above embodiments.

[0049] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0050] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0051] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0052] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

Claims

1. A method for detecting and locating operational failures in large-scale parallel programs, characterized in that: The following steps are involved: Collect the features of the sparse matrix based on the array name obtained by the irregular calculation kernel function; Build a performance model for irregular computing kernel functions based on characteristics closely related to sparse matrices and hardware architecture; predicting the performance of the irregular computing kernel function using the performance model and comparing it with the measured performance to detect performance fluctuations; Collect performance data and use analysis methods based on performance and debugging data to gradually identify potential causes and output diagnostic information.

2. The method according to claim 1, characterized in that Before collecting the features of the sparse matrix according to the array name obtained by the irregular calculation kernel function, the method further includes: Identifying performance-critical kernel functions and their inputs by analyzing source code, wherein the kernel functions include regular kernel functions and irregular kernel functions; Use DWARF-compiled executables to collect debug data and build function call paths.

3. The method according to claim 1, characterized in that For the regular kernel function, data smoothing based on fixed workload is used to detect performance fluctuations.

4. The method according to claim 1, wherein When a large number of performance fluctuations are detected, it is determined whether the program workload has changed. If the workload has changed, the performance model is re-modeled based on the changed workload before the performance fluctuation detection is performed.

5. The method according to claim 1, wherein Performance fluctuation diagnosis uses performance counters and debugging data for analysis to generate hardware diagnostic information and software diagnostic information.

6. The method according to claim 1, characterized in that Performance modeling extracts features related to sparse matrices and hardware architecture as input for the regression problem. The performance model is trained using the ridge regression method. The training dataset consists of matrices from the fields of computational fluid dynamics and structural engineering from the SuiteSparse matrix collection. The ideal regularization parameter is determined through cross-validation, and the prediction accuracy of the model is evaluated.

7. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

8. A computing terminal comprising a memory and a processor, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.

Images