Large-scale computing cluster fault tracking system oriented to production environment

Abstract

A large-scale computing cluster fault tracking system oriented to a production environment comprises a high-performance computing program monitoring interface based on an OTF, a high-performance computing application program monitoring system oriented to the production environment and a fault analysis and prediction system based on machine learning. Wherein the OTF-based high-performance computing program monitoring interface is used for describing a program state based on an OTF technology, the production environment-oriented high-performance computing application program monitoring system is used for monitoring the program state and generating monitoring data, and the machine learning-based fault analysis and prediction system analyzes and predicts a fault based on the monitoring data. According to the large-scale computing cluster fault tracking system oriented to the production environment, automatic instrumentation is carried out on a distributed high-performance computing program running on a large-scale computing cluster, the state of each node and the state of each program in the cluster are analyzed according to the collected data, and a user can conveniently and rapidly find the state and the fault between the nodes.

Description

Technical Field

[0001] The present invention relates to the field of high performance computing, and in particular to a large-scale computing cluster fault tracking system for production environments. Background Art

[0002] High-performance computing technology is an important branch of computer science, usually referring to computing systems and environments that use multiple processors on a single computer or multiple independent computers to form a cluster. High-performance computing technology makes up for the lack of computing performance of a single computer by enabling multiple computers to work together. Currently, high-performance computing technology has been widely used in specific application scenarios in multiple disciplines and fields such as physics, astronomy, medicine, and geology. Many universities, companies, research institutes and other institutions around the world are using public high-performance computing services or building their own private computing clusters to meet actual needs.

[0003] However, since high-performance computing technology requires the collaboration of multiple processors and multiple computers, the development of high-performance computing programs requires extra attention to additional processes such as phase synchronization between nodes, data transmission, and error handling, and requires the writing of additional code. Compared with the development of stand-alone programs, the development of programs for high-performance computing clusters is difficult and inefficient, and it is difficult to detect potential faults and performance bottlenecks in a timely manner, wasting a lot of human resources.

[0004] In order to solve the above-mentioned difficulties in developing high-performance computing programs, many organizations have conducted research on fault analysis and performance tuning of high-performance computing programs. Among them, the best ones include TAU (Tuning and Analysis Utilities) software developed by the relevant team of the University of Oregon. However, the current mainstream high-performance computing fault tracking tools are all oriented to the development environment, and there are few fault tracking tool systems for the production environment. It is true that the development environment can reflect the characteristics of the production environment to a certain extent, but there are differences after all. It cannot be simply assumed that if the program is debugged in the development environment, it can be safely used in the production environment. Moreover, the factors that affect the correctness of program operation are not only determined by the program itself, but also the fluctuations of the disk, network, memory, etc. in its operating environment may also cause the program to fail. Such accidental fluctuations often occur irregularly and for a short time. By the time the operation and maintenance personnel find that the program is not running normally, the culprit has often disappeared. Such accidental fluctuations are difficult to simulate in the development environment, which greatly increases the difficulty of finding and solving problems.

[0005] Distributed traceback is a method for analyzing and monitoring applications, especially those built with microservices architecture. Distributed traceback helps to pinpoint where failures occur and what causes poor performance. OTF (Open Tracing Format) is not a specific program or technology, but a recommended specification proposed by the Cloud Native Computing Foundation (CNCF), which specifies how to define the concepts in a microservices architecture, how to describe a request built on a microservice, and how to trace these requests. OTF consists of an API (Application Programming Interface) specification, frameworks and libraries that implement the specification, and project documentation. As long as the OTF API specification is followed, developers can use any tool to add custom tracing items to the code.

[0006] OTF is an interface specification used for microservice link tracking. In the mobile Internet, microservices can call each other in any number of layers, and each service may depend on any number of other services. The order of calls between nodes is not fixed and may change with changes in service requirements. In high-performance computing, however, it is rare for nodes to call each other at will. It is usually the master node that assigns tasks to each child node, and the child node returns the results to the master node after the calculation is completed. Therefore, it is impossible to directly apply the OTF link tracking technology for microservices to the field of high-performance computing.

[0007] As high-performance clusters get older, the frequency of the above problems increases. If not solved in time, it can easily lead to frequent program failures, or even successful program outputs wrong results, wasting a lot of time and energy. Therefore, a program is needed that can monitor the status of the cluster and program at runtime, so that when problems occur in the cluster or program, the fault can be discovered and repaired in time.

[0008] Based on this, there is room for improvement in fault tracking in large-scale computing clusters. Summary of the invention

[0009] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a large-scale computing cluster fault tracing system for production environments. By combining the ideas of OTF and distributed link tracing, a method for fault analysis and fault prediction of high-performance computing clusters and programs for production environments is provided, and it is efficiently implemented according to the actual requirements and limitations of the production environment.

[0010] The large-scale computing cluster fault tracking system for production environment of the present invention is for the following purposes: to monitor the status of large-scale distributed clusters and their running high-performance computing programs, and to find abnormal status in time; to monitor high-performance computing programs by using the lightest and least intrusive means possible, so that a large amount of source code does not need to be modified when monitoring high-performance computing programs; to capture as much runtime information as possible on the basis of the lowest runtime overhead possible, so as to find faults; to use data mining related algorithms to process the captured information in real time, to find faults as early as possible and to notify relevant maintenance personnel.

[0011] In order to achieve the above object, the technical solution provided by the present invention is:

[0012] A large-scale computing cluster fault tracking system for production environments, comprising: an OTF-based high-performance computing program monitoring interface, a production-oriented high-performance computing application monitoring system, and a machine learning-based fault analysis and prediction system, wherein the OTF-based high-performance computing program monitoring interface is used to describe program status based on OTF technology, the production-oriented high-performance computing application monitoring system is used to monitor program status and generate monitoring data, and the machine learning-based fault analysis and prediction system analyzes and predicts faults based on the monitoring data;

[0013] The high-performance computing application monitoring system for the production environment is configured with an application-level acquisition module, a system-level acquisition module, and a data transmission and storage module. The application-level acquisition module is configured to collect information related to the high-performance computing program and the business during the operation of the high-performance computing program, the current stage of the high-performance computing program, and the task progress. The system-level acquisition module is configured to collect information related to the cluster and the operating system. The data transmission and storage module is configured to transmit and store the data collected by the application-level acquisition module and the system-level acquisition module, and finally save the data in a database.

[0014] The machine learning-based fault analysis and prediction system is configured to use machine learning related technologies to automatically classify and predict data collected by the high-performance computing application monitoring system for production environments, thereby accurately evaluating the current status of the cluster and discovering possible faults in the cluster in advance.

[0015] In some embodiments, the OTF-based high-performance computing program monitoring interface is configured with a task start interface, a task end interface, and a progress report interface, wherein the master node calls the task start interface when assigning a computing task to a child node, and then the child node calls the progress report interface to report the current computing progress, and when the calculation is completed, the master node calls the task end interface to report that the computing task has ended.

[0016] In some embodiments, if it is necessary for a child node to obtain the results of other child nodes during the calculation process, a task start interface and a task end interface are called to perform a nested call.

[0017] In some embodiments, the OTF-based high performance computing program monitoring interface has a Tracer class as a starting point for program monitoring.

[0018] In some embodiments, the Tracer class exists as a singleton mode, and there is only one instance in the program, and the Tracer interface is implemented by the Tracer class.

[0019] In some embodiments, the Tracer class includes a Task subclass and an IO subclass, and the OTF-based high-performance computing program monitoring interface is configured to subdivide the Span into a Task subclass and an IO subclass, corresponding to running a sub-Task and starting an IO operation, respectively.

[0020] In some embodiments, in the case of a Task subclass, the OTF-based high performance computing program monitoring interface is configured to allow the same Task to be created on multiple nodes.

[0021] In some embodiments, in the case of an IO subclass, in an OTF-based high-performance computing program monitoring interface, error information of an IO operation error is collected only when an IO operation error occurs.

[0022] In some embodiments, the Tracer interface includes a Task interface and an IO interface, and the Task interface and the IO interface are used to monitor computing tasks and data IO, respectively.

[0023] In some embodiments, the program state includes system resources consumed when the program is running and internal information of the program.

[0024] In some embodiments, the application layer acquisition module includes a build-time symbol renaming program data state acquisition module, a runtime library injection program data state acquisition module, and a system call monitoring program data state acquisition module.

[0025] The build-time symbol renaming program data status acquisition module is configured to rename the symbols of its ELF file when the program is built, rename the original call to a specific function to another function, realize automatic instrumentation of the application, and collect program status.

[0026] The runtime library injection program data status collection module is configured to automatically inject the third-party library that the program depends on, and add the program status collection function to the third-party library.

[0027] The system call monitoring program data status acquisition module is configured to implement the tracking of system calls.

[0028] In some embodiments, the system-level collection module includes a module for collecting resource consumption metrics when a high-performance computing program is running. The module for collecting resource consumption metrics when a high-performance computing program is running is configured to use various tools to monitor the program running status in real time during the program running process, collect the resources occupied by the program at each moment during the running process, and summarize and organize them to form the program's running characteristic information. The occupied resources include CPU and memory.

[0029] In some embodiments, the symbol renaming program data status collection module is configured to perform instrumentation collection on IO related functions, and the IO related functions include the following IO functions: write, read, connect, accept, open, close, recv, send, readv and writev.

[0030] In some embodiments, the runtime library injection program data status collection module is configured to replace vtable->__write with a function with data collection function when the GPP program starts, thereby achieving seamless recording of all log outputs of the program without modifying the code.

[0031] In some embodiments, the runtime library injection program data status acquisition module is configured to modify the virtual function table area permissions through the mprotect system call and then replace the virtual function table content of the library space.

[0032] In some embodiments, the system call monitoring program data status collection module implements the tracking of system calls through the Ptrace function, and discovers system call failures by monitoring the results of each system call.

[0033] In some embodiments, the high performance computing program runtime resource consumption metric collection module is configured to design an independent thread for periodically reading / proc / <pid> / statm and / proc / <pid> / stat file, and save the parsed information into the database to collect CPU usage and memory usage.

[0034] In some embodiments, the data transmission and storage module includes a high-performance computing program status data transmission and storage module, and the high-performance computing program status data transmission and storage module includes traffic collection specifications, IO error collection specifications, task status collection specifications, log collection specifications, and node status collection specifications. Each node should follow the relevant specifications when sending collected data to the database.

[0035] In some embodiments, the high performance computing program state data transmission and storage module is configured with two levels of buffer queues.

[0036] In some embodiments, the machine learning-based fault analysis and prediction system includes a LightGBM-based classification module. The LightGBM-based classification module is configured to use a grid search algorithm to find the optimal parameters when classifying the standardized data using the LightGBM algorithm. The evaluation indicators are the AUC curve and F1 value of the model on the test set.

[0037] In some embodiments, the grid search algorithm selects the number of leaves, the learning rate, the number of base learners, the maximum depth of the decision tree, and data sampling as the hyperparameters to be adjusted.

[0038] In some embodiments, the LightGBM-based classification module uses the Antarex dataset to train a logistic regression-based classification model.

[0039] In some embodiments, the machine learning-based fault analysis and prediction system includes a KMeans clustering module based on time series data. The KMeans clustering module based on time series data automatically clusters the resource usage of each node collected from the cluster, thereby attempting to discover abnormal points.

[0040] The beneficial effects of the present invention are:

[0041] The large-scale computing cluster fault tracking system for production environment of the present invention designs a set of interfaces for link tracking and status collection of high-performance computing clusters based on OTF. The interface uses a simple and clear API to collect data from various aspects of computing programs in the cluster, and is easy to understand and use. It automatically inserts distributed high-performance computing programs running on large-scale computing clusters, and users can analyze the calls and dependencies between program nodes without writing additional code, and analyze the status of each node and program in the cluster based on the collected data, so that users can quickly discover the status and faults between nodes. The OTF-based high-performance computing program monitoring interface and the data captured by the high-performance computing application monitoring system for production environment can be used for automatic fault analysis and fault prediction. When a large-scale cluster system fault is detected or a failure of an application running on the cluster occurs, the relevant operation and maintenance personnel can be notified in time by phone or email, and the alarm can be marked according to the feedback of the operation and maintenance personnel for training models to obtain more accurate analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram showing the process of program state tracking of the high performance computing program monitoring interface based on OTF;

[0043] Figure 2 An example of a program data status acquisition module for symbol renaming during build time is shown; a schematic diagram of a program status acquisition module based on link time symbol redirection;

[0044] Figure 3 A schematic diagram showing an example of a runtime library injection program data status acquisition module;

[0045] Figure 4 Example Glibc Hook algorithm diagram showing runtime library injection into program data state acquisition module;

[0046] Figure 5 Displays the monitoring process diagram of the system call monitoring program data status collection module using Ptrace;

[0047] Figure 6 A schematic diagram showing the collection results of the resource consumption measurement collection module for a certain MapReduce program network traffic change when a high-performance computing program is running;

[0048] Figure 7 A schematic diagram showing the CPU and memory usage collected by the resource consumption measurement collection module when a high-performance computing program is running;

[0049] Figure 8 A schematic diagram showing a two-level buffer queue for high performance computing program state data transfer and storage modules;

[0050] Fig. 9 Schematic diagram showing the ROC curve of the optimal model of the LightGBM-based classification module of the machine learning-based fault analysis and prediction system;

[0051] Fig.10 A schematic diagram showing the change of silhouette distance score of a machine learning-based fault analysis and prediction system clustering computing task data in a certain oil field using the KMeans clustering module based on time series data;

[0052] Fig.11 Schematic diagram showing the clustering results of clustering the resource consumption of computing task nodes in a certain oil field using the KMeans clustering module based on time series data in the machine learning-based fault analysis and prediction system. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] The present invention provides a large-scale computing cluster fault tracing system for production environment, comprising: an OTF-based high-performance computing program monitoring interface, a production environment-oriented high-performance computing application monitoring system, and a machine learning-based fault analysis and prediction system, wherein the OTF-based high-performance computing program monitoring interface is used to describe the program state based on the OTF technology, the production environment-oriented high-performance computing application monitoring system is used to monitor the program state and generate monitoring data, and the machine learning-based fault analysis and prediction system analyzes and predicts faults based on the monitoring data. The OTF-based high-performance computing program monitoring interface is used to implement the production environment-oriented high-performance computing application monitoring system, and the machine learning-based fault analysis and prediction system is implemented through the monitoring data of the production environment-oriented high-performance computing application monitoring system.

[0055] In order to perform fault analysis on a program, we first need to obtain the internal information of the program when it is running, including the current running stage of the program, the network communication status between nodes, the error information when the program makes system calls, and the runtime log (collectively referred to as the program status). Based on the existing high-performance program status collection method, combined with the current mainstream distributed microservice link tracking idea of ​​mobile Internet, we design an OTF-based high-performance computing program monitoring interface that can be used to describe the program status based on OTF technology. The OTF-based high-performance computing program monitoring interface appropriately simplifies and expands OTF to meet the actual needs of high-performance computing, and is simple to use and easy to understand.

[0056] In some embodiments, the program state includes system resources consumed when the program is running and internal information of the program. All interfaces described herein are used to describe the program state.

[0057] The OTF-based high-performance computing program monitoring interface of the present invention is configured with a task start interface, a task end interface, and a report progress interface, and the three interfaces correspond to Table 2 below. The OTF-based high-performance computing program monitoring interface of the present invention is configured as follows: when the master node assigns a computing task to a child node, the task start interface is called, and then the child node can call the report progress interface to report the current computing progress. When the calculation is completed, the master node calls the task end interface to report that the computing task has ended. If it is necessary for the child node to obtain the results of other child nodes during the calculation process, the task start interface and the task end interface are called for nested calls. In one example, the program status tracking process is as follows: Figure 1 As shown in the figure, when the master node Rank0 assigns the computing task to the child nodes Rank1, Rank2, and Rank3, it first calls the task start interface, and then the child nodes Rank1, Rank2, and Rank3 can each call the report progress interface to report the current computing progress. When the computing is completed, the master node Rank0 calls the task end interface to report that the computing task has ended. If the child node Rank1, Rank2, or Rank3 needs to obtain the results of other child nodes during the computing process, the task start interface and the task end interface are called for nested calls.

[0058] In addition, since the communication between nodes in the high-performance computing framework is often based on the transport layer's self-designed protocol, rather than using the HTTP (Hyper Text Transfer Protocol) protocol for communication like microservice calls, it will become very complicated to design the accompanying element function for in-band transmission for the high-performance computing framework. In addition, each node in the high-performance cluster itself has a very powerful data transmission capability, and does not require the accompanying element function to realize data transmission between nodes. Therefore, the accompanying element function is also cancelled in the OTF-based high-performance computing program monitoring interface of the present invention to simplify the design and implementation.

[0059] Similar to OTF, the high-performance computing program monitoring interface based on OTF of the present invention also has a Tracer class as the starting point of program monitoring. In some examples, in the high-performance computing program, the Tracer class will exist as a singleton mode, and there is only a unique instance in the program. The Tracer interface is implemented by the Tracer class, and the Tracer class contains Task subclasses and IO subclasses. When using the interface, the function of the instance corresponding to the class corresponding to the interface is directly called. In the high-performance computing program monitoring interface based on OTF of the present invention, the functions of the Tracer interface are shown in Table 1. When using the Tracer interface, the function of the instance corresponding to the Tracer class corresponding to the Tracer interface is directly called, such as Init, CreateTask, etc.

[0060] Table 1 Tracer interface

[0061]

[0062]

[0063] The method name is the function that needs to be called when using the interface. In the high-performance computing program monitoring interface based on OTF of the present invention, in order to meet the actual needs of high-performance computing clusters and programs, combined with the actual situation of high-performance computing (HPC) clusters, Span is refined into two subclasses, Task and IO, corresponding to running a sub-Task and starting an IO operation, respectively, where Span is an abstraction of the subprogram operation in the original OTF interface. In some examples, due to the characteristics of distributed tasks being more "flat" rather than "deep" compared to microservice calls, the high-performance computing program monitoring interface based on OTF of the present invention no longer contains complex Task (referring to the return value Task of a Task created by the CreateTask function in Table 1), but only describes the relationship between tasks by the "initiating node" and "assigned node" of the task, that is, there is only a parent-child relationship between tasks, and the parent task is used to initiate the subtask and wait for the result of the subtask. In actual use, the Tracer interface is the only instance that exists in the program. When the CreateTask function in the Tracer interface is called, the instance automatically becomes the parent task, and the Task returned is the subtask.

[0064] In the case of Task subclasses, since the initiating node and the assigned node of the task on the cluster are often not the same node, and the high-performance computing program monitoring interface based on OTF of the present invention simplifies the in-band transmission function in OTF. In order to obtain the return value Task of the CreateTask function on the child node, the high-performance computing program monitoring interface based on OTF of the present invention is configured to allow the creation of the same Task on multiple nodes. When the RankId parameter value of the CreateTask function is the same as the Rank parameter value of the instance corresponding to the program where the Tracer interface is located, the node corresponding to the Rank parameter value is considered to be a child node. The operation of this CreateTask is only to obtain the return value Task of the CreateTask function, and there is no real operation to create a task. In this case, the CreateTask function only returns the return value Task of the CreateTask function, and the information of this created task is not recorded in the database. However, it should be noted that the Id field in the parameter should be consistent when the CreateTask function is called on the parent node and the child node respectively, otherwise it will cause errors in subsequent analysis.

[0065] In the case of the IO subclass, when the node performs an IO operation, the CreateIO function should be called for collection. The IO operations that can be collected include opening and closing files and Sockets, reading and writing, etc., and their specific types should be specified in the Type parameter of the CreateIO function. When the IO operation type is specified as opening a file, the Target parameter should be the path of the opened file; when the IO operation type is specified as opening a Socket and performing network communication, the Target parameter should be the address and port number of the communication partner. Since high-performance computing programs require a large number of read and write disk files during operation, it is very expensive to collect all disk operations. In the OTF-based high-performance computing program monitoring interface of the present invention, this error information should be collected only when an IO operation fails, and it should not be recorded if the IO operation ends successfully.

[0066] For network operation, users can expand the OTF-based high-performance computing program monitoring interface of the present invention by themselves, and collect network traffic, data packet headers or network communication content according to actual conditions. In the OTF-based high-performance computing program monitoring interface of the present invention, the Tracer interface includes a Task interface and an IO interface, and the Task interface and the IO interface are used to monitor computing tasks and data IO, respectively, and the Task interface and the IO interface are shown in Table 2.

[0067] Table 2 Task interface and IO interface

[0068]

[0069] The OTF-based high-performance computing program monitoring interface of the present invention includes the following features: it is designed based on the existing OTF specification and is easy to understand and use; it provides four different monitoring modes, namely log, measurement, task and IO, so as to facilitate the collection of information from all aspects of program and cluster runtime; it can distinguish whether the current operation is wrong, and only collects data when an error occurs, thereby saving space.

[0070] The mainstream high-performance computing application analysis and debugging tools on the market are all oriented to the development environment. Although they can capture a large number of application runtime details, the additional runtime overhead brought by their monitoring function is large, and the application running speed will be significantly slowed down when the monitoring function is turned on. In addition, overly detailed monitoring will output a large amount of data, requiring a large amount of disk space, and can only be used in the development environment, and is difficult to apply in the production environment. In order to solve this problem, the high-performance computing application monitoring system for the production environment provided by the present invention is a lightweight distributed high-performance computing application fault monitoring and analysis system.

[0071] The high-performance computing application monitoring system for production environment of the present invention is an algorithm that efficiently implements the high-performance computing program monitoring interface based on OTF on a high-performance production cluster, and is configured to collect and monitor the runtime status of the program in real time in the high-performance computing cluster of the production environment, including both the system resources consumed when the program is running and the internal information of the program. Monitoring the high-performance computing application in the production environment can enable development and operation and maintenance personnel to understand the program running status in a timely manner, thereby discovering potential faults or performance weaknesses.

[0072] The high-performance computing application monitoring system for production environment of the present invention starts from the system call level, and uses the wrap function provided by the LD program in Linux binutils to automatically wrap the system calls related to network communication using the function proxy design pattern. Therefore, the high-performance computing application monitoring system for production environment of the present invention is configured as follows: when the monitored distributed high-performance program has inter-node communication, the communication process between nodes can be automatically captured, so as to establish a call relationship flow chart between nodes of the distributed program, and analyze the parallel program execution process from a more advanced and comprehensive perspective, so as to dig out the problem; use multi-threaded asynchronous operation to speed up the recording process, and when a network communication event occurs in the application, only this network communication event is inserted into the queue, and another monitor thread regularly takes out this network communication event from the queue and stores it in the distributed database, so as to reduce the runtime burden of the application itself to the greatest extent; capture the header part of the data packet transmitted between nodes, and automatically unpack it according to the preset rules, and analyze the dependency and call relationship between nodes. Compared with traditional distributed computing application analysis and debugging tools, this method requires fewer resources, has little impact on program runtime overhead, and can be applied to production environments. Because it monitors applications by automatically packaging system calls, it avoids the need for developers to manually modify the code to add instrumentation code, thus greatly improving usage efficiency.

[0073] In order to solve the key problem that it is difficult to modify the source code when tracking the status of high-performance programs in a production environment, the OTF-based high-performance computing program monitoring interface of the present invention is conveniently implemented. The high-performance computing application monitoring system for production environments of the present invention is configured with an application-level acquisition module, a system-level acquisition module, and a data transmission and storage module, wherein the application-level acquisition module is configured to collect information related to high-performance computing programs and services during the operation of high-performance computing programs (such as file and network IO information), the current stage of the high-performance computing program, and task progress, etc., and the system-level acquisition module is configured to collect information related to the cluster and operating system, such as CPU and memory occupancy, network transmission rate, etc. The data transmission and storage module is configured to transmit and store data collected by the application-level acquisition module and the system-level acquisition module, and finally save the data in a database as monitoring data.

[0074] The application level acquisition module and the system level acquisition module are in parallel relationship, respectively collecting the status of the high-performance computing cluster and the program at runtime from different angles, and after the application level acquisition module and the system level acquisition module collect the data, the data transmission and storage module are used to transmit and store the data, and finally the data is saved in the database. The application level acquisition module includes a symbol renaming program data status acquisition module at construction time, a runtime library injection program data status acquisition module and a system call monitoring program data status acquisition module. The application level acquisition module can collect as much program and cluster runtime information as possible from multiple different levels under the premise of rarely modifying the source code, and at the same time achieve the lowest possible runtime performance loss, thereby meeting the actual needs of the production environment. The large-scale computing cluster fault tracking system for the production environment of the present invention selects one or more of the symbol renaming program data status acquisition module at construction time, the runtime library injection program data status acquisition module and the system call monitoring program data status acquisition module for different situations. The system level acquisition module includes a high-performance computing program runtime resource consumption measurement acquisition module. The data transmission and storage module includes a high-performance computing program state data transmission and storage module. Therefore, in the high-performance computing application monitoring system for production environments of the present invention, the build-time symbol renaming program data status collection module, the runtime library injection program data status collection module and the system call monitoring program data status collection module are adopted according to different situations, and the high-performance computing program runtime resource consumption measurement collection module is adopted in parallel. The data collected by the build-time symbol renaming program data status collection module, the runtime library injection program data status collection module and / or the system call monitoring program data status collection module, as well as the data collected by the high-performance computing program runtime resource consumption measurement collection module, are all transmitted and stored through the high-performance computing program status data transmission and storage module, and the data are saved in a database.

[0075] The comparison of the data state acquisition module of the build-time symbol renaming program, the data state acquisition module of the runtime library injection program and the data state acquisition module of the system call monitoring program of the present invention is shown in Table 3 below.

[0076] Table 3 Comparison of three program data status acquisition modules

[0077]

[0078] In program development, having source code means having the richest and most complete information, and modifying the source code means being able to implement any function. However, it is not realistic to make a large number of modifications to the existing source code in a production environment. The symbol renaming program data state acquisition module during construction of the present invention is based on dynamically inserting specific functions of the program during construction, thereby collecting the program state. The symbol renaming program data state acquisition module during construction is configured to rename the symbols of its ELF (Executable and Linkable Format) file when the program is built, and rename the original call to the specific function to other functions, thereby realizing automatic insertion of the program, thereby performing program state acquisition. The symbol renaming program data state acquisition module during construction can realize automatic insertion of the program without modifying the source code, which greatly reduces the burden on developers.

[0079] The construction process of C / C++ program is divided into two parts: compilation and linking. Compilation refers to translating the source code in text format into a binary file in ELF format, while linking refers to combining multiple ELF files and library files together to form a final executable file. Symbol lookup operation is required during linking, that is, finding the address of functions and global variables and filling them into the final machine code instructions. An example of the program data status acquisition module for symbol renaming during construction of the present invention is a program status acquisition module based on link-time symbol redirection. With the support of linker-related functions, this process can be manually intervened, that is, when a certain section of code attempts to call the read function in other files, the linker may not fill the address of read into the function call instruction, but fill in the address of the pre-agreed __wrap_read function. After properly recording the specific information required for this call in __wrap_read, the real read function is called through the __real_read symbol to complete the corresponding function. Figure 2 As shown, when the user program is started after being processed by the program status acquisition module of the symbol redirection at link time, under the effect of the -Wl,wrap=read parameter, each time the user program accesses a specific function, it will be redirected to a specific wrapper function, such as read will be redirected to __wrap_read, and the start information of the call, such as the time when the call occurred, the parameters used, etc., will be recorded in the wrapper function, that is, the start of the call will be recorded. The wrapper function then calls the real library function (such as __real_read) to complete the corresponding function. When the real library function returns, the end time, whether the call is successful, and other information are recorded, that is, the end of the call is recorded. This method can realize the instrumentation of specific functions without modifying the existing source code. By carefully controlling the number of instrumented functions, the runtime burden of instrumentation collection on the program can be greatly reduced.

[0080] Although failures in clusters and programs often involve multiple different aspects such as disks, networks, memory, and CPUs, when a failure occurs, its phenomenon is often first manifested in IO, such as slow IO speeds, increased failure rates, etc. Therefore, in order to comprehensively and efficiently capture the internal state of high-performance computing programs and quickly discover failures, the build-time symbol renaming program data state acquisition module of the present invention can be used to perform instrumentation acquisition on IO-related functions. The selected IO-related functions include ten IO functions: write, read, connect, accept, open, close, recv, send, readv, and writev, which are sufficient to cover most situations of disk IO and network acquisition.

[0081] The large-scale computing cluster fault tracking system for production environment of the present invention utilizes the symbol renaming program data status acquisition module during construction to collect functions related to network communication and disk IO, automatically records the communication time, size, content and information of both parties when the program has a network communication process, and can analyze the running status of the high-performance computing program in combination with the subsequent network traffic analysis method; when the program has a disk IO operation and the operation fails, the large-scale computing cluster fault tracking system for production environment of the present invention can automatically record the time of failure, error code and related file path and other necessary information, which is convenient for subsequent fault analysis. Since in the Linux system, network communication and file operation share a unified API, the mechanism of automatically registering the file descriptor when opening the file is used to solve the problem that the call of the collection function itself to the Linux API is mistakenly regarded as part of the HPC communication process, and repeated collection causes an infinite recursive dead loop.

[0082] The method of inserting piles during construction can solve most of the problems, but has a major disadvantage, that is, inserting piles can only be performed when the program is built through source code. However, the operation of the program depends on the runtime library environment, such as Linux runtime libraries such as Glibc and Musl. These third-party libraries have many functions encapsulated in a variety of different ways for the convenience of users. For example, the output function has different interfaces such as printf, fprintf, puts, std::cout, etc. These interfaces encapsulate the system call to read into a precompiled library, and the user program only needs a simple link library when building, without the need to compile from the source code of the library. Since the user program is independent of the code in the library when it is built, the symbol renaming program data state acquisition module during the construction of the present invention cannot be applied in the library function.

[0083] The runtime library injection program data status collection module of the present invention is configured to automatically inject the third-party library that the program depends on, add a program status collection function to the third-party library, and can make up for the defect that the symbol renaming program data status collection module cannot collect data due to the lack of source code of the third-party library during construction.

[0084] The large-scale computing cluster fault tracking system for production environments of the present invention uses a hook method for Glibc to achieve the function of unified tracking of all output information. Through research, it can be known that in Glibc, stdout is a pointer to a FILE structure, and the FILE structure is the first member of _IO_FILE_plus. In addition to the FILE structure, _IO_FILE_plus also has a vtable member variable, which contains various IO-related underlying function pointers. The runtime library injection program data status acquisition module of the present invention is configured to replace vtable->__write with a function with a data acquisition function when the GPP (GeoEast Parallel Programming) program is started, thereby realizing seamless recording of all log outputs of the program without modifying the code. An example of the runtime library injection program data status acquisition module of the present invention is as follows Figure 3 As shown. In the operating system, the memory area where the library function is located has only access permissions by default, and the virtual function table of the library space cannot be modified. Therefore, it is necessary to modify the virtual function table area permissions through the mprotect system call before replacing the virtual function table content. The complete Glibc Hook algorithm is shown in Figure 4 shown.

[0085] Most computing programs fail when accessing system functions, such as errors when performing disk or network operations. In modern operating systems, accessing system hardware functions requires system calls. If the results of each system call can be monitored, a large number of faults can be found. The system call monitoring program data status acquisition module of the present invention realizes the tracking of system calls through the Ptrace function, and discovers the failure of system calls by monitoring the results of each system call. Ptrace is a function provided in the Linux kernel, which can make the program pause when a specific signal or when entering a system call, and transfer control to another monitoring program. The monitoring program can record the state of the controlled program at this moment, and can even modify the memory content of the controlled program, and then return control to the controlled program to continue execution. Ptrace is often used to develop various debugging functions and is the basis of debuggers such as GDB. Based on the feature that Ptrace can pause when the controlled program enters a system call, it can be used to collect parallel program behavior. When using Ptrace for monitoring, you should first use fork (copy) to call to create a child process as a monitoring process, and use the parent process as the controlled process. Then call PTRACE_ATTACH in the child process to attach itself as a debugger to the parent process. After this operation, the parent process will be suspended by SIGSTOP whenever it calls a system call or returns from a system call, and the child process can use the waitpid system call to be awakened when the parent process is suspended. When the child process is awakened, it can obtain the parent process register and memory content, thereby inferring the system call number and related parameters currently being performed by the parent process, and save this information to the database. Then use PTRACE_CONT or PTRACE_SYSCALL to notify the parent process that it can continue to execute. The monitoring process using Ptrace is as follows: Figure 5 As shown in the figure. Tracing through Ptrace can also capture program exits caused by system signals, such as SIGSEGV, to facilitate locating program failures.

[0086] In the fault analysis process of the program, in addition to the status data of the program itself, which is of great significance, the resource situation consumed when the program is running is also worthy of attention. The resource situation consumed at every moment when the program is running can reflect the computing characteristics and running status of the program to a great extent. The resource consumption measurement collection module of the high-performance computing program runtime of the present invention can better capture the resource situation information consumed when the program is running, and perform a more comprehensive performance analysis on the computing program. The resource consumption measurement collection module of the high-performance computing program runtime is configured to use various tools to monitor the running of the program in real time during the program running process, collect the resources (such as CPU, memory, network, disk, etc.) occupied by the program at every moment during the running process, and form the running characteristic information of the program through induction and collation. The running characteristic information can be used to monitor whether the program is running normally, analyze the computing characteristic type of the program, and discover the performance weaknesses or potential faults of the program. In the design of high-performance computing programs, it is particularly necessary to use the above information to optimize the program in a targeted manner, thereby shortening the running time of the computing program and making full use of computer resources.

[0087] An example of the resource consumption measurement collection module of the high-performance computing program runtime of the present invention is the system resource consumption measurement collection module of the program runtime under the Linux system. In the Linux system, due to the existence of the proc virtual file system, each running process will be mapped to a subdirectory in proc, and its directory name is the pid (Process Identifier) ​​of the process. In / proc / <pid>The / statm file stores the memory usage of the current process, while the / proc / <pid>The / stat file provides statistics of the current process. Read / proc / <pid> / statm and / proc / <pid> / stat file to obtain the CPU and memory usage information of the current program. In order to meet the requirements of minimizing the performance loss during data collection of high-performance computing programs and minimizing the impact on the normal execution of the original computing programs, the resource consumption measurement collection module of the high-performance computing program runtime of the present invention is configured to design an independent thread for periodically reading / proc / <pid> / statm and / proc / <pid> / stat file, and save the parsed information into the database, thereby realizing the resource consumption measurement collection work when the high-performance computing program is running. Figure 6 This is the result of collecting the changes in network traffic of a MapReduce program. The vertical axis is the network IO rate of the node, in MB / s. The program running process can be clearly divided into two stages: Map and Reduce. In the Map stage, the network traffic occupies less and is mainly used for calculation. When it comes to the Reduce stage, the traffic suddenly increases and fluctuates significantly. This information has certain guiding significance for the subsequent tuning of the program.

[0088] The resource consumption measurement collection module of the high-performance computing program of the present invention collects CPU occupancy during operation. In the Linux system, by periodically reading / proc / <pid> / stat file to obtain statistical information of processes running in the system, / proc / <pid>The format of the / stat file is multiple fields separated by spaces, each of which represents a different meaning. Columns 14-17 respectively show the time the process was executed in user mode (utime), the time it was executed in kernel mode (stime), the time it spent waiting for child processes in user mode (cutime), and the time it spent waiting for child processes in kernel mode (cstime). By reading / proc / <pid>The above four fields in the / stat file can obtain the CPU resources consumed by the process.

[0089] It is worth noting that the above four indicators collected from the large-scale computing cluster fault tracking system for production environments are all in "clock ticks", which are the counts of clock interrupts received by the kernel, not the real wall clock time. The real time consumed by the process can be obtained by dividing the clock tick by sysconf(_SC_CLK_TCK) (the number of ticks per second, recorded as T). In addition, this consumed time value is the time running on a CPU. If the computer has M CPUs, each CPU has N cores, and it is assumed that the utime obtained from two sampling values ​​with an interval of t (unit: seconds) are u1 and u2 respectively, then the CPU occupancy calculation formula of the process is shown as follows.

[0090]

[0091] The resource consumption measurement collection module of the high-performance computing program of the present invention collects the memory usage. In the Linux system, / proc / <pid>The / statm file stores the memory statistics of the current process. <pid> / statm file format and / proc / <pid> / proc / is the same as / stat, but the data volume is smaller, with only 7 fields, namely the total size of memory occupied by the process, the size of the memory resident set, the number of shared pages of the resident set, the code segment, the library size, the data stack size, and the number of dirty pages. The resource consumption measurement collection module of the high-performance computing program runtime of the present invention mainly focuses on the total memory size requested by the computing process when collecting the memory occupancy, that is, / proc / <pid>The sixth field in the / statm file is the stack size. Unlike CPU usage, / proc / <pid>The data unit obtained in / statm is the number of bytes of memory, so no further special processing is required and it can be directly saved to the database. The resource consumption measurement collection module of the high-performance computing program runtime of the present invention collects the CPU and memory occupancy conditions. Figure 7 shown.

[0092] Since the high-performance computing cluster in the production environment has high requirements for operating efficiency, the cluster network load is large. When the program in the existing high-performance computing cluster is running, each node frequently stores a large amount of data to the database, which will greatly increase the burden on the cluster network, which is not conducive to the execution of high-performance computing programs and cannot meet the needs of the production environment. In addition, the programs in the existing high-performance computing cluster all use custom data formats, and the data lacks structured information or the structured format is not unified, which hinders data analysts from understanding and using data, and is not conducive to subsequent fault analysis.

[0093] The high-performance computing program state data transmission and storage module of the present invention includes a data collection specification, which can unify the information sent by each node to the database, and retrieve and analyze the information more efficiently. The data collection specification is a program state data transmission algorithm, and each node should follow the data collection specification when sending collected data to the database. The data collection specification includes flow collection, IO error collection, task status collection, log collection and node status collection, which can cover the various collection contents mentioned. According to the basic functional division, it can be divided into two parts: flow and IO error collection and task status, node status and log collection. The flow and IO error collection specifications are shown in Table 4, and the task status, node status and log collection specifications are shown in Table 5.

[0094] Table 4 Traffic and IO error collection specifications

[0095]

[0096] Table 5 Task status, node status and log collection specifications

[0097]

[0098] Since there are many nodes when the distributed parallel computing program is running, a large amount of data can be generated every second. According to research, if each computing node directly sends this data to the database, it will cause great pressure on the network. Therefore, in order to solve the problem of tight cluster resources and large computing tasks, it is necessary to reduce the extra burden brought by the acquisition module as much as possible. The high-performance computing program state data transmission and storage module of the present invention is configured with a two-level buffer queue to reduce cluster pressure, such as Figure 8 As shown in the figure, the first-level queue is a process-level buffer queue. Each computing thread only needs to put the information to be recorded directly into the queue, and the log processing thread is responsible for sending it to the second-level queue to prevent information collection from blocking computing tasks. The second-level queue is a host-level queue, and the second-level queue is from "log processing thread" to "log processing process". The log processing thread of each process sends information to the host-specific log processing process through the unixdomain socket, and the log processing process then packages and compresses the data and stores it in the database to prevent the host from creating excessive TCP (Transmission Control Protocol) connections that impact the cluster network.

[0099] In addition, data transmission through Unix domain socket actually utilizes the function provided by Linux kernel for IPC (Inter-Process Communication). Although the interface is in the form of socket, the actual data does not pass through the network card. Therefore, for the application, it can achieve extremely high bandwidth and extremely low latency, which objectively further reduces the performance loss of the acquisition module on the application.

[0100] In the high-performance computing application monitoring system for production environments of the present invention, flow collection, IO error collection, task status collection, log collection and node status collection are the general aspects of collection, and the relevant collection details have been described above. For example, IO error collection can be performed through IO related functions, log collection can be performed through standard output and standard error streams, and flow collection can be performed through functions related to network communication.

[0101] The high-performance computing application monitoring system for production environments of the present invention has the following characteristics: automatic monitoring at the system call level without manual intervention by the user; low-intrusive and low-overhead monitoring methods are designed at three levels: symbol renaming at build time, runtime library injection, and system call monitoring; two-level buffer queue operations are used to accelerate the data recording process to avoid excessive runtime overhead caused by the monitoring program (referring to the example corresponding to the high-performance computing application monitoring system for production environments); CPU and memory usage are collected as resource metrics, which can reflect the system runtime situation and the computing characteristics of the program to a certain extent; the collection results are stored in a distributed database instead of in ordinary files; and a rich collection API is provided, so that developers can actively report more information to the system according to their own needs.

[0102] Since the large-scale computing cluster fault tracking system for production environment is oriented to the production environment, in actual production, the monitoring system will capture a large amount of data every second, including both the status information of the cluster itself and the log information output by the high-performance computing program running on the cluster. It is not realistic for the operation and maintenance personnel to manually analyze such a large amount of data. The machine learning-based fault analysis and prediction system of the present invention is configured to use machine learning related technologies to automatically classify and predict the data collected by the high-performance computing application monitoring system for production environment, so as to accurately evaluate the current state of the cluster and discover possible faults in the cluster in advance. That is to say, the machine learning-based fault analysis and prediction system of the present invention also provides automatic fault detection and certain fault prediction functions.

[0103] In some examples, the machine learning-based fault analysis and prediction system includes a classification module based on LightGBM, which continuously monitors the runtime status of a single node based on the LightGBM clustering algorithm based on the collected information, so as to promptly discover whether the node has entered an unstable state or has failed. Since there are many hyperparameters that need to be pre-specified in the LightGBM algorithm, the classification module based on LightGBM is configured to use the grid search algorithm to find the optimal parameters when the standardized data is classified using the LightGBM algorithm, and the evaluation indicators are the AUC curve and F1 value of the model on the test set. The grid search algorithm selects five hyperparameters, namely the number of leaves, the learning rate, the number of base learners, the maximum depth of the decision tree, and the data sampling, as the hyperparameters to be adjusted. The grid search algorithm obtains different combinations by calculating the Cartesian product of the above five hyperparameters, and trains the LightGBM model using each hyperparameter combination. The value ranges of the five hyperparameters are shown in Table 6.

[0104] Table 6 Hyperparameters in grid search

[0105]

[0106] The LightGBM-based classification module divides the Antarex dataset into training set, validation set, and test set according to 6:1:3 during training. For each model obtained in the grid search, the training set is used to train its parameters, and the AUC curve score and F1 value of each model are evaluated on the validation set. The AUC curve score and F1 value of each model are cross-compared, and the model with the highest index value is taken as the optimal model as the final training result, where the index value refers to the average of the accuracy, AUC curve score, and F1 value of the model on the validation set.

[0107] In one example, 30% of the data in the training set, a total of 311,043 test samples, were randomly selected as the test set. The optimal model obtained by the grid search algorithm was selected for effectiveness verification. The accuracy, AUC curve score and F1 value of the model on the training set and test set were calculated respectively. The final model accuracy on the test set was 0.975, the AUC curve score was 0.976, and the F1 value was 0.975. The above indicators can show that the model can effectively point out the possible fault type based on the collected indicators, and there is no overfitting phenomenon. In the process of machine learning, the indicators on the training set are not very meaningful for the performance evaluation of the model, so they are omitted here. The ROC curve of the optimal model on the training set and the test set is shown in Figure 2. Fig. 9 As shown in the figure. The ROC curve (receiver operating characteristic curve) is also called the sensitivity curve. Each point on the curve reflects the same sensitivity. They are all responses to the same signal stimulus, but the results are obtained under several different judgment criteria.

[0108] In terms of model training of the classification module based on LightGBM, due to many limitations in the production environment, it is difficult to fully apply the fault analysis and prediction system based on machine learning in the production environment. Therefore, the present invention also uses the Antarex data set to train a classification model based on logistic regression. Since the data is standardized, the size of the coefficient value of each indicator of the logistic regression model can reflect the contribution of the indicator to the prediction result. The top five indicators that best reflect each error are shown in Table 7. In a production environment where it is impossible to deploy a fault analysis and prediction system based on machine learning, the coefficient values ​​of each indicator of the logistic regression model have a certain reference role for operation and maintenance personnel to discover the cause of the fault. When the operation and maintenance personnel find that the collection index of a node in the cluster is abnormally high as shown in Table 7, it means that the node is very likely to be experiencing a fault corresponding to the indicator.

[0109] Table 7 Error types and their related indicators

[0110]

[0111]

[0112] In some examples, the machine learning-based fault analysis and prediction system includes a KMeans clustering module based on time series data. KMeans is an unsupervised clustering algorithm that iteratively solves the distance from each point to the classification center. Its basic principle is to pre-set K categories and randomly specify their cluster centers, then calculate the distance from each point to each classification center, divide the data point into the category with the smallest distance from it, and then calculate a new cluster center in each category, and repeat the above process until the category to which each data point belongs is stable or the error is less than a specific threshold. The KMeans algorithm is simple to implement and easy to calculate, and is widely used in many occasions. Using the KMeans clustering module based on time series data, the runtime conditions of different nodes in the cluster can be cross-compared, trying to find nodes that are different from the others in the cluster, so as to focus on the situation of the node and find possible faults or performance weaknesses. In other words, the KMeans clustering module based on time series data can automatically cluster the resource usage of each node collected from the cluster, thereby trying to find abnormal points.

[0113] When using the KMeans clustering module based on time series data, you first need to define the distance between data points. In some embodiments, the data point type is a time series curve of resource usage. For convenience of calculation, the data point distance is defined as the Euler distance between curves, as shown in the following formula, where x is the current moment in seconds, y(x) is a function of how resource usage changes over time, which represents the percentage of resource utilization in percentage, and d is the Euler distance between curves in percentage.

[0114]

[0115] In addition, the KMeans clustering module based on time series data also requires that the number of categories be specified before calculation. For fault analysis of high-performance computing clusters, each node may exhibit different resource utilization characteristics when performing a single task, and it is also possible that more than one node may fail at a certain moment. Therefore, it is difficult to determine the number of categories in advance before calculation. Therefore, the KMeans clustering module based on time series data of the present invention tries a variety of different numbers of categories while using KMeans classification, calculates the silhouette score of each clustering to judge the classification effect, and selects the number of categories with the highest silhouette score as the final clustering result. In one example, after clustering a certain petroleum field computing task data using the KMeans clustering module based on time series data, the changes in the silhouette distance score are as follows Fig.10 As shown, it can be seen that when the number of categories is 8, the silhouette score is the highest, which is 0.6616, so the number of categories is selected as 8.

[0116] In one example, the KMeans clustering module based on time series data is used to cluster the resource consumption of computing task nodes in a certain oil field. The clustering results are as follows: Fig.11 As shown in the figure, the ordinate is the CPU occupancy, that is, the sum of the occupancy rates of each CPU core. It can be seen that the KMeans clustering module based on time series data can better cluster curves with different characteristics into the same category, and extract nodes that obviously have different computing characteristics from other nodes separately, which can be used to remind operation and maintenance personnel to focus on the status of these nodes.

[0117] The machine learning-based fault analysis and prediction system of the present invention has the following characteristics: the LightGBM-based classification module automatically classifies the fault data collected from the cluster, and can quickly identify whether a node has a fault and what kind of fault exists; the KMeans clustering module based on time series data clusters each node in the cluster, so as to quickly discover unique nodes and remind relevant personnel to check the node status in time.

[0118] The OTF-based high-performance computing program monitoring interface not only defines the content of the data collected in the cluster, but also standardizes the format of the data collected in the cluster. When each node in the cluster collects data, it transmits and stores the data in the same format through this interface, which is convenient for subsequent data processing and analysis. In addition, defining a standardized data collection and transmission interface is also of great positive significance for the subsequent expansion of the collection content.

[0119] In the high-performance computing application monitoring system for production environments, all acquisition modules collect data according to the contents contained in the OTF-based high-performance computing program monitoring interface, and transmit and save the collected data to the database to form the monitoring data of the high-performance computing application monitoring system for production environments. Subsequently, the machine learning-based fault analysis and prediction system retrieves the data from this database and applies the machine learning-based fault analysis and prediction algorithm to it to realize large-scale computing cluster fault tracking for production environments.

[0120] This high-performance computing application monitoring system for production environments is aimed at large-scale HPC production clusters. It solves the problems faced in implementing data collection on large-scale clusters through various methods, and collects data from both the application level and the system level, organically combining the collected data to achieve more comprehensive and accurate large-scale computing cluster fault tracking for production environments.

[0121] A specific implementation of the large-scale computing cluster fault tracing system for production environments of the present invention is a specific fault tracing process, which is mainly divided into three aspects: cluster fault tracing, application fault tracing and fault prediction. The specific steps of the three aspects are as follows:

[0122] Application fault tracing process:

[0123] (Step 1) The developer imports the OTF-based high-performance computing program monitoring interface C source code file provided by the large-scale computing cluster fault tracking system for production environments into the project and compiles it together, and uses the four parameters -wrap=read-wrap=connect-wrap=write-wrap=accept to link it.

[0124] (Step 2) The OTF-based high-performance computing program monitoring interface will package the read system call, write system call, connect system call, and accept system call. When the user calls the read system call, write system call, connect system call, and accept system call, the large-scale computing cluster fault tracking system for the production environment will record it.

[0125] (Step 3) When the user calls the connect system call and the accept system call, the large-scale computing cluster fault tracking system for production environments will record the TCP link address associated with the file descriptor generated when the connect system call and the accept system call are used to create a TCP network connection.

[0126] (Step 4) When the user calls the read system call and the write system call, the large-scale computing cluster fault tracking system for production environments will find the TCP link address associated with the file descriptor according to the file descriptor, and insert the number of bytes read and written to the TCP link address into the write queue.

[0127] (Step 5) The queue monitoring process is awakened when it finds that an element is inserted into the write queue, pops the top element of the queue and writes its information into the public database.

[0128] Application fault tracing process based on runtime library:

[0129] (Step I) The large-scale computing cluster fault tracking system for production environments reverses the address of the _IO_FILE_plus structure in the library through the stdout global variable. The second member in the _IO_FILE_plus structure is the _IO_jump_t callback function jump table.

[0130] (Step II) The 15 members in the _IO_jump_t callback function jump table are the real write function addresses. Record the write function addresses.

[0131] (Step III) Prepare a wrapper function to replace the real write function. In the wrapper function, first determine if the output target address is stdout or stderr, then record the output content into the database, and then call the real write function to complete the original output function.

[0132] (Step IV) Use mprotect to make the memory page where the _IO_jump_t callback function jump table is located writable.

[0133] (Step V) Set the value of the 15th member in the _IO_jump_t callback function jump table to the address of the wrapped write function.

[0134] (Step VI) Use mprotect to set the memory page where the _IO_jump_t callback function jump table is located to read-only.

[0135] Application fault tracing process based on system calls:

[0136] (Step i) A large-scale computing cluster fault tracing system for production environments uses fork to create a child process, and uses PTRACE_ATTACH in the child process to attach to the parent process to monitor the behavior of the parent process.

[0137] (Step ii) The child process uses the waitpid system call to wait for the parent process to enter the system call.

[0138] (Step iii) When the parent process calls the system call, the Linux operating system automatically uses SIGSTOP to suspend the parent process and wake up the child process.

[0139] (Step iv) The child process can obtain the current state of the parent process through the return value of the waitpid system call and the Ptrace related system call, including the system call number, register values ​​and memory data, and record these data in the database.

[0140] (Step v) After the child process has properly recorded the relevant data, it uses PTRACE_CONT to notify the parent process to continue execution, and uses the waitpid system call to continue waiting for the next system call of the parent process.

[0141] Consuming system resources collection process:

[0142] (Step A) A large-scale computing cluster fault tracking system for production environments creates a thread for periodically collecting and recording system resource consumption.

[0143] (Step B) In the acquisition thread, open / proc / <pid> / statm files, / proc / <pid> / stat file, from which the memory usage and the current CPU time can be parsed and read.

[0144] (Step C) Subtract the CPU time obtained from two consecutive acquisitions and divide it by the sampling period to obtain the CPU occupancy rate.

[0145] (Step D) Save the memory usage and CPU usage in the database.

[0146] (Step E) Call sleep to wait for 10 seconds, and then execute the acquisition process again.

[0147] Data transmission and storage process:

[0148] (Step a) When the program starts, it first checks whether the Unix domain socket file monitored by the host-level queue exists and is valid. If the file is invalid, a host-level queue process is started.

[0149] (Step b) Create a buffer queue and a worker thread inside the application to receive data from the application and send it to the host-level queue, wherein the buffer queue is part of the process-level queue.

[0150] (Step c) The application worker thread periodically checks the buffer queue, regularly takes out elements from the buffer queue, and sends them to the host-level queue through the Unix domain socket file.

[0151] (Step d) The host-level queue listens to a specific Unix domain socket file. When receiving data from the process-level queue, the host-level queue takes the data out and temporarily stores it in the memory, waiting for the right time to write the data into the database.

[0152] Fault analysis and prediction system:

[0153] (Step ①) Obtain the operation data of the computing cluster in the production environment for a period of time, such as the fluctuation of CPU usage and memory usage, and calculate the first-order derivative and second-order derivative of these data as the basis for judging their degree of fluctuation.

[0154] (Step ②) Label the type of running data and whether the cluster status is normal during data collection as the original training data.

[0155] (Step ③) Use the original training data to train the LightGBM model and the KMeans clustering module based on time series data to automatically classify and cluster the data.

[0156] (Step ④) Deploy the trained model in the production environment, and continuously use the monitored data as data input to the model. If a fault is found in the data input model, notify the relevant personnel in a timely manner.

[0157] (Step ⑤) If a false alarm occurs, the operating data that causes the data input model to produce an erroneous judgment is used as a negative sample, and step ③ is repeated to further train the model.

[0158] The large-scale computing cluster fault tracing system for production environments of the present invention also includes a unified user interface, which allows users to conveniently overview the current status of the cluster and the running status of applications on the cluster, as well as adjust fault detection and prediction algorithms and other operations.

[0159] The large-scale computing cluster fault tracking system for production environment of the present invention designs a set of OTF-based high-performance computing program monitoring interfaces for link tracking and status collection of high-performance computing clusters based on OTF. The OTF-based high-performance computing program monitoring interface uses a simple and clear API to collect data from various aspects of computing programs in the cluster, and has the characteristics of being easy to understand and easy to use; the distributed high-performance computing programs running on the large-scale computing cluster are automatically plugged in, and users do not need to write additional code. The high-performance computing application monitoring system for production environment can analyze the call and dependency relationship between program nodes, and analyze the status of each node and program in the cluster based on the collected data, so that users can quickly discover the status and faults between nodes; the OTF-based high-performance computing program monitoring interface and the data captured by the high-performance computing application monitoring system for production environment can be used for automatic fault analysis and fault prediction, and when a large-scale cluster system fault is detected or a failure occurs in the application running on the cluster, the relevant operation and maintenance personnel can be notified in time by telephone or email, and the alarm is marked according to the feedback of the operation and maintenance personnel for training models to obtain more accurate analysis results.

[0160] The above embodiments only express the implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.< / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid> < / pid>

Claims

1. A large-scale computing cluster fault tracking system for production environments, characterized in that: include: An OTF-based high-performance computing program monitoring interface, a production-oriented high-performance computing application monitoring system, and a machine learning-based fault analysis and prediction system, wherein the OTF-based high-performance computing program monitoring interface is used to describe program status based on OTF technology, the production-oriented high-performance computing application monitoring system is used to monitor the program status and generate monitoring data, and the machine learning-based fault analysis and prediction system analyzes and predicts faults based on the monitoring data; The high-performance computing application monitoring system for production environment is configured with an application-level acquisition module, a system-level acquisition module, and a data transmission and storage module, wherein the application-level acquisition module is configured to collect information related to the high-performance computing program and the business during the operation of the high-performance computing program, the current stage of the high-performance computing program, and the task progress, the system-level acquisition module is configured to collect information related to the cluster and the operating system, and the data transmission and storage module is configured to transmit and store the data collected by the application-level acquisition module and the system-level acquisition module, and finally save the data in a database; The machine learning-based fault analysis and prediction system is configured to use machine learning related technologies to automatically classify and predict data collected by a high-performance computing application monitoring system for a production environment, thereby accurately evaluating the current state of the cluster and discovering possible faults in the cluster in advance.

2. The large-scale computing cluster fault tracing system for production environment according to claim 1 is characterized in that: The OTF-based high-performance computing program monitoring interface is configured with a task start interface, a task end interface and a progress report interface, wherein the task start interface is called when the master node assigns a computing task to a child node, and then the child node calls the progress report interface to report the current computing progress, and when the computing is completed, the master node calls the task end interface to report that the computing task has ended.

3. The large-scale computing cluster fault tracing system for production environment according to claim 2 is characterized in that: If it is necessary for a child node to obtain the results of other child nodes during the calculation process, the task start interface and the task end interface are called to perform nested calls.

4. The large-scale computing cluster fault tracing system for production environment according to claim 1, characterized in that: The OTF-based high performance computing program monitoring interface has a Tracer class as a starting point for program monitoring.

5. The large-scale computing cluster fault tracing system for production environment according to claim 4, characterized in that: The Tracer class exists as a singleton mode, and there is only one instance in the program. The Tracer interface is implemented by the Tracer class.

6. The large-scale computing cluster fault tracing system for production environment according to claim 5, characterized in that: The Tracer class includes a Task subclass and an IO subclass. The OTF-based high-performance computing program monitoring interface is configured to subdivide Span into Task subclasses and IO subclasses, corresponding to running a sub-Task and starting an IO operation, respectively.

7. The large-scale computing cluster fault tracing system for production environment according to claim 6, characterized in that: In the case of the Task subclass, the OTF-based high performance computing program monitoring interface is configured to allow the same Task to be created on multiple nodes.

8. The large-scale computing cluster fault tracing system for production environment according to claim 6, characterized in that: In the case of the IO subclass, in the OTF-based high-performance computing program monitoring interface, error information of the IO operation error is collected only when an IO operation error occurs.

9. The large-scale computing cluster fault tracing system for production environment according to claim 6, characterized in that: The Tracer interface includes a Task interface and an IO interface, and the Task interface and the IO interface are used to monitor computing tasks and data IO respectively.

10. The large-scale computing cluster fault tracing system for production environment according to claim 1, characterized in that: The program status includes system resources consumed when the program is running and internal information of the program.

11. The large-scale computing cluster fault tracing system for production environment according to claim 10, characterized in that: The application layer acquisition module includes a build-time symbol renaming program data status acquisition module, a runtime library injection program data status acquisition module, and a system call monitoring program data status acquisition module. The build-time symbol renaming program data status acquisition module is configured to rename the symbols of its ELF file when the program is built, rename the original call to a specific function to another function, realize automatic plugging of the application program, and perform the program status acquisition. The runtime library injection program data status acquisition module is configured to automatically inject the third-party library that the program depends on, and add a program status acquisition function to the third-party library. The system call monitoring program data status acquisition module is configured to implement tracking of system calls.

12. The large-scale computing cluster fault tracing system for production environment according to claim 10, characterized in that: The system-level collection module includes a resource consumption measurement collection module when a high-performance computing program is running. The resource consumption measurement collection module when a high-performance computing program is running is configured to use various tools to monitor the program running in real time during the program running process, collect the resources occupied by the program at each moment during the running process, and summarize and organize them to form the running characteristic information of the program. The occupied resources include CPU and memory.

13. The large-scale computing cluster fault tracing system for production environment according to claim 11, characterized in that: The build-time symbol renaming program data status acquisition module is configured to perform instrumentation acquisition on IO related functions, and the IO related functions include the following IO functions: write, read, connect, accept, open, close, recv, send, readv and writev.

14. The large-scale computing cluster fault tracing system for production environment according to claim 11, characterized in that: The runtime library injection program data status acquisition module is configured to replace vtable->__write with a function with data acquisition function when the GPP program is started, thereby realizing seamless recording of all log outputs of the program without modifying the code.

15. The large-scale computing cluster fault tracing system for production environment according to claim 14, characterized in that: The runtime library injection program data status acquisition module is configured to modify the virtual function table area permissions through the mprotect system call and then replace the virtual function table content of the library space.

16. The large-scale computing cluster fault tracing system for production environment according to claim 11, characterized in that: The system call monitoring program data status acquisition module realizes the tracking of system calls through the Ptrace function, and discovers system call failures by monitoring the results of each system call.

17. The large-scale computing cluster fault tracing system for production environment according to claim 12, characterized in that: The resource consumption measurement collection module of the high performance computing program is configured to design an independent thread for periodically reading / proc / <pid> / statm and / proc / <pid> / stat file, and save the parsed information into the database to collect CPU usage and memory usage.< / pid> < / pid> 18. The large-scale computing cluster fault tracing system for production environment according to claim 1, characterized in that: The data transmission and storage module includes a high-performance computing program status data transmission and storage module, and the high-performance computing program status data transmission and storage module includes flow collection specifications, IO error collection specifications, task status collection specifications, log collection specifications and node status collection specifications. Each node should follow the relevant specifications when sending collected data to the database.

19. The large-scale computing cluster fault tracing system for production environment according to claim 18, characterized in that: The high performance computing program state data transmission and storage module is configured with two levels of buffer queues.

20. The large-scale computing cluster fault tracing system for production environment according to claim 1, characterized in that: The machine learning-based fault analysis and prediction system includes a LightGBM-based classification module, which is configured to use a grid search algorithm to find optimal parameters when classifying standardized data using the LightGBM algorithm, and the evaluation indicators are the AUC curve and F1 value of the model on the test set.

21. The large-scale computing cluster fault tracing system for production environment according to claim 20, characterized in that: The grid search algorithm selects the number of leaves, learning rate, number of base learners, maximum depth of decision tree, and data sampling as hyperparameters to be adjusted.

22. The large-scale computing cluster fault tracing system for production environment according to claim 20, characterized in that: The LightGBM-based classification module uses the Antarex dataset to train a logistic regression-based classification model.

23. The large-scale computing cluster fault tracing system for production environment according to claim 1, characterized in that: The machine learning-based fault analysis and prediction system includes a KMeans clustering module based on time series data. The KMeans clustering module based on time series data automatically clusters the resource usage of each node collected from the cluster, thereby attempting to discover abnormal points.

Images