A method for detecting abnormal task stalling in high-performance computing clusters

By comparing the update time and process status of the progress file of computing tasks, stalled tasks can be accurately identified, solving the problem of accuracy in task stall detection in high-performance computing clusters and improving resource utilization and user experience.

CN123114675APending Publication Date: 2026-11-17WUXI HENGDING SUPERCOMPUTING CENT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611352716.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-02
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies lack efficient and accurate methods for detecting task stalls in high-performance computing clusters, leading to wasted computing resources and increased user costs, which in turn affects cluster efficiency and user experience.

Method used

By comparing the last update time of the progress file of the computing task with the current system time, if the time difference exceeds a threshold, the task is marked as a task to be checked. Further queries are made into the process's blocking call information and status changes to determine whether the task has stalled and to notify the user.

Benefits of technology

It improves the accuracy of detecting computational task stalls, reduces the probability of missed detections, helps to release resources in a timely manner, improves cluster utilization efficiency, and enhances user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114675A_ABST
    Figure CN123114675A_ABST
Patent Text Reader

Abstract

This invention discloses a method for detecting task stagnation in a high-performance computing cluster. The method includes: S1: obtaining a list of running computing tasks on the cluster; S2: querying the last update time of the latest file in the working directory of the computing task; S3: comparing the current system time with the last update time of the latest progress file. If the time difference exceeds a preset threshold, the computing task is marked as a task to be checked; S4: querying the process of the task to be checked, collecting blocking call information and status change parameters of the process within a set time. If the process is in a blocked call state for a long time, or repeatedly executes the same no-progress operation, the task to be checked is determined to be stagnant; S5: querying user information and notifying the user. By first comparing the update time of the computing task file and then observing the process of the computing task, the probability of stagnant tasks being missed is reduced, which can effectively improve the accuracy of detecting whether a computing task is stagnant.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of high-performance computing, and in particular relates to a method for detecting task stalling in a high-performance computing cluster. Background Technology

[0002] The construction and operation costs of high-performance clusters are quite high, making efficient use and maximizing their value particularly important. Currently, users pay for computing resources in computing centers based on the amount of resources used and the duration of use. This pricing mechanism makes computing costs a key factor affecting user experience. However, in high-performance clusters, due to various reasons such as node failures, network interruptions, read / write anomalies, and program bugs, computing tasks sometimes become stalled, meaning they do not execute as expected and do not exit or terminate normally. These stalled tasks not only waste computing resources and affect cluster efficiency but also increase user expenses, further impacting the user experience. Therefore, timely detection of these stalled tasks and release of occupied resources not only helps achieve efficient cluster use but also provides users with a better computing experience.

[0003] Currently, there is a lack of efficient and accurate methods for detecting task stagnation in high-performance computing clusters. Traditional monitoring methods may only focus on task running status indicators, failing to analyze whether the task is truly in a prolonged state of stagnation, thus hindering timely detection and resolution of problems and impacting the overall performance of high-performance clusters and user experience. Summary of the Invention

[0004] To address the problem of accurately detecting stalled tasks in existing technologies, this invention provides a high-performance computing cluster task stall detection method that can accurately identify whether a computing task is in a stalled state.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for detecting task stagnation in a high-performance computing cluster includes the following steps: S1: Obtaining a list of computing tasks running on the cluster; S2: Querying the last update time of the progress file of the computing task; S3: Comparing the current system time with the last update time of the progress file; if the time difference exceeds a preset threshold, the computing task is marked as a task to be checked; S4: Querying the process of the task to be checked, collecting the blocking call information and status change parameters of the process within a set time; if the process remains in a blocked call state for a long time within the set time, or repeatedly executes the same no-progress operation, the task to be checked is determined to be stagnant; S5: Querying the user information that submitted the task to be checked and notifying the user running the task to be checked.

[0006] The beneficial effect of this scheme is that it first compares the file update times of computing tasks, selects computing tasks that have not updated their files within a certain period of time, and then observes the process changes of the selected computing tasks to finally determine whether the computing tasks are stalled. By observing and judging the files and processes in turn, the probability of stalled tasks being missed is reduced, and the accuracy of detecting whether computing tasks are stalled can be effectively improved.

[0007] The high-performance computing cluster task stall detection method provided by the present invention preferably includes step S2 as follows: according to the list obtained in step S1, a query command is sent to the computing tasks on the list to obtain the path information of the submission directory of the computing tasks, and after entering the submission directory according to the path information, the last update time of the progress file of the computing tasks is obtained.

[0008] The high-performance computing cluster task stall detection method provided by the present invention preferably includes a task execution log file and a data output file in the progress file.

[0009] The high-performance computing cluster task stall detection method provided by the present invention preferably has a threshold of 2 hours in step S3.

[0010] The high-performance computing cluster task stall detection method provided by the present invention preferably includes, in step S4, querying the process of the task to be checked: querying the node number of the task to be checked through the scheduling system and logging into the node, then using the process query command to find the process ID corresponding to the task to be checked, and finding the corresponding process based on the process ID.

[0011] The above technical solution has the following beneficial effects: This invention discloses a method for detecting task stagnation in a high-performance computing cluster. S1: Obtain a list of computing tasks running on the cluster; S2: Query the last update time of the progress file of the computing task; S3: Compare the current system time with the last update time of the progress file. If the time difference exceeds a preset threshold, the computing task is marked as a task to be checked; S4: Query the process of the task to be checked, collect the blocking call information and status change parameters of the process within a set time. If the process remains in a blocked call state for a long time within the set time, or repeatedly executes the same no-progress operation, the task to be checked is determined to be stagnant; S5: Query the submitting user information of the task to be checked and notify the user running the task. By first comparing the file update time of the computing tasks, computing tasks that have not updated their files within a certain period are selected. Then, the process change status of the selected computing tasks is observed to finally determine whether the computing task is stagnant. By observing and judging the file and process twice in sequence, the probability of stagnant tasks being missed is reduced, and the accuracy of detecting whether a computing task is stagnant can be effectively improved. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0013] Figure 1 This is a flowchart illustrating a high-performance computing cluster task stall detection method provided by the present invention. Detailed Implementation

[0014] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0015] Currently, there is a lack of efficient and accurate methods for detecting task stagnation in high-performance computing clusters. Traditional monitoring methods may only focus on task running status indicators, failing to analyze whether a task is truly in a prolonged state of stagnation, thus hindering timely detection and resolution of problems and impacting the overall performance and user experience of the high-performance cluster. To address these technical issues, such as... Figure 1 As shown: This application provides a method for detecting task stagnation in a high-performance computing cluster, which may include steps S1-S5.

[0016] S1: Obtain a list of computing tasks currently running on the cluster. Before obtaining the list, it is necessary to traverse the computing tasks, using cluster management tools or writing scripts to obtain a list of all computing tasks currently running on the cluster by interfacing with the scheduling system.

[0017] S2: Query the last update time of the progress file for the computation task; S3: Compare the current system time with the last update time of the progress file. If the time difference exceeds a preset threshold, mark the calculation task as a task to be checked. The purpose of this step is to check the file update status. By comparing the time difference, it is determined whether the progress file has not been updated for a long time. If it has not been updated for a long time, there may be a situation where the task is stalled. The tasks that may be stalled are marked and further checked.

[0018] S4: Query the process of the task to be checked, collect the blocking call information and status change parameters of the process within a set time. If the process remains in a blocked call state for a long time within the set time, or repeatedly executes the same unprogressive operation, the task to be checked is determined to be in a stagnant state. Before querying the process, the node number needs to be obtained first. When there is a potentially stagnant task, it is connected to the scheduling system again. The node number occupied by the task to be checked is queried through the scheduling system command. Log in to the corresponding computing node according to the node number, and use the process query command (such as "ps -ef|grep [task related keywords]") to find the process ID corresponding to the task to be checked. After locating the process, use the strace tool to trace and analyze the found process. For example, execute the command "strace -p [process ID] -te trace=all" to continuously observe the system calls and status changes of the process within a period of time (such as 30 minutes). If it is found that the process is always in a specific system call (such as waiting for I / O operation for a long time without returning) or repeatedly executes the same unprogressive operation during this period, it can be determined that the process is stagnant.

[0019] S5: Query the user information of the user who submitted the task to be inspected and notify the user who ran the task to be inspected; after obtaining the user information, notify the user to handle the stalled task via email, WeChat message, SMS or other instant messaging methods.

[0020] The above steps S1-S5 first compare the update time of the computing task file, and then observe the progress of the computing task. This reduces the probability of missing stalled tasks and can effectively improve the accuracy of detecting whether computing tasks are stalled. It helps users find tasks in a stalled state and release the occupied resources in time. The method provided in this application can improve the efficiency of cluster usage and avoid resource waste.

[0021] In some optional embodiments, step S2 specifically involves: based on the list obtained in step S1, sending a query command to the computation tasks on the list to obtain the path information of the computation task's submission directory; after entering the submission directory based on the path information, obtaining the last update time of the computation task's progress file. According to the command format and interface specifications provided by the scheduling system, for each computation task, a query command is sent to obtain its submission directory path information; for example, in the Slurm scheduling system, which is most common in current high-performance clusters, the command "scontrol show taskID" is used to obtain the task's submission directory.

[0022] In some optional embodiments, the progress files include task execution log files and data output files. Task execution log files are primarily used to record various information during task execution, including but not limited to startup information, running status, error messages, and warning messages. Task log files help in problem localization, monitoring system status, and analyzing task execution. The content format of the task log file is usually determined by the application that generates it; it can be in text format or a specific binary format. Data output files mainly store the result data of task execution. Depending on the application, this data can be very diverse, such as calculation results, processed images, generated report documents, etc. The format of the data output file also depends on specific application requirements; common formats include CSV, JSON, XML, Excel, etc., and there are also domain-specific data formats.

[0023] In some optional embodiments, the threshold in step S3 can be set to 2 hours, and the preset time length of the threshold can be set by the user according to their needs.

[0024] In some optional embodiments, querying the process of the task to be inspected in step S4 includes: querying the node number of the task to be inspected through the scheduling system and logging into the node, then using the process query command to find the process ID corresponding to the task to be inspected, and finding the corresponding process based on the process ID.

[0025] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application filed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

[0026] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for detecting task stalling in a high-performance computing cluster, characterized in that, The method includes: S1: Get a list of computing tasks currently running on the cluster; S2: Query the last update time of the progress file of the computing task; S3: Compare the current system time with the last update time of the progress file. If the time difference exceeds a preset threshold, mark the calculation task as a task to be checked. S4: Query the process of the task to be checked, collect the blocking call information and status change parameters of the process within a set time. If the process is in a blocked call state for a long time within the set time, or repeatedly executes the same no-progress operation, it is determined that the task to be checked is in a stagnant state. S5: Query the user information of the user who submitted the task to be inspected and notify the user who ran the task to be inspected.

2. The high-performance computing cluster task stall detection method according to claim 1, characterized in that, The step of querying the last update time of the progress file of the computing task includes: sending a query command to the computing task on the list obtained in step S1, obtaining the path information of the submission directory of the computing task, and after entering the submission directory according to the path information, obtaining the last update time of the progress file.

3. The method according to claim 1, characterized in that, The progress files include task execution log files and data output files.

4. The method according to claim 1, characterized in that, The threshold in step S3 can be set to 2 hours.

5. The method according to claim 1, characterized in that, The process of querying the task to be inspected in step S4 includes: querying the node number of the task to be inspected through the scheduling system and logging into the node, then using the process query command to find the process ID corresponding to the task to be inspected, and finding the corresponding process based on the process ID.