Fault sensing scheduling method, device and system based on Volcano

Abstract

The invention provides a Volcano-based fault sensing scheduling method, device and system, and a monitoring system, and the method comprises the steps: monitoring a node operation state in a K8s cluster in real time, and writing the generated node state information into a Redis node state table; wherein the node state information comprises a node name and a corresponding node operation state; the scheduling system updates the node information table in the session based on the node state information in the Redis node state table so as to determine a target node corresponding to the Job to be scheduled, thereby completing Job scheduling; wherein the step of updating the node information table in the session comprises the substeps of deleting nodes of which the corresponding node running states are not found in the Redis node state table in the node information table and the found nodes of which the node running states are fault states. Through the cooperation of the monitoring system and the scheduling system, the timeliness and correctness of task calculation are ensured, the use efficiency of computing power resources is optimized, and the operation and maintenance efficiency is improved at the same time.

Description

Technical Field

[0001] The present application relates to the technical field of computing power scheduling, and in particular to a Volcano-based fault-aware scheduling method, device, and system. Background Art

[0002] With the continuous evolution of container technology, various batch computing applications have gradually migrated to the Kubernetes platform, ranging from the initial artificial intelligence applications to big data processing, and then to the recent high-performance computing (HPC) applications, such as gene sequencing, video transcoding, and scientific computing. In the process of such application migration, they have put forward new specific requirements for the Kubernetes platform, especially in terms of scheduling, networking, and storage, requiring the platform to provide higher performance support.

[0003] Artificial intelligence computing is rapidly emerging, and the high costs of operating and maintaining large-scale clusters have become a prominent problem that needs to be solved in the process of training large models. As AI models become larger and larger, K8s clusters provide the necessary tools and support to support the deployment and management of these large models.

[0004] The container scheduling of a standard K8s cluster considers each container independently: that is, take a container, find its suitable node, and then take the next container to schedule. However, distributed AI training containers are different. They are a group of containers. This group of containers must run at the same time to communicate collectively, which is the so-called All_or_Nothing. It is also often called "GangScheduling", which is a strong demand of distributed AI scenarios. Otherwise, multiple distributed jobs will compete at the resource scheduling level, resulting in a deadlock in the resource dimension, and the result is that no one can train normally.

[0005] Volcano is a container batch computing platform based on Kubernetes, mainly used in high-performance computing scenarios. It provides a set of mechanisms that Kubernetes currently lacks, which are usually required for a variety of high-performance workloads such as machine learning big data applications, scientific computing, special effects rendering, etc. As a general batch processing platform, Volcano seamlessly connects with almost all mainstream computing frameworks, such as Spark, TensorFlow, PyTorch, Flink, PaddlePaddle, etc. It also provides hybrid scheduling capabilities for heterogeneous devices including CPUs and GPUs based on various mainstream architectures.

[0006] Volcano provides the following services for mainstream computing frameworks: (1) unified container infrastructure to improve resource utilization; (2) general job management, queues, Fair-share, Gang, bin-pack and other advanced scheduling algorithms; (3) simplified operation and maintenance management. Volcano is naturally compatible with Kubernetes and was born for high-performance computing, following the design philosophy and style of Kubernetes.

[0007] Although Volcano provides a computing power scheduling solution for AI and high-performance computing on Kubernetes, it has not yet been integrated with the user's local monitoring system. Volcano's scheduling system uses the nodes connected to the Kubernetes cluster as the basis for task allocation. However, during the AI ​​computing process, various hardware-related problems may occur on the nodes, such as GPU disconnection, NVMe hard disk detachment, or storage failure. These problems may result in the inability to perform computing tasks normally even if the node is still connected to the Kubernetes cluster, resulting in a waste of computing power resources. Summary of the invention

[0008] The purpose of this application is to provide a fault-aware scheduling method, device and system based on Volcano. When using a K8s cluster and a Volcano-based scheduling system for data computing scheduling, integrating the user's own monitoring system can significantly improve efficiency; this integration allows the scheduling system to perceive faulty nodes in real time, and intelligently assign tasks to normally operating nodes during the scheduling process, avoiding assigning tasks to faulty nodes, thereby ensuring the timeliness and correctness of task calculations, optimizing the use efficiency of computing resources, and improving operation and maintenance efficiency.

[0009] In a first aspect, the present application provides a fault-aware scheduling method based on Volcano, which is applied to a fault-aware scheduling system based on Volcano, and the fault-aware scheduling system includes: a monitoring system and a scheduling system based on Volcano; the monitoring system and the scheduling system are respectively connected to the K8s cluster and establish a connection through the Redis node status table; the method includes: a monitoring system, which monitors the node operation status of the nodes in the K8s cluster in real time, and writes the generated node status information into the Redis node status table; wherein the node status information includes: a node name and a corresponding node operation status; a scheduling system, based on the node status information in the Redis node status table, updates the node information table in the session to determine the target node corresponding to the Job to be scheduled, thereby completing the Job scheduling; wherein updating the node information table in the session includes: deleting nodes in the node information table whose corresponding node operation status is not found in the Redis node status table, and nodes whose node operation status is a faulty state.

[0010] Furthermore, the above-mentioned monitoring system monitors the node operation status in the K8s cluster in real time, and writes the generated node status information into the Redis node status table, including: the monitoring system monitors multiple key indicators for each node in the K8s cluster, and determines the indicator identifier corresponding to each key indicator; wherein the multiple key indicators include: hardware information, operating system level information and business level information; the indicator identifier is used to characterize whether the indicator is normal or abnormal; the indicator identifiers corresponding to multiple key indicators constitute an intermediate value; according to the intermediate value, the node operation status of the node is determined; according to the node name and node operation status corresponding to the node, the node status information is generated and written into the Redis node status table.

[0011] Furthermore, the above-mentioned indicator identifier includes 0 or 1; 0 represents an abnormal indicator; 1 represents a normal indicator; the intermediate value is a string composed of 0 and / or 1, and the length of the string is equal to the number of multiple key indicators; based on the intermediate value, the step of determining the node operation status of the node includes: judging whether each character of the string is a continuous 1, if so, determining that the node operation status of the node is a normal state; if not, determining that the node operation status of the node is a fault state.

[0012] Furthermore, the above-mentioned step of generating node status information according to the node name corresponding to the node and the node operating status includes: using the node name corresponding to the node as the key and the preset identifier corresponding to the node operating status as the value, generating the node status information in the form of a key-value pair; when the preset identifier is 0, it indicates that the node operating status is a fault state; when the preset identifier is 1, it indicates that the node operating status is a normal state.

[0013] Furthermore, the above-mentioned hardware information includes: the status of the GPU, the connection status of the NVME disk, and the adequacy of the disk space; the operating system level information includes: whether the GPU accelerator card driver is normal, whether the system resource utilization rate is normal, whether key processes exist, and whether the system log is abnormal; the business level information includes: whether the business operation environment meets the requirements and whether the business operation performance indicators meet the standards.

[0014] Furthermore, the node status information written into the Redis node status table is configured with a life cycle; the life cycle is a specified duration.

[0015] Furthermore, the above-mentioned scheduling system updates the node information table in the session based on the node status information in the Redis node status table to determine the target node corresponding to the job to be scheduled, including: the scheduling system establishes a session and sends the job to be scheduled to the scheduling queue of the session; traverses all the jobs to be scheduled in the queue, and executes the processing flow from opening the session operation to closing the session operation in sequence according to the defined order to determine the target node corresponding to each job to be scheduled, and binds each job to be scheduled to the corresponding target node; wherein the processing flow from opening the session operation to closing the session operation includes: opening the session operation, obtaining the node information of the K8s cluster, and writing it into the session as snapshot information to initialize the node information table in the session; performing a node detection operation based on the Redis node status table to update the node information table; and continuing to execute enqueue, allocate, preempt, reclaim, backfill, and close the session operations.

[0016] Furthermore, the above-mentioned step of performing node detection operation based on the Redis node status table to update the node information table includes: for each node information in the node information table, determining whether to search the node operation status corresponding to the same node name from the Redis node status table; if not, deleting the node information; if yes, determining whether the found node operation status is a fault state; if yes, deleting the node information.

[0017] In the second aspect, the present application also provides a fault-aware scheduling device based on Volcano, which is applied to a fault-aware scheduling system based on Volcano, and the fault-aware scheduling system includes: a monitoring system and a scheduling system based on Volcano; the monitoring system and the scheduling system are respectively connected to the K8s cluster and establish a connection through the Redis node status table; the device includes: a node status writing module, which is arranged in the monitoring system, for real-time monitoring of the node operation status in the K8s cluster, and writing the generated node status information into the Redis node status table; wherein the node status information includes: node name and node operation status; a job scheduling module, which is arranged in the scheduling system, for updating the node information table in the session based on the node status information in the Redis node status table, so as to determine the target node corresponding to the Job to be scheduled, thereby completing the Job scheduling; wherein, updating the node information table in the session includes: deleting the nodes in the node information table whose corresponding node operation status is not found in the Redis node status table, and the nodes whose node operation status is found to be a faulty state.

[0018] In the third aspect, the present application also provides a fault-aware scheduling system based on Volcano, which includes: a monitoring system and a scheduling system based on Volcano; the monitoring system and the scheduling system are respectively connected to the K8s cluster and establish a connection through the Redis node status table; the fault-aware scheduling system is used to execute the method described in the first aspect.

[0019] The Volcano-based fault-aware scheduling method, device and system provided in the present application are applied to the Volcano-based fault-aware scheduling system, and the fault-aware scheduling system includes: a monitoring system and a Volcano-based scheduling system; the monitoring system and the scheduling system are respectively connected to the K8s cluster and establish a connection through the Redis node status table; the method includes: the monitoring system monitors the node operation status of the K8s cluster in real time, and writes the generated node status information into the Redis node status table; wherein the node status information includes: the node name and the corresponding node operation status; the scheduling system updates the node information table in the session based on the node status information in the Redis node status table to determine the target node corresponding to the Job to be scheduled, thereby completing the Job scheduling; wherein updating the node information table in the session includes: deleting the nodes in the node information table whose corresponding node operation status is not found in the Redis node status table, and the nodes whose node operation status is found to be a fault state. When this application uses the K8s cluster and the Volcano-based scheduling system for data computing scheduling, integrating the user's own monitoring system can significantly improve efficiency; this integration allows the scheduling system to perceive faulty nodes in real time and intelligently assign tasks to normally operating nodes during the scheduling process, avoiding assigning tasks to faulty nodes, thereby ensuring the timeliness and correctness of task calculations, optimizing the use efficiency of computing resources, and improving operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A structural diagram of a Volcano-based fault perception scheduling system provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of a Volcano-based fault perception scheduling method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a Volcano-based fault-aware scheduling process provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a node status information generation process provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a scheduling process provided in an embodiment of the present application;

[0026] Figure 6 A processing flow chart in checkNode provided in an embodiment of the present application;

[0027] Figure 7 A structural block diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solution of the present application will be described clearly and completely below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0029] In the prior art, the scheduling system based on Volcano uses the nodes connected to the Kubernetes cluster as the basis for task allocation, and the nodes may have various hardware-related problems, such as GPU disconnection, NVMe hard disk detachment or storage failure, which may cause the node to be unable to perform computing tasks normally even if it is still connected to the Kubernetes cluster, thereby causing a waste of computing resources. The embodiment of the present application provides a fault-aware scheduling method, device and system based on Volcano. When using the K8s cluster and the scheduling system based on Volcano for data computing scheduling, integrating the user's own monitoring system can significantly improve efficiency; this integration allows the scheduling system to perceive the faulty nodes in real time, and intelligently assign tasks to normally operating nodes during the scheduling process, avoiding assigning tasks to faulty nodes, thereby ensuring the timeliness and correctness of task calculations, optimizing the efficiency of computing resources, and improving operation and maintenance efficiency.

[0030] To facilitate understanding of this embodiment, a Volcano-based fault-aware scheduling method disclosed in an embodiment of the present application is first introduced in detail.

[0031] The embodiment of the present application provides a Volcano-based fault-aware scheduling method, which is applied to a Volcano-based fault-aware scheduling system, see Figure 1As shown, the fault-aware scheduling system includes: a monitoring system and a scheduling system based on Volcano; the monitoring system 11 and the scheduling system 12 are respectively connected to the K8s cluster 13 and establish a connection through the Redis node status table 14; in this embodiment, the Redis service is added to the system, and a node status table is created therein, that is, the above-mentioned Redis node status table, which is used to record the status information of the K8s node. By querying this status table, it can be understood whether the node allows task allocation.

[0032] See also Figure 2 As shown, the above Volcano-based fault-aware scheduling method includes the following steps:

[0033] Step S202, the monitoring system monitors the running status of nodes in the K8s cluster in real time, and writes the generated node status information into the Redis node status table; wherein the node status information includes: node name and corresponding node running status;

[0034] The above monitoring system is the enterprise's monitoring and operation system, which is responsible for monitoring K8s nodes and judging whether there are any problems with the nodes based on comprehensive monitoring rules. If there are problems and the task scheduling conditions are not met, it will be marked as unavailable in the status table; conversely, if it was originally unavailable but the fault has been restored, it will be marked as available.

[0035] Step S204, the scheduling system updates the node information table in the session based on the node status information in the Redis node status table to determine the target node corresponding to the Job to be scheduled, thereby completing the Job scheduling; wherein, updating the node information table in the session includes: deleting the nodes in the node information table whose corresponding node running status is not found in the Redis node status table, and deleting the nodes whose node running status is found to be a faulty state.

[0036] In the scheduling process of Volcano, an action is added to process node status information. First, a list of available node information is obtained from K8s, and then the node status table is queried to obtain the status information of each node, and this information is cached in the scheduling system for subsequent processing.

[0037] In the scheduling process of Volcano, which is actually the calculation process of allocating tasks to the allocable K8s nodes, corresponding modifications are made to increase the judgment of the nodes to determine whether the status of the nodes in the status table is available. Tasks are allocated only when they are available, otherwise no tasks are allocated.

[0038] The embodiment of the present application successfully built a Kubernetes container batch computing platform based on Volcano, and realized the fault perception function in combination with the enterprise's operation and maintenance system. The computing scheduling platform can monitor the status of K8s nodes in real time. Once a node failure is found, the scheduling system will automatically avoid assigning tasks to the faulty node, thereby ensuring the stable operation of the system. Using this technology, significant technical advantages have been achieved in creating high-performance computing scenarios. By integrating computing power scheduling, K8s computing platform and enterprise user monitoring system, this technology ensures that tasks will not be assigned to nodes that have failed or are scheduled for maintenance. This intelligent scheduling avoids the waste of computing resources and ensures the smooth computing process of user business.

[0039] The embodiment of the present application also provides another Volcano-based fault-aware scheduling method, which is implemented on the basis of the above embodiment; this embodiment focuses on describing the specific working process of the monitoring system and the scheduling system.

[0040] See also Figure 3 As shown, the following parts are explained in detail:

[0041] (1) Node status table:

[0042] The node status table is the link between the monitoring system and the scheduling system. The two systems do not interact directly, which avoids the intrusion of the system. Instead, they control whether the node is allowed to be scheduled by maintaining the status data of the status table. In order to simplify the status table, the data of the status table is written in redis. The data is in the form of key / value. The key value is the name of the node, and the value is the node status. 1 means the node is available, and 0 means the node is unavailable.

[0043] For example, if the name of the node is node-10-72, then the value of the key is the string "node-10-72". The value is an integer, 1 means the node is normal, 0 means the node is abnormal. If the corresponding node is not in the status table, then the node is also considered abnormal.

[0044] (2) The monitoring system writes the node status table:

[0045] The enterprise's monitoring system is modified to enable it to evaluate whether these situations will affect AI and big data computing when an alarm or recovery occurs at a monitoring node. This information is then summarized as the node status and output to the status table. This will serve as the basis for the scheduling system to determine whether a node should be ignored.

[0046] The user monitoring system is responsible for monitoring the hardware information of the node, the operating system level, and multiple key indicators at the business level to ensure that the node is allowed to participate in the computing task calculation only when these key indicators are all normal. These key indicators include the status of the GPU, the connection status of the NVME disk, the adequacy of the disk space, etc.

[0047] That is, the above monitoring system monitors the running status of nodes in the K8s cluster in real time, and writes the node status information generated based on the node running status into the Redis node status table. The specific implementation process of this step is as follows:

[0048] (1) The monitoring system monitors multiple key indicators for each node in the K8s cluster and determines the indicator identifier corresponding to each key indicator; wherein the multiple key indicators include: hardware information, operating system level information and business level information; the hardware information includes: GPU status, NVME disk connection status, and disk space sufficiency; the operating system level information includes: whether the GPU accelerator card driver is normal, whether the system resource utilization rate is normal, whether the key process exists, and whether the system log is abnormal; the business level information includes: whether the business operation environment meets the requirements and whether the business operation performance indicators meet the standards.

[0049] The above indicator identifier is used to indicate whether the indicator is normal or abnormal. In this embodiment, the above indicator identifier includes 0 or 1; 0 indicates that the indicator is abnormal; 1 indicates that the indicator is normal;

[0050] (2) The indicator identifiers corresponding to the multiple key indicators constitute an intermediate value; the intermediate value is a string consisting of 0 and / or 1, and the length of the string is equal to the number of the multiple key indicators;

[0051] (3) Determine the node operation state of the node according to the intermediate value; specifically, determine whether each character of the string is a continuous 1, if so, determine that the node operation state of the node is a normal state; if not, determine that the node operation state of the node is a fault state.

[0052] (4) Generate node status information based on the node name and node operation status corresponding to the node and write it into the Redis node status table.

[0053] The node name corresponding to the node is used as the key, and the preset identifier corresponding to the node operation status is used as the value to generate the node status information in the form of a key-value pair; when the preset identifier is 0, it indicates that the node operation status is a fault state; when the preset identifier is 1, it indicates that the node operation status is a normal state.

[0054] exist Figure 4In the diagram shown, a node is monitored by five rules for alarm calculation. Only when the calculation results of these five rules are all normal, the node is considered normal. If any of the rules shows an abnormality in the alarm calculation, the node will be marked as abnormal.

[0055] Since the data based on the five rules involved in the calculation may not be collected and processed at the same time, the concept of intermediate value is used to process the data when writing it into the state table. This intermediate value is a string of length 5, where each character is 1 or 0. The lowest bit of this string corresponds to the result of rule 1, and the highest bit corresponds to the result of rule 5.

[0056] If the calculation results of the five rules are normal, the intermediate value will be set to "11111". When writing data to the status table, the system will check whether the intermediate value is composed of five consecutive '1's. Only when the intermediate value is a continuous '1', the value "1" will be written to the status table, indicating that all rules are normal. If there are non-continuous '1's in the intermediate value, such as "11011", it means that the third rule has a problem and an alarm is generated, then the value "0" will be written to the status table, indicating that the node status is abnormal.

[0057] Under such rules, if an alarm is originally generated, normal judgment can be made when the alarm is restored, and the abnormal state can be restored to the normal state.

[0058] In order to ensure the timeliness of the data written to the status table, the life cycle of the data is managed when the data is written to the status table (actually written to Redis). That is, the node status information written to the Redis node status table is configured with a life cycle; the life cycle is a specified duration. For example, the data life cycle can be set to 1 minute or 5 minutes, and the specific duration is determined by the monitoring system. This is done to avoid the status data becoming garbage data and affecting the efficiency of the scheduling system when the node is no longer monitored.

[0059] The above two points are processed outside the scheduling system. The next part is the part that needs to be modified for Volcano. In the modification of Volcano, a new Action is introduced for preprocessing. The work of this Action is to obtain the status information of each node from the status table when the scheduling system collects all node information to determine whether the node is available. The modified system ensures that only those nodes with normal status will be considered for resource allocation, and any nodes with abnormal status will be automatically excluded from the resource allocation calculation.

[0060] Volcano Scheduler is a component responsible for Pod scheduling. It consists of a series of actions and plugins. Actions define the actions that need to be performed in each step of scheduling; plugins provide specific implementation details of the algorithms in actions according to different scenarios.

[0061] Each scheduling will be executed periodically. First, a session is established and unscheduled jobs are sent to the session's queue. All jobs to be scheduled are traversed and enqueue, allocate, preempt, reclaim, backfill and other actions are executed in the defined order. The parameters passed in these actions are used to find the most suitable node for each job in the session. The job is bound to this node.

[0062] In this embodiment, the Session contains a list of K8s nodes, such as Nodes and NodeList. During each scheduling, the created Session is opened and the information of the current K8s cluster is obtained and written into the Session as snapshot information. Therefore, the Nodes information is obtained, that is, all the cluster nodes of K8s are obtained, and the Node and NodeList data of the Session are initialized at the same time. After completing this work, it becomes the data source for the scheduling to see the node information in the cluster.

[0063] Volcano's scheduling process handles the execution of all actions in sequence. According to the configured actions, they are executed separately, and the parameters passed in are Session.

[0064] According to the above analysis, the modification of Volcano in this embodiment will add an Action processing named checkNode, which is the first action to be processed. In this checkNode, the node information of K8s obtained by the Node and NodeList of the Session in Volcano will be traversed, and the status table will be used to determine whether the node is normal, and the node information table in the Session will be re-maintained.

[0065] That is, the above scheduling system updates the node information table in the session based on the node status information in the Redis node status table to determine the target node corresponding to the job to be scheduled, including the following steps, see Figure 5 As shown:

[0066] (1) The scheduling system establishes a session and sends the jobs to be scheduled to the session's queue.

[0067] (2) Traverse all the jobs to be scheduled in the queue, and execute the processing flow from opening the session operation to closing the session operation in the defined order to determine the target node corresponding to each job to be scheduled, and bind each job to be scheduled to the corresponding target node; wherein, the processing flow from opening the session operation to closing the session operation includes: opening the session operation, obtaining the node information of the K8s cluster, writing it into the session as snapshot information, so as to initialize the node information table in the session; performing node detection operation based on the Redis node status table to update the node information table; and continuing to execute enqueue, allocate, preempt, reclaim, backfill, and close the session operations.

[0068] The above-mentioned step of performing node detection operation based on the Redis node status table to update the node information table includes: for each node information in the node information table, determining whether to search the node operation status corresponding to the same node name from the Redis node status table, if not, deleting the node information; if yes, determining whether the found node operation status is a fault state; if yes, deleting the node information.

[0069] That is, the processing flow in checkNode is as follows Figure 6 As shown, the following steps are included:

[0070] 1. Loop through the Node information in the Session. Each loop process corresponds to a single node. If the node process is completed, the corresponding checkNode process is completed.

[0071] 2. Get the status information from redis based on the node name. If it cannot be obtained, delete the data corresponding to the node in Node and NodeList from the Session, and then process the next node.

[0072] 3. If the status information obtained from Redis is not empty, then judge the status result. If the status is abnormal, delete the data corresponding to the node in Node and NodeList from the Session, and then process the next node.

[0073] 4. If the status information obtained from Redis is normal, continue processing the next node.

[0074] Through the above processing flow, the node information cached in the Session includes both the online node information of K8s and ensures that all nodes are in normal status. In this way, the subsequent scheduling process can automatically detect the faulty node and perform corresponding processing without any changes.

[0075] The Volcano-based fault-aware scheduling method provided in the embodiment of the present application has the following beneficial effects:

[0076] 1. By organically combining the Kubernetes-based AI and big data computing scheduling system with the user's monitoring system, it is possible to detect and identify faulty nodes in real time during the scheduling process, thereby avoiding assigning tasks to these nodes and ensuring the smooth operation of computing tasks.

[0077] 2. The combination of the dispatching system and the user monitoring system provides a perfect supplement to the dispatching system, ensuring the closed-loop management of task allocation and forming a more complete system.

[0078] 3. In the entire system, not only can the node status be automatically monitored and maintained, but also active management of the nodes is supported. For example, when a node is scheduled to be offline for maintenance, the corresponding status can be set in advance so that the system can transition smoothly and ensure system stability and continuity during maintenance.

[0079] 4. In the process of scheduling calculations, the computing nodes used are usually expensive devices, so it is particularly important to improve the utilization rate of these devices. By sensing and identifying faulty nodes in real time during the scheduling process, the execution speed of computing tasks and the efficiency of computing power can be improved, thereby optimizing the overall processing performance and effectively reducing the total energy consumption, thereby achieving the purpose of reducing computing costs.

[0080] Based on the above method embodiment, the embodiment of the present application also provides a fault-aware scheduling device based on Volcano, which is applied to a fault-aware scheduling system based on Volcano. The fault-aware scheduling system includes: a monitoring system and a scheduling system based on Volcano; the monitoring system and the scheduling system are respectively connected to the K8s cluster and establish a connection through the Redis node status table; see Figure 7 As shown, the device includes: a node status writing module 72, which is arranged in the monitoring system, and is used to monitor the node operation status of the K8s cluster in real time, and write the generated node status information into the Redis node status table; wherein the node status information includes: node name and node operation status; a job scheduling module 74, which is arranged in the scheduling system, and is used to update the node information table in the session based on the node status information in the Redis node status table, so as to determine the target node corresponding to the Job to be scheduled, thereby completing the Job scheduling; wherein, updating the node information table in the session includes: deleting the nodes in the node information table whose corresponding node operation status is not found in the Redis node status table, and the nodes whose node operation status is found to be a fault state.

[0081] Furthermore, the node status writing module 72 is also used for: the monitoring system monitors multiple key indicators for each node in the K8s cluster, and determines the indicator identifier corresponding to each key indicator; wherein the multiple key indicators include: hardware information, operating system level information and business level information; the indicator identifier is used to characterize whether the indicator is normal or abnormal; the indicator identifiers corresponding to multiple key indicators constitute an intermediate value; according to the intermediate value, the node operation status of the node is determined; according to the node name and node operation status corresponding to the node, the node status information is generated and written into the Redis node status table.

[0082] Furthermore, the above-mentioned indicator identifier includes 0 or 1; 0 represents an abnormal indicator; 1 represents a normal indicator; the intermediate value is a string composed of 0 and / or 1, and the length of the string is equal to the number of multiple key indicators; the above-mentioned node status writing module 72 is also used to: determine whether each character of the string is a continuous 1, if so, determine that the node operation status of the node is a normal state; if not, determine that the node operation status of the node is a fault state.

[0083] Furthermore, the above-mentioned node status writing module 72 is also used to: use the node name corresponding to the node as the key and the preset identifier corresponding to the node operation status as the value to generate node status information in the form of a key-value pair; when the preset identifier is 0, it indicates that the node operation status is a fault state; when the preset identifier is 1, it indicates that the node operation status is a normal state.

[0084] Furthermore, the above-mentioned hardware information includes: the status of the GPU, the connection status of the NVME disk, and the adequacy of the disk space; the operating system level information includes: whether the GPU accelerator card driver is normal, whether the system resource utilization rate is normal, whether key processes exist, and whether the system log is abnormal; the business level information includes: whether the business operation environment meets the requirements and whether the business operation performance indicators meet the standards.

[0085] Furthermore, the node status information written into the Redis node status table is configured with a life cycle; the life cycle is a specified duration.

[0086] Furthermore, the above-mentioned job scheduling module 74 is also used to: the scheduling system establishes a session and sends the job to be scheduled to the queue to be scheduled of the session; traverses all the jobs to be scheduled in the queue, and executes the processing flow from opening the session operation to closing the session operation in the defined order to determine the target node corresponding to each job to be scheduled, and binds each job to be scheduled to the corresponding target node; wherein, the processing flow from opening the session operation to closing the session operation includes: opening the session operation, obtaining the node information of the K8s cluster, and writing it into the session as snapshot information to initialize the node information table in the session; performing node detection operations based on the Redis node status table to update the node information table; continuing to execute enqueue, allocate, preempt, reclaim, backfill, and close the session operations.

[0087] Furthermore, the above-mentioned Job scheduling module 74 is also used to: for each node information in the node information table, determine whether to search the node operation status corresponding to the same node name from the Redis node status table, if not, delete the node information; if yes, determine whether the found node operation status is a fault state; if yes, delete the node information.

[0088] The device provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the device, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0089] Based on the above method embodiment, the present application embodiment also provides a fault perception scheduling system based on Volcano, see Figure 1 As shown, the fault-aware scheduling system includes: a monitoring system and a scheduling system based on Volcano; the monitoring system and the scheduling system are respectively connected to the K8s cluster and establish a connection through the Redis node status table; the fault-aware scheduling system is used to execute the method described in the above embodiment.

[0090] The system provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the system, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0091] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above method. The specific implementation can be found in the aforementioned method embodiment, which will not be repeated here.

[0092] The computer program products of the methods, devices, and electronic devices provided in the embodiments of the present application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0093] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0094] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0095] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0096] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A Volcano-based fault-aware scheduling method, characterized in that: The method is applied to a fault-aware scheduling system based on Volcano, and the fault-aware scheduling system includes: a monitoring system and a scheduling system based on Volcano; the monitoring system and the scheduling system are respectively connected to a K8s cluster and establish a connection through a Redis node status table; the method includes: The monitoring system monitors the running status of nodes in the K8s cluster in real time, and writes the generated node status information into the Redis node status table; wherein the node status information includes: node name and corresponding node running status; The scheduling system updates the node information table in the session based on the node status information in the Redis node status table to determine the target node corresponding to the job to be scheduled, thereby completing the job scheduling; wherein, the updating of the node information table in the session includes: deleting the nodes in the node information table whose corresponding node running status is not found in the Redis node status table, and the nodes whose found node running status is a faulty state.

2. The method according to claim 1, characterized in that The monitoring system monitors the running status of nodes in the K8s cluster in real time and writes the generated node status information into the Redis node status table, including: The monitoring system monitors multiple key indicators for each node in the K8s cluster and determines the indicator identifier corresponding to each key indicator; wherein the multiple key indicators include: hardware information, operating system level information and business level information; the indicator identifier is used to characterize whether the indicator is normal or abnormal; The indicator identifiers corresponding to the multiple key indicators respectively constitute an intermediate value; according to the intermediate value, the node operation state of the node is determined; Node status information is generated according to the node name corresponding to the node and the node operation status, and is written into the Redis node status table.

3. The method according to claim 2, characterized in that The indicator identifier includes 0 or 1; 0 indicates an abnormal indicator; 1 indicates that the indicator is normal; the intermediate value is a string consisting of 0 and / or 1, and the length of the string is equal to the number of multiple key indicators; The step of determining the node operation state of the node according to the intermediate value comprises: Determine whether each character of the string is a continuous 1, and if so, determine that the node operation state of the node is a normal state; If not, it is determined that the node operation state of the node is a fault state.

4. The method according to claim 2, characterized in that: The step of generating node status information according to the node name corresponding to the node and the node operation status includes: The node name corresponding to the node is used as the key, and the preset identifier corresponding to the node operation status is used as the value to generate the node status information in the form of a key-value pair; when the preset identifier is 0, it indicates that the node operation status is a fault state; when the preset identifier is 1, it indicates that the node operation status is a normal state.

5. The method according to claim 2, characterized in that: The hardware information includes: the status of the GPU, the connection status of the NVME disk, and the adequacy of the disk space; the operating system level information includes: whether the GPU accelerator card driver is normal, whether the system resource utilization rate is normal, whether key processes exist, and whether the system log is abnormal; the business level information includes: whether the business operation environment meets the requirements and whether the business operation performance indicators meet the standards.

6. The method according to claim 2, characterized in that The node status information written into the Redis node status table is configured with a life cycle; the life cycle is a specified duration.

7. The method according to claim 1, characterized in that The scheduling system updates the node information table in the session based on the node status information in the Redis node status table to determine the target node corresponding to the job to be scheduled, including: The scheduling system establishes a session and sends the jobs to be scheduled to the queue to be scheduled of the session; Traverse all the jobs to be scheduled in the queue, and execute the processing flow from opening the session operation to closing the session operation in the defined order to determine the target node corresponding to each job to be scheduled, and bind each job to be scheduled to the corresponding target node; Among them, the processing flow from opening the session operation to closing the session operation includes: opening the session operation, obtaining the node information of the K8s cluster, and writing it into the session as snapshot information to initialize the node information table in the session; performing a node detection operation based on the Redis node status table to update the node information table; continuing to execute enqueue, allocate, preempt, reclaim, backfill, and close the session operations.

8. The method according to claim 7, characterized in that The step of performing a node detection operation based on the Redis node status table to update the node information table includes: For each node information in the node information table, determine whether to search the node operation status corresponding to the same node name from the Redis node status table. If not, delete the node information; if yes, determine whether the found node operation status is a fault state; if yes, delete the node information.

9. A fault-aware scheduling device based on Volcano, characterized in that: The device is applied to a Volcano-based fault-aware scheduling system, which includes: a monitoring system and a Volcano-based scheduling system; the monitoring system and the scheduling system are respectively connected to a K8s cluster and establish a connection through a Redis node status table; the device includes: A node status writing module is provided in the monitoring system, and is used to monitor the node operation status in the K8s cluster in real time, and write the generated node status information into the Redis node status table; wherein the node status information includes: node name and node operation status; A job scheduling module is provided in the scheduling system and is used to update the node information table in the session based on the node status information in the Redis node status table to determine the target node corresponding to the job to be scheduled, thereby completing the job scheduling; wherein, the updating of the node information table in the session includes: deleting the nodes in the node information table whose corresponding node running status is not found in the Redis node status table, and the nodes whose found node running status is a faulty state.

10. A fault-aware scheduling system based on Volcano, characterized in that: The fault-aware scheduling system includes: a monitoring system and a scheduling system based on Volcano; the monitoring system and the scheduling system are respectively connected to the K8s cluster and establish a connection through the Redis node status table; the fault-aware scheduling system is used to execute the method described in any one of claims 1-8.

Images