Fault processing method, system, device, equipment, medium and program of cluster system

Abstract

The invention provides a fault processing method, system, device and equipment of a cluster system, a medium and a program, and relates to the technical field of computers, in particular to the cloud computing technology, artificial intelligence, big data and big model technology, and the method comprises the steps that fault sensing data of AI computing power equipment in the cluster system is acquired; carrying out fault positioning on the AI computing power equipment according to the fault sensing data; under the condition of determining that the target AI computing power equipment has the target fault, performing fault isolation processing on the target AI computing power equipment; and performing automatic fault self-healing processing on the target AI computing power equipment after fault isolation. According to the embodiment of the invention, the fault automatic detection and processing flow of the cluster system in the AI scene can be realized, and the automation, intelligence and processing efficiency of fault processing of the cluster system in the AI scene are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to cloud computing technology, artificial intelligence, big data and big model technology. Background Art

[0002] A computer cluster system is a group of loosely integrated computers connected by software or hardware, which work together closely to complete computing tasks. A single computer in a computer cluster system is usually called a node. The computer cluster system can be connected through a local area network and also supports other connection methods. Computer cluster systems are usually used to improve the computing speed of a single computer and the load balancing of data flows. They are widely used in technical fields such as cloud computing and big data. In the AI ​​(Artificial Intelligence) scenario, the rapid development of large models (LLM, also known as large language models or large language models) has also promoted the proportion of AI computing power devices hosted in cloud native environments. Since AI computing power devices have a higher failure rate than ordinary cluster nodes and can be caused by a variety of factors, it may lead to performance degradation and business interruption of AI model training and reasoning tasks. Therefore, it is crucial to implement an automated fault handling process for AI computing power devices in the cluster system to keep cluster system policies running and improve cluster system operation efficiency. Summary of the invention

[0003] The embodiments of the present disclosure provide a cluster system fault handling method, system, device, equipment, medium and program, which can realize the automatic fault detection and processing flow of the cluster system in the AI ​​scenario, and improve the automation, intelligence and processing efficiency of the cluster system fault handling in the AI ​​scenario.

[0004] In a first aspect, an embodiment of the present disclosure provides a cluster system fault handling method, which is applied to a cluster system fault handling system, wherein the cluster system includes an AI computing device, and the method includes:

[0005] Acquire fault perception data of the AI ​​computing device in the cluster system;

[0006] Perform fault location on the AI ​​computing device according to the fault perception data;

[0007] When it is determined that the target AI computing device has a target fault, perform fault isolation processing on the target AI computing device;

[0008] Perform automated fault self-healing processing on the target AI computing device after fault isolation.

[0009] In a second aspect, an embodiment of the present disclosure provides a fault handling system of a cluster system, including a fault sensing component and a fault handling component; the fault sensing component is communicatively connected with the fault handling component, wherein:

[0010] The fault sensing component is used to obtain fault sensing data of the AI ​​computing device in the cluster system, and send the fault sensing data to the fault processing component;

[0011] The fault processing component is used to locate the fault of the AI ​​computing power device according to the fault perception data; when it is determined that the target AI computing power device has a target fault, perform fault isolation processing on the target AI computing power device; and perform automatic fault self-healing processing on the target AI computing power device after fault isolation.

[0012] In a third aspect, an embodiment of the present disclosure provides a fault handling device of a cluster system, which is configured in a fault handling system of a cluster system, wherein the cluster system includes an AI computing device, and the device includes:

[0013] A fault perception data acquisition module, used to acquire fault perception data of the AI ​​computing power device in the cluster system;

[0014] A fault location module, used to locate the fault of the AI ​​computing device according to the fault perception data;

[0015] A fault isolation processing module, used to perform fault isolation processing on the target AI computing power device when it is determined that the target AI computing power device has a target fault;

[0016] The fault self-healing processing module is used to perform automatic fault self-healing processing on the target AI computing power device after fault isolation.

[0017] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the cluster system fault handling method provided by the embodiment of the first aspect.

[0021] In a fifth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the cluster system fault handling method provided by the embodiment of the first aspect.

[0022] In a sixth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements the cluster system fault handling method provided by the embodiment of the first aspect.

[0023] The disclosed embodiment obtains fault perception data of the AI ​​computing power device in the cluster system through the fault processing system of the cluster system, locates the fault of the AI ​​computing power device according to the fault perception data, and isolates the target AI computing power device when it is determined that the target AI computing power device has a target fault, thereby performing automatic fault self-healing processing on the target AI computing power device after fault isolation, thereby solving the problem of low efficiency of manual processing in the fault processing of the cluster system in the existing AI scenario, realizing the automatic fault detection and processing flow of the cluster system in the AI ​​scenario, and improving the automation, intelligence and processing efficiency of the fault processing of the cluster system in the AI ​​scenario.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0026] Figure 1 is a flow chart of a cluster system fault handling method provided by an embodiment of the present disclosure;

[0027] Figure 2 is a flow chart of another cluster system fault handling method provided by an embodiment of the present disclosure;

[0028] Figure 3 It is a schematic diagram of the overall process of a method for handling a cluster system failure in an AI scenario provided by an embodiment of the present disclosure;

[0029] Figure 4 It is a structural diagram of a cluster system fault handling system provided by an embodiment of the present disclosure;

[0030] Figure 5 It is a schematic diagram of a process of performing fault processing by a fault processing system of a cluster system provided by an embodiment of the present disclosure;

[0031] Figure 6 is a structural diagram of another cluster system fault handling system provided by an embodiment of the present disclosure;

[0032] Figure 7It is a schematic diagram of a processing flow of data flow in a fault processing system of a cluster system provided by an embodiment of the present disclosure;

[0033] Figure 8 It is a schematic diagram of a processing flow of a control flow in a fault processing system of a cluster system provided by an embodiment of the present disclosure;

[0034] Fig. 9 is a structural diagram of a fault handling device for a cluster system provided by an embodiment of the present disclosure;

[0035] Fig.10 The structure diagram of an electronic device used to implement the fault handling method of the cluster system according to the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] Computer clusters can provide cluster computing technology and are an important support for artificial intelligence technology. At the same time, in today's digital age, cloud computing and artificial intelligence are developing at an astonishing speed, forming a close collaborative relationship. At present, in the computer cluster system under the AI ​​scenario, when the cluster system supporting the AI ​​scenario fails, the handling and recovery of the fault strongly rely on manual operations. For example, in a typical cloud scenario based on Kubernetes (referred to as K8s, which is an open source for managing containerized applications on multiple hosts in the cloud platform), when a hardware failure of the AI ​​computing device is detected, the failure is reported to Kubernetes, and the blocking (Cordon) and drainage (Drain) operations are performed. At this time, Kubernetes will not schedule new tasks to the faulty AI node, but the faulty AI node stays in the cluster and relies on manual execution such as removal from the cluster, machine maintenance, and rejoining the cluster. After confirming the health status, the machine will be available again. However, the timeliness of manual execution is low, and it does not meet the user's requirements for machine online rate.

[0038] In one example, Figure 1This is a flowchart of a cluster system fault handling method provided by an embodiment of the present disclosure. This embodiment is applicable to the situation where fault detection and self-healing of AI computing devices in a cluster system are automatically performed. The method can be executed by a cluster system fault handling device, which can be implemented by software and / or hardware, and can generally be integrated into an electronic device. The electronic device can be a terminal device or a server device. As long as the cluster system fault handling system can be integrated to execute the cluster system fault handling method, the embodiment of the present disclosure does not limit the specific device type of the electronic device. Accordingly, Figure 1 As shown, the method includes the following operations:

[0039] S110. Obtain fault perception data of the AI ​​computing device in the cluster system.

[0040] Among them, the fault perception data can be the fault data obtained by automatically detecting faults of AI computing devices in the cluster system.

[0041] In AI scenarios, cluster systems can provide cluster computing technology for AI applications as the underlying support technology. Specifically, the cluster system can provide AI computing services to AI upper-layer applications through AI computing devices. AI computing devices can be types of hardware devices that can provide AI computing power. The so-called AI computing power, or artificial intelligence computing capability, refers to the computing resources and processing power required to execute artificial intelligence algorithms. Exemplarily, AI computing power devices may include, but are not limited to, devices configured with GPUs (Graphics Processing Units, graphics processors), etc. As long as they can provide AI computing power, the embodiments of this disclosure do not limit the specific device types of AI computing power devices.

[0042] In the disclosed embodiment, the fault handling system of the cluster system can use an automated fault detection method to perform fault detection on the AI ​​computing devices in the cluster system, obtain fault data of the faulty AI computing devices, and thus obtain fault perception data of the AI ​​computing devices. Optionally, the fault perception data can be multi-dimensional fault detection data.

[0043] S120. Locate the fault of the AI ​​computing device according to the fault perception data.

[0044] Accordingly, after acquiring the fault perception data of the AI ​​computing devices in the cluster system, the fault processing system of the cluster system can parse and process the fault perception data, thereby determining which AI computing devices in the cluster system have failed, and determining the specific fault type currently occurring in the failed AI computing devices. It is understandable that the number of failed AI computing devices can be one or more.

[0045] S130. When it is determined that a target fault exists in the target AI computing power device, perform fault isolation processing on the target AI computing power device.

[0046] Among them, the target AI computing power device may be an AI computing power device with a fault.

[0047] If the fault handling system determines that the target AI computing device has a fault through fault location, the target AI computing device can be subjected to targeted fault isolation processing according to the current specific fault type of the target AI computing device. It is understandable that the method of fault isolation processing for the target AI computing device may also be different depending on the type of the target fault.

[0048] S140. Perform automated fault self-healing processing on the target AI computing device after fault isolation.

[0049] Correspondingly, after the fault handling system completes fault isolation for the target AI computing power device, it can perform an automated fault self-healing process on the target AI computing power device to achieve automated repair of the target fault of the target AI computing power device.

[0050] It can be seen that the fault handling system can support the automatic recovery function of faulty AI computing device nodes in the cluster system by automating the processes of fault perception, fault diagnosis and location, fault isolation, and fault repair of AI computing equipment, significantly reducing the manpower operation and maintenance costs, and improving the automation, intelligence, and processing efficiency of fault handling in AI scenarios.

[0051] The disclosed embodiment obtains fault perception data of the AI ​​computing power device in the cluster system through the fault processing system of the cluster system, locates the fault of the AI ​​computing power device according to the fault perception data, and isolates the target AI computing power device when it is determined that the target AI computing power device has a target fault, thereby performing automatic fault self-healing processing on the target AI computing power device after fault isolation, thereby solving the problem of low efficiency of manual processing in the fault processing of the cluster system in the existing AI scenario, realizing the automatic fault detection and processing flow of the cluster system in the AI ​​scenario, and improving the automation, intelligence and processing efficiency of the fault processing of the cluster system in the AI ​​scenario.

[0052] In one example, Figure 2 It is a flowchart of another cluster system fault handling method provided by an embodiment of the present disclosure. Based on the technical solutions of the above embodiments, the embodiment of the present disclosure has been optimized and improved, and a variety of specific optional implementation methods for obtaining fault perception data and locating faults, isolating faults and automatically self-healing processing for AI computing power equipment are provided.

[0053] like Figure 2A cluster system fault handling method is shown, comprising:

[0054] S210: Perform hardware fault detection on the AI ​​computing device to obtain hardware fault perception data.

[0055] Among them, hardware fault perception data can be fault-related data of the hardware of AI computing power equipment. The hardware fault-related data can include but are not limited to hardware fault type, fault duration, fault code, fault log data and specific fault classification, etc. As long as it can be related to hardware faults, the embodiments of the present disclosure do not limit the data content and data type included in the hardware fault perception data.

[0056] Optionally, the hardware fault types in the hardware fault perception data may include but are not limited to at least one of CPU (Central Processing Unit) failure, memory failure, motherboard failure, GPU failure and RDMA (Remote Direct Memory Access) network card failure.

[0057] Hardware fault detection for AI computing devices can be performed on the entire underlying hardware structure of the AI ​​computing devices to obtain a full amount of hardware fault perception data.

[0058] S220: Perform software fault detection on the AI ​​computing device to obtain software fault perception data.

[0059] Among them, software fault perception data can be fault-related data in the software of AI computing equipment. Software fault-related data can include but are not limited to software fault type, fault duration, fault code, fault log data and specific fault classification, etc. As long as it can be related to software faults, the embodiments of the present disclosure do not limit the data content and data type included in the software fault perception data.

[0060] Optionally, the software fault types in the software fault perception data may include but are not limited to at least one of runtime faults, compile-time faults, configuration information faults, computing resource faults, network connection faults, software update faults, and API (Application Programming Interface) call faults.

[0061] Software fault detection for AI computing devices can be performed on all software applications and software data of the AI ​​computing devices, thereby obtaining a full amount of software fault perception data.

[0062] The above-mentioned technical solution can realize comprehensive fault perception function by performing hardware fault detection and software fault detection on the AI ​​computing power devices in the cluster system, thereby obtaining comprehensive fault perception data, which can improve the efficiency and reliability of fault detection and processing.

[0063] At present, the application of large models in AI scenarios is becoming more and more extensive. A large model refers to a deep learning model that can process text sequences by training with a large amount of relevant data (such as text data, voice data, or text and image data). It can generate natural language text or understand the meaning of language text. Such models usually have a parameter scale of more than billions. Large language models can handle a variety of natural language tasks, such as text classification, question and answer, and dialogue, and are widely used. The input of a large language model is data, such as text data, voice data, or text and image data. The large language model can obtain the corresponding word vector representation by encoding the input data, and further decodes the encoded word vector, thereby automatically completing the processing of the input data and obtaining the output data corresponding to the input data. Exemplarily, text can be input into a large language model, and the large language model processes and predicts the input text, and outputs the response text corresponding to the text.

[0064] In the inference and training scenarios of large AI models, the most common core fault problems are GPU failures, and the GPU failure rate is higher than that of the CPU. Common GPU failures include: GPU card drop, GPU network jitter, reduced model training accuracy, and RDMA network card failure. GPU failures may be caused by a variety of factors, including but not limited to: hardware failure, driver failure, GPU container runtime failure, and upper-layer application failure. These types of failures may cause performance degradation and business interruption in model training and inference tasks.

[0065] Figure 3 : is a schematic diagram of the overall process of a cluster system fault handling method in an AI scenario provided by an embodiment of the present disclosure. In a specific example, the automatic handling process of GPU faults in an AI scenario is used as an example. Figure 3As shown, assuming that the AI ​​computing device runs in a cloud platform based on a Kubernetes cluster, the fault handling system based on the Kubernetes cluster (hereinafter referred to as the fault handling system) can realize automatic fault perception in the general and structured automated processing process of the GPU. In the fault perception stage, multiple information sources can be used to perceive the fault and obtain fault perception data. For example, by performing fault detection on source data such as Kubernetes events, Promtheus indicators (indicator types such as Counter, Gauge, Histogram, and Summary), BCM (Business Continuity Management) events, IaaS (Infrastructure as a Service) maintenance platform maintenance orders and scheduled inspection data, and using system logs or related monitoring tools to achieve fault diagnosis. Optionally, after performing fault detection on a dedicated fault detection platform, the relevant detection data can be uploaded to the cloud platform, and the fault handling system can connect to the cloud platform to obtain fault perception data in real time. Alternatively, the internal components of the fault handling system may also perform fault detection and acquire fault perception data in real time. The embodiment of the present disclosure does not limit the method for acquiring the fault perception data.

[0066] For example, in the fault perception stage, the fault handling system can support the collection and detection of various hardware faults, including but not limited to CPU faults, memory faults, motherboard faults, and GPU faults. It also supports the use of logs, events, system monitoring indicators, and custom scripts to check faults. The fault handling system can periodically perform fault checking tasks in the cluster system to check various faults, such as checking the health status of components and nodes in Kubernetes.

[0067] S230. Locate the fault of the AI ​​computing device according to the fault perception data.

[0068] In an optional embodiment of the present disclosure, locating the fault of the AI ​​computing power device according to the fault perception data may include: locating the fault of the underlying hardware of the AI ​​computing power device according to the hardware fault perception data; locating the fault of the upper-layer software of the AI ​​computing power device according to the software fault perception data when it is determined that there is no fault in the underlying hardware of the AI ​​computing power device; and performing customized fault location on the AI ​​computing power device according to the hardware fault perception data and / or the software fault perception data when it is determined that there is no fault in the upper-layer software of the AI ​​computing power device.

[0069] Fault location, also known as fault diagnosis, is a key link for the fault handling system of the entire cluster system to restore the fault link diagram. Considering that the upper-layer fault may be caused by the underlying fault, in order to achieve accurate fault location, the principle of layer-by-layer diagnosis and location from the hardware bottom layer to the application upper layer can be followed. Specifically, the underlying hardware of the AI ​​computing device can be firstly fault located according to the hardware fault perception data to determine the location and fault type of the hardware fault of the AI ​​computing device. If it is determined that there is no fault in the underlying hardware of the AI ​​computing device, the upper-layer software of the AI ​​computing device can be further fault located according to the software fault perception data to determine the location and fault type of the software fault of the AI ​​computing device. Correspondingly, if both the hardware fault and the software fault are not located, the AI ​​computing device can also be customized for fault location processing according to the hardware fault perception data and / or the software fault perception data to diagnose the fault type that is not easy to troubleshoot and avoid the failure of fault location.

[0070] For example, when an application cannot use GPU resources, there may be a problem with the underlying GPU device. This diagnostic process from the bottom layer to the upper layer can avoid interference from the bottom layer failure when diagnosing the upper layer problem.

[0071] In an optional embodiment of the present disclosure, locating the fault of the upper-layer software of the AI ​​computing power device according to the software fault perception data may include: detecting and locating the driver fault of the AI ​​computing power device according to the software fault perception data; when it is determined that the AI ​​computing power device does not have the driver fault, detecting and locating the container environment fault of the AI ​​computing power device according to the software fault perception data; when it is determined that the AI ​​computing power device does not have the container environment fault, detecting and locating the upper-layer application fault of the AI ​​computing power device according to the software fault perception data.

[0072] Specifically, when locating the fault of the upper-layer software of the AI ​​computing device according to the software fault perception data, the driver fault can be located first, that is, the driver fault of the AI ​​computing device can be detected and located according to the software fault perception data, such as detecting whether the AI ​​computing device has the fault types such as the GPU driver cannot be accessed, the GPU driver cannot read the card, the card driver is not mounted, and the card driver cannot be used. If the driver fault is not detected, the container environment fault of the AI ​​computing device can be further detected and located according to the software fault perception data to locate the fault in the container environment, such as detecting whether the container crashes, cannot be started, container network failures, container storage failures, and container configuration information errors. If the container environment fault is not detected, the upper-layer application fault of the AI ​​computing device can be further detected and located according to the software fault perception data to locate the upper-layer application fault, such as software faults such as operating systems, databases, and software programs, and the fault types in data, algorithms, and system integration can also be located.

[0073] The above technical solution can achieve comprehensive positioning of software faults in AI computing equipment in the cluster system, improving the accuracy and comprehensiveness of software fault positioning.

[0074] S240. Determine the fault type of the target fault, and perform fault isolation processing on the target AI computing power device according to the fault type of the target fault.

[0075] If a target fault is detected in the target AI computing device of the cluster system during the fault location process, the target AI computing device can be further isolated to perform a subsequent automated fault repair process on the target AI computing device. Fault isolation means isolating a faulty GPU card or a faulty AI computing node from the cluster system to prevent the fault from spreading further. It is understandable that the method of fault isolation for the target AI computing device may also be different depending on the fault type of the target fault.

[0076] In an optional embodiment of the present disclosure, the fault isolation processing of the target AI computing power device according to the fault type of the target fault may include: when it is determined that the target fault is an upper-layer application fault, the target AI computing power device is isolated at the application layer; when it is determined that the target fault is a non-upper-layer application fault, the target AI computing power device is scheduled and isolated.

[0077] Among them, the non-upper-layer application fault may be other fault types except the upper-layer application fault.

[0078] When the target AI computing device is isolated according to the fault type of the target fault, the fault type of the target fault can be determined first, and specific fault isolation measures can be performed on the target AI computing device according to the fault type of the target fault. Specifically, if the target fault is an upper-layer application fault, the target AI computing device can be isolated at the application layer. The so-called application layer isolation can be understood as isolating and controlling data at the application layer to achieve data isolation and access control between different applications. Specifically, application layer isolation focuses on the traffic generated by specific applications, and can block access to specific applications or limit the speed of certain applications. If the target fault is a non-upper-layer application fault, the target AI computing device can be scheduled and isolated. The so-called scheduling isolation can be understood as ensuring that different tasks or services do not interfere with each other when sharing resources through resource scheduling and isolation mechanisms, thereby optimizing resource utilization and system performance. In the AI ​​scenario of the cluster system, scheduling isolation can be achieved by taking the following measures: dynamically adjusting resource allocation according to the actual needs of the task to avoid resource waste; setting priorities for different tasks to ensure that critical tasks can obtain resources first; real-time monitoring of resource usage, and timely alarming and processing once an abnormality is found to prevent system crashes. Through these measures, scheduling isolation in computer cluster systems can be effectively achieved, resource utilization can be optimized, and system performance can be improved.

[0079] In an optional embodiment of the present disclosure, when it is determined that the target fault is a non-upper-layer application fault, scheduling and isolating the target AI computing power device may include: when it is determined that the target fault is a bottom-level hardware fault, isolating the target AI computing power device by a single card or node according to the faulty hardware components associated with the target fault; when it is determined that the target fault is a driver fault and / or a container environment fault, isolating the target AI computing power device by a node.

[0080] The associated faulty hardware component may be a related component that has a hardware fault. Single card isolation may be understood as isolating a single card. Node isolation may be understood as isolating the corresponding AI computing device.

[0081] Specifically, if the target fault is an underlying hardware fault, the associated faulty hardware component of the target fault can be located, the target AI computing device to which it belongs can be determined, and the target AI computing device can be isolated by single card or node. If the target fault is a driver fault and / or a container environment fault, the target AI computing device can be isolated by node.

[0082] In a specific example, let's continue to use the automated processing flow of GPU failures in AI scenarios as an example. Figure 3As shown, fault isolation can be divided into scheduling isolation and application layer isolation. Among them, scheduling isolation can be further divided into: GPU card isolation (i.e. single card isolation) and node isolation. GPU card isolation usually uses the Device Plugin (device plugin mechanism) mechanism to re-report the number of healthy GPU cards to kubelet (Kubernetes node agent), and ignore the unhealthy GPU card number reported. In a hardware failure, if a single GPU card is broken without affecting the use of other GPU cards on the node, the single GPU card can be isolated to reduce the waste of GPU resources; if a single GPU card is broken and affects the use of other GPU cards on the node at the same time, node isolation is required. Node isolation usually performs Cordon operations on the node. Among them, driver failure, container environment failure, CPU failure and GPU failure will affect the use of GPU cards on the entire AI computing power device node, and the entire node needs to be isolated.

[0083] In addition to scheduling isolation, in some scenarios, the controller of the node where the application is located also needs to be isolated to prevent some applications configured with "tolerance of all taints" from being scheduled to the faulty node again, and to prevent applications that directly specify nodes or GPU cards from being scheduled to the faulty node. In the elastic training scenario, the controller of the node where the application is located may terminate other Pods (the smallest deployable computing unit created and managed in Kubernetes) of the training task on the current faulty node, and then start these Pods on other nodes, while preventing other tasks from being scheduled to the faulty node during the fault and fault recovery period. For example, in the elastic training scenario, if a Pod of a training task is restarted due to a node failure, the problem event source of the GPU node will report the faulty node to the controller of the training task, so that the controller can perceive the fault and related events in advance and perform targeted scaling actions, thereby completing the isolation of the problem node or device. During the node failure and during the repair of the faulty node, the controller of the node where the application is located can avoid the need to repeatedly migrate the training task Pod.

[0084] The above technical solution, by adopting different scheduling isolation to isolate non-upper-layer application faults, can not only ensure the stability of the cluster system to the greatest extent, but also ensure effective isolation of faults and improve the success rate of fault repair.

[0085] S250: Confirm the target fault of the target AI computing device.

[0086] S260: Determine whether the target fault has passed fault confirmation. If yes, execute S270; otherwise, execute S280.

[0087] S270. Perform automated fault self-healing processing on the target AI computing device after fault isolation.

[0088] After the fault isolation operation is completed, the target fault needs to be further confirmed to identify possible false alarms in fault location. After confirming that the target fault does need to be recovered, you can further evaluate and confirm the fault handling measures that can be executed for the target fault based on your own business scenario, and enter the fault recovery process to perform automated fault self-healing processing on the target AI computing power device after fault isolation, and perform some operations that may affect the business (such as restarting the node). During the fault recovery phase, you can execute a recovery plan based on the cause of the fault. During the fault recovery phase, you need to select an appropriate recovery strategy for different fault types. If the fault diagnosis and location is indeed a false alarm, the subsequent fault recovery process can be canceled. By confirming the target fault, you can avoid invalid fault handling due to false alarms and improve resource utilization for fault handling.

[0089] In an optional embodiment of the present disclosure, the automated fault self-healing processing of the target AI computing power device after fault isolation may include: when it is determined that the target fault is an underlying hardware fault or a driver fault, migrating the application computing unit of the target AI computing power device, performing a node emptying operation on the target AI computing power device after the migration is completed, and performing automated fault self-healing processing on the target AI computing power device according to the target fault data after the node emptying operation is completed; when it is determined that the target fault is a container environment fault, reinstalling the container configuration tool or restarting the hardware resource management mechanism for the target AI computing power device; when it is determined that the target fault is an upper-layer application fault, relocating the upper-layer application of the target AI computing power device, and performing automated fault self-healing processing according to the fault relocation result.

[0090] The application computing unit may be the smallest computing unit in the cluster system. For example, in a cloud platform based on K8s, the application computing unit may be a Pod. The target fault data may be related fault description data of the target fault.

[0091] In a specific example, let's continue to use the automated processing flow of GPU failures in AI scenarios as an example. Figure 3As shown. For underlying hardware failures (including CPU failures and GPU failures) or driver failures, it is necessary to ensure that there are no applications using GPU resources on the target AI computing device, that is, to perform the corresponding recovery operation after draining the node. However, not all types of applications support the operation of draining nodes. Taking the elastic training task scenario as an example, rashly draining the Pod of the node may cause the training task to fail, for example, the task status may not be saved correctly. Therefore, before draining the node of the target AI computing device, the fault handling system can notify the node controller where the application is located to start migrating the Pod of the GPU application on the node, and receive a notification when the migration is completed. After the migration is completed, the target AI computing device can be automatically self-healed according to the target fault data. For container environment failures, the container configuration tool can be reinstalled or the hardware resource management mechanism Device Plugin Pod can be restarted without migrating the GPU application already running on the target AI computing device. For upper-layer application failures, the upper-layer application of the target AI computing device can be fault relocated, and automatic fault self-healing can be performed according to the fault relocation results.

[0092] The above technical solution adopts specific fault recovery processing strategies for different types of fault types, which can ensure that various types of faults can be automatically repaired in AI scenarios, thereby reducing labor costs.

[0093] S280. Cancel the automated fault self-healing process of the target AI computing device.

[0094] In an optional embodiment of the present disclosure, after performing automated fault self-healing processing on the target AI computing power device after fault isolation, it may also include: obtaining the fault recovery status of the target fault; when it is determined that the target fault recovery is successful, releasing the fault isolation of the target AI computing power device; when it is determined that the target fault recovery has failed, re-performing automated fault self-healing processing on the target AI computing power device after fault isolation.

[0095] When the target AI computing device completes fault recovery, the cluster system's fault handling also needs to confirm whether the target AI computing device has recovered. If it is determined that the target fault has been recovered successfully, the target AI computing device can be released from fault isolation, the repaired resources can be put back online, and the original business applications can be restored to ensure that the faulty node or GPU card can be put online normally and the business applications run smoothly. If it is determined that the target fault has not been recovered successfully, the target AI computing device after fault isolation can be automatically self-healed. The advantage of this setting is that it can ensure that fault-free AI computing devices are reconnected to the cluster system, avoiding unsuccessful fault repairs that cause nodes to still have faults after being connected to the cluster system, affecting the node or even the cluster, thereby ensuring the normal operation of the cluster system and improving the reliability and stability of the cluster system.

[0096] Among them, the automated fault self-healing processing operation can be an operation in which the fault handling system of the cluster system handles the fault by automated means, such as automatically draining the faulty AI computing device and removing it from the cluster, and requesting to connect to the fault repair platform to repair the removed faulty AI computing device. At the same time, the cluster system's fault handling system can try to obtain the repair progress of the faulty AI computing device by the fault repair platform, and after determining that the faulty AI computing device has completed the fault repair, delete the fault information of the faulty AI computing device, and perform a health check on it, so that it can be added to the cluster system after the health check passes.

[0097] The above technical solution uses a variety of means to achieve fault perception, obtain fault perception data, and locate faults in AI computing devices in a hierarchical order. Different isolation methods are used to isolate the faulty AI computing devices according to different fault types, thereby realizing the automatic fault detection and processing flow of the cluster system in the AI ​​scenario, and improving the automation, intelligence and processing efficiency of the fault processing of the cluster system in the AI ​​scenario.

[0098] In one example, Figure 4 is a schematic diagram of a structure of a cluster system fault handling system provided by an embodiment of the present disclosure, such as Figure 4 As shown, the fault processing system 400 of the cluster system may include a fault sensing component 410 and a fault processing component 420, and the fault sensing component 410 is in communication with the fault processing component 420, wherein:

[0099] The fault sensing component 410 is used to obtain the fault sensing data of the AI ​​computing device in the cluster system, and send the fault sensing data to the fault processing component 420;

[0100] The fault processing component 420 is used to locate the fault of the AI ​​computing power device according to the fault perception data; when it is determined that the target AI computing power device has a target fault, perform fault isolation processing on the target AI computing power device; and perform automatic fault self-healing processing on the target AI computing power device after fault isolation.

[0101] Optionally, the fault perception component 410 is also used to: perform hardware fault detection on the AI ​​computing power device to obtain hardware fault perception data; wherein the hardware fault types in the hardware fault perception data include at least one of CPU fault, memory fault, motherboard fault, GPU fault and RDMA network card fault; perform software fault detection on the AI ​​computing power device to obtain software fault perception data.

[0102] Optionally, the fault processing component 420 is also used to: locate the fault of the underlying hardware of the AI ​​computing power device according to the hardware fault perception data; when it is determined that there is no fault in the underlying hardware of the AI ​​computing power device, locate the fault of the upper-level software of the AI ​​computing power device according to the software fault perception data; when it is determined that there is no fault in the upper-level software of the AI ​​computing power device, perform customized fault location on the AI ​​computing power device according to the hardware fault perception data and / or the software fault perception data.

[0103] Optionally, the fault processing component 420 is also used to: detect and locate the driver fault of the AI ​​computing power device according to the software fault perception data; when it is determined that the AI ​​computing power device does not have the driver fault, detect and locate the container environment fault of the AI ​​computing power device according to the software fault perception data; when it is determined that the AI ​​computing power device does not have the container environment fault, detect and locate the upper-layer application fault of the AI ​​computing power device according to the software fault perception data.

[0104] Optionally, the fault handling component 420 is also used to: determine the fault type of the target fault; when it is determined that the target fault is an upper-layer application fault, isolate the target AI computing power device at the application layer; when it is determined that the target fault is a non-upper-layer application fault, schedule and isolate the target AI computing power device.

[0105] Optionally, the fault handling component 420 is also used to: when it is determined that the target fault is an underlying hardware fault, perform single-card isolation or node isolation on the target AI computing power device according to the faulty hardware component associated with the target fault; when it is determined that the target fault is a driver fault and / or a container environment fault, perform node isolation on the target AI computing power device.

[0106] Optionally, the fault handling component 420 is also used to: perform fault confirmation on the target fault of the target AI computing power device; when it is determined that the target fault passes the fault confirmation, perform automatic fault self-healing processing on the target AI computing power device after fault isolation; when it is determined that the target fault does not pass the fault confirmation, cancel the automatic fault self-healing processing process of the target AI computing power device.

[0107] Optionally, the fault handling component 420 is also used to: when it is determined that the target fault is a bottom-level hardware fault or a driver fault, migrate the application computing unit of the target AI computing power device, perform a node emptying operation on the target AI computing power device after the migration is completed, and after the node emptying operation is completed, perform automatic fault self-healing processing on the target AI computing power device according to the target fault data; when it is determined that the target fault is a container environment fault, reinstall the container configuration tool for the target AI computing power device or restart the hardware resource management mechanism; when it is determined that the target fault is an upper-layer application fault, relocate the upper-layer application of the target AI computing power device, and perform automatic fault self-healing processing according to the fault relocation result.

[0108] Optionally, the fault processing component 420 is also used to: obtain the fault recovery status of the target fault; when it is determined that the target fault recovery is successful, lift the fault isolation of the target AI computing power device; when it is determined that the target fault recovery has failed, re-perform automated fault self-healing processing on the target AI computing power device after fault isolation.

[0109] The disclosed embodiment obtains fault perception data of the AI ​​computing power device in the cluster system through the fault processing system of the cluster system, locates the fault of the AI ​​computing power device according to the fault perception data, and isolates the target AI computing power device when it is determined that the target AI computing power device has a target fault, thereby performing automatic fault self-healing processing on the target AI computing power device after fault isolation, thereby solving the problem of low efficiency of manual processing in the fault processing of the cluster system in the existing AI scenario, realizing the automatic fault detection and processing flow of the cluster system in the AI ​​scenario, and improving the automation, intelligence and processing efficiency of the fault processing of the cluster system in the AI ​​scenario.

[0110] The above cluster system fault handling system can execute the cluster system fault handling method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the cluster system fault handling method provided by any embodiment of the present disclosure.

[0111] In one example, Figure 5is a schematic diagram of a process flow of a cluster system fault handling system provided by an embodiment of the present disclosure for performing fault handling. Figure 6 This is a structural diagram of another cluster system fault handling system provided by an embodiment of the present disclosure. The present disclosure embodiment takes a Kubernetes-based cloud platform as a specific application scenario to illustrate the specific implementation process of the cluster system fault handling system performing fault handling.

[0112] In the actual practice of Kubernetes production environment, nodes and Pods may be abnormal due to various reasons, which may affect the overall availability of cluster system services. Since the control plane of Kubernetes cannot perceive node abnormalities in time by default, instances are scheduled to unavailable nodes during the scheduling process. As users of the cluster, users are often unable to perceive node or instance failures in the cluster in time, resulting in failure to stop losses in time. When encountering node or instance failures, users expect the platform to achieve automatic fault recovery, such as automatically migrating instances on failed nodes.

[0113] like Figure 5 As shown, the cluster system fault handling system provided by the embodiment of the present disclosure implements a general cloud native node fault self-healing system, supports automatic completion of cluster node fault self-healing, realizes Kubernetes cluster maintenance-free operation in the event of a fault, and can be applied to the cluster system fault self-healing process in AI scenarios. The core process of the cluster system fault handling system includes:

[0114] Fault perception: A general fault perception framework based on cluster node health monitoring components such as the open source project NPD (Node-Problem-Detector) to perceive various software and hardware faults. In addition, it can also support docking with other fault detection platforms. When the NPD itself cannot operate normally due to a node failure of the health monitoring component itself, it ensures the perception of node failures.

[0115] Fault decision: Supports determining the self-healing strategy to be executed based on the fault type, such as "automatically replace the underlying machine when a GPUPanic fault occurs."

[0116] Fault handling: triggers the execution of fault self-healing actions, such as blocking, draining, replacing, and on-site repair. Various systems are called in this process, such as Kubernetes and public cloud IaaS.

[0117] Fault visualization: supports direct observation of the overall fault status of cluster nodes and the progress of node fault repair through monitoring dashboards.

[0118] Optionally, the cluster system's fault handling system may be applied to a node fault self-healing function of a Cloud Container Engine (CCE) service.

[0119] like Figure 6 As shown, the fault handling system of the cluster system can be composed of multiple CCE components and other related components, and different CCE components can be configured in different nodes of the cluster system. Among them, area A represents the platform side, and area B represents the user side. Specifically, CCE-NPD can realize the perception of various software and hardware faults. The hardware fault detection stand-alone component can be configured in the node of the cluster system to collect machine fault information and report it to the hardware fault management platform. The hardware fault management platform receives the fault information reported by the hardware fault detection stand-alone component for storage / processing and pushes it to the data operation platform. The data operation platform receives the machine fault alarm and distributes the fault to different fault pools according to the fault type. The cloud maintenance center can generate a fault ticket for the fault information contained in the fault pool. The CCE-node fault management service can interact with the cloud maintenance center to obtain node fault information and push it to the Condition of the Node object corresponding to the K8S node (Node Condition is used to store node fault information). CCE cluster management is responsible for implementing CCE clusters / nodes and managing related OpenAPIs (open APIs, also known as open platforms). Optionally, among the above components, the hardware fault management platform, data operation platform, cloud maintenance center, CCE-node fault management service and CCE cluster management can be deployed in area B.

[0120] Among them, Kube-APIServer is a K8S component and is the entry point for K8S cluster request access. Kubelet is a K8S stand-alone component that performs specific tasks such as creating / deleting / configuring Pods. CCE-Node-Remedier (CCE node fault self-healing component) is a cluster node fault self-healing component that can automatically perform node fault repairs based on reported faults. CCE-Node-Remedier-Controller is one of the controllers of CCE-Node-Remedier, responsible for monitoring the node fault information in the cluster stored in Node Condition, and responsible for tuning the two data of NodeRemedier (node ​​repair) and RemedyMachineTemplate (machine repair template). Among them, NodeRemedier is used to control the range of nodes of concern and save node repair security policies; such as the maximum number of nodes processed per day and per hour. RemedyMachineTemplate is used to store machine repair configuration templates, which are used to generate the final repair steps, such as drainage configuration and default repair steps, to obtain one or more RemedyTasks. CCE-Remedy-Task-Controller is another controller of CCE-Node-Remedier, responsible for repairing each specific machine, saving the repair records of specific machines, and monitoring RemedyTask. CCE-Health-Check-Controller is another controller of CCE-Node-Remedier, responsible for health checks on nodes in the cluster, checking healthy nodes, uncordoning them, and resuming scheduling.

[0121] Figure 6 In the example, BCC provider, EBC provider, and EPC provider represent different node equipment suppliers, which can implement specific maintenance actions in response to maintenance requests initiated by the fault handling system. CCE-Autoscaler is used to expand the number of nodes in the cluster system.

[0122] like Figure 6 As shown, the automatic fault handling process of the cluster system in the AI ​​scenario by the cluster system fault handling system may include the following specific operations:

[0123] Step 1: Hardware fault detection: The stand-alone component detects fault information on the stand-alone machine and reports the fault information to the hardware fault management platform to implement the fault perception process. At this stage, the fault data obtained by automated fault detection of the AI ​​computing power equipment in the cluster system can be obtained.

[0124] Step 2: The hardware fault management platform completes the storage and preprocessing of fault information internally and sends it to the data operation platform in the form of an alarm.

[0125] Step 3: The data operation platform completes the fault classification internally according to the fault information and adds it to the Error pool or Fail pool; the fault information in the Fail pool can be pushed to the cloud maintenance center.

[0126] Step 4: The cloud maintenance center receives the fault information and generates a fault ticket.

[0127] Step 5: The CCE-Node Fault Management Service obtains the fault ticket from the cloud maintenance center and locates the specific cluster / node to which the node belongs.

[0128] Step 6: The CCE-node fault management service accesses the API Server of the user cluster and writes the fault information into the Condition of the Node object corresponding to the faulty node.

[0129] Step 7: CCE-Node-Remedier-Controller observes the change in the fault information of the Node object and can perform fault location, for example, determine which AI computing devices in the cluster system have failed and determine the specific fault type of the failed AI computing device.

[0130] Step 8: CCE-Node-Remedier-Controller simultaneously observes changes in two data objects: NodeRemedier and RemedyMachineTemplate. NodeRemedier can reference (ref) the data of RemedyMachineTemplate.

[0131] Step 9: CCE-Node-Remedier-Controller confirms the maintenance steps for a specific machine and creates / updates the RemedyTask object corresponding to the faulty node. The RemedyTask object may include relevant data such as fault isolation, fault confirmation, and fault recovery.

[0132] Step 10: CCE-Remedy-Task-Controller detects a creation / update event of a RemedyTask object.

[0133] Step 11: CCE-Remedy-Task-Controller generates executable maintenance tasks such as cordon / drain, fault isolation processing, etc. according to the RemedyTask object, triggering the maintenance control flow of the machine.

[0134] Step 12: CCE-Remedy-Task-Controller instructs APIServer to perform operations such as blocking, draining, removing from the cluster, application layer isolation, or scheduling isolation for the maintenance task of the faulty node.

[0135] Step 13: CCE-Remedy-Task-Controller requests maintenance authorization for the faulty machine.

[0136] Step 14: The CCE-node fault management service responds to the maintenance authorization request and requests maintenance authorization for the faulty machine through the corresponding platform.

[0137] Step 15: After completing the authorization step, CCE-Remedy-Task-Controller requests the cloud maintenance center to trigger machine maintenance.

[0138] Step 16: The cloud maintenance center can perform machine maintenance by notifying each equipment provider for offline maintenance. Alternatively, it can also perform maintenance online directly, such as restarting the container.

[0139] Step 17: Obtain the machine maintenance progress from the cloud maintenance center through the CCE-Node Fault Management Service.

[0140] Step 18: After the CCE-node fault management service determines that the fault ticket is completed according to the obtained machine maintenance progress, it requests the user cluster API Server to delete the node fault information.

[0141] Step 19: CCE-Health-Check-Controller performs a health check on the machine. If the health check passes, the node is unblocked and the machine returns to normal.

[0142] Correspondingly, the specific process of machine maintenance in the fault handling system of the cluster system includes: data flow and control flow.

[0143] Figure 7 FIG. 1 is a schematic diagram of a data flow processing flow in a cluster system fault processing system provided by an embodiment of the present disclosure. Figure 7 As shown, the overall data flow is as follows:

[0144] 1. Hardware fault detection The stand-alone component detects fault information on the stand-alone machine and reports it to the hardware fault management platform.

[0145] 2. The hardware fault management platform completes the storage and preprocessing of information internally and sends it to the data operation platform in the form of alarms.

[0146] 3. The data operation platform completes the fault classification internally and adds it to the Error pool or Fail pool; the fault information in the Fail pool can be pushed to the cloud maintenance center.

[0147] 4. The cloud maintenance center receives the Fail machine fault ticket and generates a fault ticket.

[0148] 5. CCE-Node Fault Management Service obtains the fault ticket from the cloud maintenance center and locates the fault according to the fault ticket to the specific cluster / node to which the node belongs.

[0149] 6. The CCE-node fault management service accesses the API Server of the user cluster and writes the fault information into the Condition of the Node object corresponding to the faulty node.

[0150] 7.CCE-Node-Remedier-Controller observes changes in Node object fault information.

[0151] 8.CCE-Node-Remedier-Controller simultaneously observes changes in two data objects: NodeRemedier and RemedyMachineTemplate.

[0152] 9. CCE-Node-Remedier-Controller confirms the repair steps for a specific machine and creates / updates the RemedyTask object corresponding to the failed node.

[0153] 10. CCE-Remedy-Task-Controller detects the creation / update event of the RemedyTask object.

[0154] 11. CCE-Remedy-Task-Controller generates executable maintenance tasks based on the RemedyTask object and triggers the maintenance control flow of the machine.

[0155] Figure 8 FIG. 1 is a schematic diagram of a control flow processing flow in a fault handling system of a cluster system provided by an embodiment of the present disclosure. Figure 8 As shown, the overall control flow is as follows:

[0156] 1. CCE-Remedy-Task-Controller performs operations such as blocking, draining, and removing from the cluster for the maintenance task of the faulty node. The operations of draining and removing from the cluster are optional and need to be determined according to the specific fault isolation measures corresponding to the fault type.

[0157] 2. K8S completes the drainage operation of the faulty node.

[0158] 3. CCE-Remedy-Task-Controller requests maintenance authorization for the machine.

[0159] 4. Pre-authorization: CCE directly triggers the maintenance of the faulty machine without interacting with the user; or

[0160] Request authorization: Request user authorization through relevant platforms.

[0161] 5. After CCE-Remedy-Task-Controller completes the authorization step, it requests the cloud maintenance center to trigger machine maintenance.

[0162] 6. The cloud maintenance center can perform machine maintenance and notify each equipment provider to perform offline maintenance. Alternatively, it can also perform maintenance online directly, such as restarting the container.

[0163] 7. CCE-Node Fault Management Service obtains machine maintenance progress from the cloud maintenance center.

[0164] 8. After the CCE-Node Fault Management Service completes the fault ticket, it requests the user cluster API Server to delete the node fault information.

[0165] 9. CCE-Health-Check-Controller performs a health check on the machine. If the health check passes, the node is unblocked and the machine returns to normal.

[0166] The above technical solution improves the node fault self-healing function of the cluster system by applying the cluster system's fault handling system to the cloud native scenario, supports the automatic recovery of faulty nodes in the cluster system, and significantly reduces the manpower operation and maintenance costs of cluster system fault handling. For users, the overall operation and maintenance of the cluster system can be free. That is, users only need to pay attention to and iterate upper-level AI services, such as model training or reasoning, and do not need to pay attention to the underlying infrastructure of the cluster system, thereby reducing user usage costs.

[0167] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information (such as authorization information, etc.) are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0168] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present disclosure.

[0169] In one example, Fig. 9This is a structural diagram of a cluster system fault handling device provided by an embodiment of the present disclosure. The embodiment of the present disclosure can be applied to the situation where the AI ​​computing power device in the cluster system is automatically fault-detected and self-healed. The device is implemented by software and / or hardware and is specifically configured in an electronic device. The electronic device can be a terminal device or a server device. As long as the cluster system fault handling system can be integrated to execute the cluster system fault handling method, the embodiment of the present disclosure does not limit the specific device type of the electronic device.

[0170] like Fig. 9 The fault processing device 900 of a cluster system shown is configured in a fault processing system of the cluster system, wherein the cluster system includes an AI computing device, including: a fault perception data acquisition module 910, a fault location module 920, a fault isolation processing module 930 and a fault self-healing processing module 940. Among them,

[0171] A fault perception data acquisition module 910 is used to acquire fault perception data of the AI ​​computing device in the cluster system;

[0172] A fault location module 920 is used to locate the fault of the AI ​​computing device according to the fault perception data;

[0173] A fault isolation processing module 930 is used to perform fault isolation processing on the target AI computing power device when it is determined that the target AI computing power device has a target fault;

[0174] The fault self-healing processing module 940 is used to perform automatic fault self-healing processing on the target AI computing power device after fault isolation.

[0175] The disclosed embodiment obtains fault perception data of the AI ​​computing power device in the cluster system through the fault processing system of the cluster system, locates the fault of the AI ​​computing power device according to the fault perception data, and isolates the target AI computing power device when it is determined that the target AI computing power device has a target fault, thereby performing automatic fault self-healing processing on the target AI computing power device after fault isolation, thereby solving the problem of low efficiency of manual processing in the fault processing of the cluster system in the existing AI scenario, realizing the automatic fault detection and processing flow of the cluster system in the AI ​​scenario, and improving the automation, intelligence and processing efficiency of the fault processing of the cluster system in the AI ​​scenario.

[0176] Optionally, the fault perception data acquisition module 910 is also used to: perform hardware fault detection on the AI ​​computing power device to obtain hardware fault perception data; wherein the hardware fault types in the hardware fault perception data include at least one of CPU fault, memory fault, motherboard fault, GPU fault and RDMA network card fault; perform software fault detection on the AI ​​computing power device to obtain software fault perception data.

[0177] Optionally, the fault location module 920 is also used to: locate the fault of the underlying hardware of the AI ​​computing power device according to the hardware fault perception data; when it is determined that there is no fault in the underlying hardware of the AI ​​computing power device, locate the fault of the upper-level software of the AI ​​computing power device according to the software fault perception data; when it is determined that there is no fault in the upper-level software of the AI ​​computing power device, perform customized fault location on the AI ​​computing power device according to the hardware fault perception data and / or the software fault perception data.

[0178] Optionally, the fault location module 920 is also used to: detect and locate the driver fault of the AI ​​computing power device according to the software fault perception data; when it is determined that the AI ​​computing power device does not have the driver fault, detect and locate the container environment fault of the AI ​​computing power device according to the software fault perception data; when it is determined that the AI ​​computing power device does not have the container environment fault, detect and locate the upper-layer application fault of the AI ​​computing power device according to the software fault perception data.

[0179] Optionally, the fault isolation processing module 930 is also used to: determine the fault type of the target fault; when it is determined that the target fault is an upper-layer application fault, perform application layer isolation on the target AI computing power device; when it is determined that the target fault is a non-upper-layer application fault, perform scheduling isolation on the target AI computing power device.

[0180] Optionally, the fault isolation processing module 930 is also used to: when it is determined that the target fault is an underlying hardware fault, perform single-card isolation or node isolation on the target AI computing power device according to the associated faulty hardware components of the target fault; when it is determined that the target fault is a driver fault and / or a container environment fault, perform node isolation on the target AI computing power device.

[0181] Optionally, the fault self-healing processing module 940 is also used to: perform fault confirmation on the target fault of the target AI computing power device; when it is determined that the target fault passes the fault confirmation, perform automatic fault self-healing processing on the target AI computing power device after fault isolation; when it is determined that the target fault does not pass the fault confirmation, cancel the automatic fault self-healing processing process of the target AI computing power device.

[0182] Optionally, the fault self-healing processing module 940 is also used to: when it is determined that the target fault is a bottom-level hardware fault or a driver fault, migrate the application computing unit of the target AI computing power device, and after the migration is completed, perform a node emptying operation on the target AI computing power device, and after the node emptying operation is completed, perform automatic fault self-healing processing on the target AI computing power device according to the target fault data; when it is determined that the target fault is a container environment fault, reinstall the container configuration tool for the target AI computing power device or restart the hardware resource management mechanism; when it is determined that the target fault is an upper-layer application fault, relocate the upper-layer application of the target AI computing power device, and perform automatic fault self-healing processing according to the fault relocation result.

[0183] Optionally, the above-mentioned device also includes a fault recovery confirmation module, which is used to: obtain the fault recovery status of the target fault; when it is determined that the target fault recovery is successful, release the fault isolation of the target AI computing power device; when it is determined that the target fault recovery has failed, re-perform automated fault self-healing processing on the target AI computing power device after fault isolation.

[0184] The above cluster system fault handling device can execute the cluster system fault handling method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the cluster system fault handling method provided by any embodiment of the present disclosure.

[0185] Since the above-mentioned cluster system fault handling device is a device that can execute the cluster system fault handling method in the embodiment of the present disclosure, based on the cluster system fault handling method in the embodiment of the present disclosure, the technicians in the field can understand the specific implementation of the cluster system fault handling device in the embodiment and its various variations, so how the cluster system fault handling device implements the cluster system fault handling method in the embodiment of the present disclosure is not described in detail here. As long as the technicians in the field implement the device used by the cluster system fault handling method in the embodiment of the present disclosure, it belongs to the scope of protection of the present disclosure.

[0186] In one example, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0187] Fig.10A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0188] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0189] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0190] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a fault handling method for a cluster system.

[0191] For example, in some embodiments, the fault handling method of the cluster system may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the fault handling method of the cluster system described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the fault handling method of the cluster system in any other appropriate manner (e.g., by means of firmware).

[0192] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0194] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0196] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0197] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0198] The disclosed embodiment obtains fault perception data of the AI ​​computing power device in the cluster system through the fault processing system of the cluster system, locates the fault of the AI ​​computing power device according to the fault perception data, and isolates the target AI computing power device when it is determined that the target AI computing power device has a target fault, thereby performing automatic fault self-healing processing on the target AI computing power device after fault isolation, thereby solving the problem of low efficiency of manual processing in the fault processing of the cluster system in the existing AI scenario, realizing the automatic fault detection and processing flow of the cluster system in the AI ​​scenario, and improving the automation, intelligence and processing efficiency of the fault processing of the cluster system in the AI ​​scenario.

[0199] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0200] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A cluster system fault handling method, applied to a cluster system fault handling system, wherein the cluster system includes an AI computing device, and the method includes: Acquire fault perception data of the AI ​​computing device in the cluster system; Perform fault location on the AI ​​computing device according to the fault perception data; When it is determined that the target AI computing device has a target fault, perform fault isolation processing on the target AI computing device; Perform automated fault self-healing processing on the target AI computing device after fault isolation.

2. The method according to claim 1, wherein: The acquiring the fault perception data of the AI ​​computing device in the cluster system includes: Performing hardware fault detection on the AI ​​computing device to obtain hardware fault perception data; wherein the hardware fault type in the hardware fault perception data includes at least one of a CPU fault, a memory fault, a motherboard fault, a GPU fault, and an RDMA network card fault; Perform software fault detection on the AI ​​computing device to obtain software fault perception data.

3. The method according to claim 2, wherein: The locating the fault of the AI ​​computing device according to the fault perception data includes: Locate the fault of the underlying hardware of the AI ​​computing device according to the hardware fault perception data; When it is determined that there is no fault in the underlying hardware of the AI ​​computing device, locating the fault of the upper-layer software of the AI ​​computing device according to the software fault perception data; When it is determined that there is no fault in the upper-layer software of the AI ​​computing power device, customized fault location is performed on the AI ​​computing power device according to the hardware fault perception data and / or the software fault perception data.

4. The method according to claim 3, wherein: The locating the fault of the upper-layer software of the AI ​​computing device according to the software fault perception data includes: Detecting and locating a driver fault of the AI ​​computing device according to the software fault perception data; When it is determined that the AI ​​computing device does not have the driver fault, detecting and locating the container environment fault of the AI ​​computing device according to the software fault perception data; When it is determined that the AI ​​computing device does not have the container environment fault, the upper-layer application fault of the AI ​​computing device is detected and located according to the software fault perception data.

5. The method according to claim 1, wherein: The performing fault isolation processing on the target AI computing device includes: Determining a fault type of the target fault; When it is determined that the target fault is an upper-layer application fault, the target AI computing device is isolated at the application layer; When it is determined that the target fault is not an upper-layer application fault, the target AI computing power device is scheduled and isolated.

6. The method according to claim 5, wherein: When it is determined that the target fault is not an upper-layer application fault, scheduling and isolating the target AI computing device includes: When it is determined that the target fault is an underlying hardware fault, single-card isolation or node isolation is performed on the target AI computing device according to the faulty hardware components associated with the target fault; When it is determined that the target fault is a driver fault and / or a container environment fault, the target AI computing device is node isolated.

7. The method according to claim 1, wherein: The automatic fault self-healing process of the target AI computing device after fault isolation includes: Perform fault confirmation on the target fault of the target AI computing device; When it is determined that the target fault has passed fault confirmation, automatic fault self-healing processing is performed on the target AI computing device after fault isolation; When it is determined that the target fault has not passed fault confirmation, the automated fault self-healing process of the target AI computing device is canceled.

8. The method according to claim 1 or 7, wherein: The automatic fault self-healing process of the target AI computing device after fault isolation includes: When it is determined that the target fault is an underlying hardware fault or a driver fault, the application computing unit of the target AI computing device is migrated, and after the migration is completed, a node emptying operation is performed on the target AI computing device. After the node emptying operation is completed, automatic fault self-healing processing is performed on the target AI computing device according to the target fault data; If it is determined that the target fault is a container environment fault, reinstall the container configuration tool or restart the hardware resource management mechanism for the target AI computing device; When it is determined that the target fault is an upper-layer application fault, the upper-layer application of the target AI computing device is fault relocated, and automatic fault self-healing processing is performed based on the fault relocation result.

9. The method according to claim 1, after performing automatic fault self-healing processing on the target AI computing device after fault isolation, further comprises: Obtaining a fault recovery status of the target fault; When it is determined that the target fault recovery is successful, the fault isolation of the target AI computing device is lifted; When it is determined that the target fault recovery fails, the target AI computing device after fault isolation is re-performed with automated fault self-healing processing.

10. A fault handling system for a cluster system, comprising a fault sensing component and a fault handling component; The fault sensing component is communicatively connected with the fault processing component, wherein: The fault sensing component is used to obtain fault sensing data of the AI ​​computing device in the cluster system, and send the fault sensing data to the fault processing component; The fault processing component is used to locate the fault of the AI ​​computing device according to the fault perception data; when it is determined that the target AI computing device has a target fault, perform fault isolation processing on the target AI computing device; Perform automated fault self-healing processing on the target AI computing device after fault isolation.

11. A cluster system fault processing device, configured in a cluster system fault processing system, wherein the cluster system includes an AI computing device, and the device includes: A fault perception data acquisition module, used to acquire fault perception data of the AI ​​computing power device in the cluster system; A fault location module, used to locate the fault of the AI ​​computing device according to the fault perception data; A fault isolation processing module, used to perform fault isolation processing on the target AI computing power device when it is determined that the target AI computing power device has a target fault; The fault self-healing processing module is used to perform automatic fault self-healing processing on the target AI computing power device after fault isolation.

12. The device according to claim 11, wherein The fault perception data acquisition module is also used for: Performing hardware fault detection on the AI ​​computing device to obtain hardware fault perception data; wherein the hardware fault type in the hardware fault perception data includes at least one of a CPU fault, a memory fault, a motherboard fault, a GPU fault, and an RDMA network card fault; Perform software fault detection on the AI ​​computing device to obtain software fault perception data.

13. The device according to claim 12, wherein: The fault location module is used for: Locate the fault of the underlying hardware of the AI ​​computing device according to the hardware fault perception data; When it is determined that there is no fault in the underlying hardware of the AI ​​computing device, locating the fault of the upper-layer software of the AI ​​computing device according to the software fault perception data; When it is determined that there is no fault in the upper-layer software of the AI ​​computing power device, customized fault location is performed on the AI ​​computing power device according to the hardware fault perception data and / or the software fault perception data.

14. The device according to claim 13, wherein: The fault location module is used for: Detecting and locating a driver fault of the AI ​​computing device according to the software fault perception data; When it is determined that the AI ​​computing device does not have the driver fault, detecting and locating the container environment fault of the AI ​​computing device according to the software fault perception data; When it is determined that the AI ​​computing device does not have the container environment fault, the upper-layer application fault of the AI ​​computing device is detected and located according to the software fault perception data.

15. The device according to claim 11, wherein The fault isolation processing module is also used for: Determining a fault type of the target fault; When it is determined that the target fault is an upper-layer application fault, the target AI computing device is isolated at the application layer; When it is determined that the target fault is not an upper-layer application fault, the target AI computing power device is scheduled and isolated.

16. The device according to claim 15, wherein: The fault isolation processing module is also used for: When it is determined that the target fault is an underlying hardware fault, single-card isolation or node isolation is performed on the target AI computing device according to the faulty hardware components associated with the target fault; When it is determined that the target fault is a driver fault and / or a container environment fault, the target AI computing device is node isolated.

17. The device according to claim 11, wherein: The fault self-healing processing module is also used for: Perform fault confirmation on the target fault of the target AI computing device; When it is determined that the target fault has passed fault confirmation, automatic fault self-healing processing is performed on the target AI computing device after fault isolation; When it is determined that the target fault has not passed fault confirmation, the automated fault self-healing process of the target AI computing device is canceled.

18. The device according to claim 11 or 17, wherein: The fault self-healing processing module is also used for: When it is determined that the target fault is an underlying hardware fault or a driver fault, the application computing unit of the target AI computing device is migrated, and after the migration is completed, a node emptying operation is performed on the target AI computing device. After the node emptying operation is completed, automatic fault self-healing processing is performed on the target AI computing device according to the target fault data; If it is determined that the target fault is a container environment fault, reinstall the container configuration tool or restart the hardware resource management mechanism for the target AI computing device; When it is determined that the target fault is an upper-layer application fault, the upper-layer application of the target AI computing device is fault relocated, and automatic fault self-healing processing is performed based on the fault relocation result.

19. The apparatus according to claim 11, further comprising a fault recovery confirmation module, configured to: Obtaining a fault recovery status of the target fault; When it is determined that the target fault recovery is successful, the fault isolation of the target AI computing device is lifted; When it is determined that the target fault recovery fails, the target AI computing device after fault isolation is re-performed with automated fault self-healing processing.

20. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the cluster system fault handling method according to any one of claims 1 to 9.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the cluster system fault handling method according to any one of claims 1 to 9.

22. A computer program product comprising a computer program / instructions, wherein: When the computer program / instruction is executed by a processor, the fault handling method of the cluster system described in any one of claims 1 to 9 is implemented.

Images