Fault processing method, device and equipment

Abstract

The invention relates to a fault processing method, device and equipment. The method comprises the steps of periodically obtaining target fault information corresponding to server nodes in an operation node group located in a target cluster; the target fault information is fault information related to a graphic processing unit (GPU) in a corresponding server node in a preset historical time period; for any server node, when it is determined that the target fault information of the server node meets a preset condition, server nodes in a standby node group are scheduled to a running node group, and the server node is scheduled to a to-be-repaired node group; and according to the preset repair strategy, the server nodes in the to-be-repaired node group are periodically repaired, and the repaired server nodes are scheduled to the standby node group, so that the GPU fault detection and repair efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a fault handling method, device and equipment. Background Art

[0002] With the continuous expansion of data scale and the continuous improvement of algorithm complexity, the demand for computing resources in the training process of large models has increased dramatically. Among them, GPU (Graphics Processing Unit) has become the key hardware support for large model training due to its powerful parallel computing capabilities.

[0003] However, in the process of training large models based on GPU server clusters, since large model training usually requires the GPU to operate at high load for a long time, the GPU hardware components are under great pressure, resulting in an increased probability of failure. At the same time, the complex mathematical operations and massive data processing involved in large model training place high demands on the GPU's computing unit and memory system, which can easily lead to calculation errors, memory failures and other problems, thus causing GPU failures.

[0004] In the related art, when a GPU failure occurs in a GPU server cluster, the training task currently being executed by the cluster will stop immediately, and manual work is required to determine the cause of the failure, migrate the tasks, and repair the GPU. This method is inefficient and prolongs the failure time of the cluster, resulting in a significant reduction in the availability of the cluster. Summary of the invention

[0005] The present invention provides a fault handling method, device and equipment, which are used to solve the problem of low efficiency of current GPU fault detection and repair.

[0006] In a first aspect, the present application provides a fault handling method, the method comprising:

[0007] Periodically obtaining target fault information corresponding to each server node in the running node group of the target cluster; the target fault information is: fault information within a preset historical time period related to the graphics processing unit GPU in the corresponding server node;

[0008] For any server node, when it is determined that the target fault information of any of the above server nodes meets the preset conditions, the server nodes in the standby node group are dispatched to the running node group, and any of the above server nodes are dispatched to the node group to be repaired;

[0009] According to the preset repair strategy, each server node in the node group to be repaired is repaired periodically, and the repaired server node is scheduled to the standby node group.

[0010] The above method can detect the fault risk of server nodes periodically based on their target fault information, thereby timely discovering GPU faults and reducing the fault discovery time of the GPU server cluster; when determining the storage fault risk of the server node, the server nodes of the standby node group are scheduled to the running node group, and the faulty nodes are scheduled from the running node group to the node group to be repaired, which can achieve timely isolation of nodes that may have faults, timely isolation of faulty resources, and improve the availability of the GPU server cluster; at the same time, the server nodes of the node group to be repaired are periodically repaired and added to the standby node group, thereby achieving automatic repair of faults.

[0011] In a possible implementation manner, it is determined that the target fault information of any of the server nodes meets the preset condition based on at least one of the following methods:

[0012] When it is detected that the target fault information indicates that an uncorrectable error exists in the GPU of any of the server nodes, it is determined that the target fault information of any of the server nodes meets a preset condition;

[0013] When it is detected that the target fault information indicates that a correctable error exists in a GPU in any of the server nodes, and the number of occurrences of the correctable error reaches a first preset threshold, it is determined that the target fault information of any of the server nodes meets a preset condition;

[0014] When it is detected that the target fault information indicates that a link fault exists in a GPU in any of the server nodes, and the number of occurrences of the link fault reaches a second preset threshold, it is determined that the target fault information of any of the server nodes meets a preset condition.

[0015] The above method determines that there is a failure risk when it is determined that the GPU has a UCE failure, and determines whether there is a failure risk based on the number of failures that occur when it is determined that the GPU has a CE failure or a link failure. Different conditions are set for different indicators, which can ensure the accuracy of fault detection of server nodes and ensure early migration and processing of faulty nodes, thereby improving the stability and availability of the cluster.

[0016] In a possible implementation manner, the above repair strategy includes at least one of the following:

[0017] Updating the GPU version of each server node in the above-mentioned node group to be repaired respectively;

[0018] Updating the BIOS of each server node in the above-mentioned node group to be repaired;

[0019] Restart each server node in the above-mentioned node group to be repaired respectively.

[0020] The above method can automatically repair the server node failure by updating the GPU firmware version and BIOS firmware version of the server node to be repaired and restarting the server.

[0021] In a possible implementation manner, after periodically repairing each server node in the node group to be repaired and before scheduling the repaired server node to the standby node group, the method further includes:

[0022] Performing different types of performance tests on the above-mentioned repaired server nodes, and determining that the test results indicate that all levels of performance tests have passed;

[0023] Among them, the above-mentioned different types of performance tests include at least one of the following: a first test for the GPU performance in the repaired server node, a second test for the interaction performance between the server and the GPU in the repaired server node, and a third test for the software function of the repaired server node.

[0024] The above method performs different types of performance tests on the server node after it is repaired, and determines whether to schedule it to the standby node group based on the test results, thereby ensuring the repair effect of the server node and avoiding putting unrepaired server nodes into use, thereby ensuring the stability and availability of the cluster.

[0025] In a possible implementation manner, if the different types of performance tests include a first test, a second test, and a third test, the different types of performance tests performed on the repaired server node include:

[0026] Performing a first test, a second test, and a third test on the repaired server node in sequence;

[0027] If the test results indicate that any of the tests fails, the method further includes:

[0028] Stop the performance test on the repaired server node, and schedule the repaired server node to the faulty node group.

[0029] The above method, when performing different types of performance tests on the repaired server node, if any test result indicates that the test fails, stops the subsequent test process and directly dispatches it to the fault group, thereby improving the efficiency of fault testing.

[0030] In a possible implementation manner, before scheduling any of the server nodes to the node group to be repaired, the method further includes:

[0031] If any of the above server nodes is in the task execution state, then any of the above server nodes is marked as a node to be repaired;

[0032] The above-mentioned dispatching any of the above-mentioned server nodes to the node group to be repaired includes:

[0033] The status of the node to be repaired is monitored, and after it is determined that the task corresponding to the node to be repaired is completed, the node to be repaired is scheduled to the node group to be repaired.

[0034] The above method, when the server node to be repaired is currently executing a related task, waits for the task to be completed and then schedules it from the running node group to the node group to be tested, which can ensure the smooth execution of the task and thus ensure the stable and efficient operation of the server cluster.

[0035] In a second aspect, the present application provides a fault handling device, the device comprising:

[0036] An acquisition module, used for periodically acquiring target fault information corresponding to each server node in the running node group of the target cluster; the target fault information is: fault information within a preset historical time period related to a graphics processing unit GPU in the corresponding server node;

[0037] A scheduling module is used for scheduling the server nodes in the standby node group to the running node group and scheduling the server nodes to the node group to be repaired when it is determined that the target fault information of any server node meets the preset conditions for any server node;

[0038] The repair module is used to periodically repair each server node in the node group to be repaired according to a preset repair strategy, and schedule the repaired server node to the standby node group.

[0039] In a possible implementation manner, the scheduling module determines that the target fault information of any of the server nodes meets a preset condition based on at least one of the following methods:

[0040] When it is detected that the target fault information indicates that an uncorrectable error exists in the GPU of any of the server nodes, it is determined that the target fault information of any of the server nodes meets a preset condition;

[0041] When it is detected that the target fault information indicates that a correctable error exists in a GPU in any of the server nodes, and the number of occurrences of the correctable error reaches a first preset threshold, it is determined that the target fault information of any of the server nodes meets a preset condition;

[0042] When it is detected that the target fault information indicates that a link fault exists in a GPU in any of the server nodes, and the number of occurrences of the link fault reaches a second preset threshold, it is determined that the target fault information of any of the server nodes meets a preset condition.

[0043] In a possible implementation manner, the above repair strategy includes at least one of the following:

[0044] Updating the GPU version of each server node in the above-mentioned node group to be repaired respectively;

[0045] Updating the BIOS of each server node in the above-mentioned node group to be repaired;

[0046] Restart each server node in the above-mentioned node group to be repaired respectively.

[0047] In a possible implementation manner, after periodically repairing each server node in the node group to be repaired, before scheduling the repaired server node to the standby node group, the repair module is further configured to:

[0048] Performing different types of performance tests on the above-mentioned repaired server nodes, and determining that the test results indicate that all levels of performance tests have passed;

[0049] Among them, the above-mentioned different types of performance tests include at least one of the following: a first test for the GPU performance in the repaired server node, a second test for the interaction performance between the server and the GPU in the repaired server node, and a third test for the software function of the repaired server node.

[0050] In a possible implementation manner, if the different types of performance tests include a first test, a second test, and a third test, the repair module is specifically configured to:

[0051] Performing a first test, a second test, and a third test on the repaired server node in sequence;

[0052] If the test results indicate that any of the tests fails, the method further includes:

[0053] Stop the performance test on the repaired server node, and schedule the repaired server node to the faulty node group.

[0054] In a possible implementation, the scheduling module is further used to:

[0055] If any of the above server nodes is in the task execution state, then any of the above server nodes is marked as a node to be repaired;

[0056] The status of the node to be repaired is monitored, and after it is determined that the task corresponding to the node to be repaired is completed, the node to be repaired is scheduled to the node group to be repaired.

[0057] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the fault handling method when executing the computer program.

[0058] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps in the above-mentioned fault handling method of the present application are implemented.

[0059] In the fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which is stored in a computer-readable storage medium; when the processor of the memory access device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the memory access device executes the steps in the above-mentioned fault handling method of the present application.

[0060] For each of the above-mentioned second to fifth aspects and the technical effects that may be achieved by each of the above-mentioned aspects, please refer to the above-mentioned description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 A schematic diagram of the structure of a fault handling system provided in an embodiment of the present application;

[0063] Figure 2 A flowchart of a fault handling method provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of a GPU fault prediction process provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of a GPU fault repair process provided in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of the structure of a fault handling device provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clear, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operation method in the method embodiment can also be applied to the device embodiment or the system embodiment.

[0069] In the description of this application, "multiple" is understood to be "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent two situations: A is directly connected to B and A is connected to B through C. In addition, in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0070] In order to facilitate those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the following is a brief description of the professional terms involved:

[0071] (1) GPU (Graphics Processing Unit): A microprocessor that performs image processing on personal computers, workstations, game consoles, and some mobile devices (such as tablets, smartphones, etc.). GPU is mainly used to process computer graphics-related tasks, such as graphics rendering in games, decoding in video playback, etc. With the development of technology, the powerful parallel computing capabilities of GPU are widely used in other fields, especially in tasks that require a large amount of parallel computing, such as deep learning in artificial intelligence, scientific computing, big data analysis, etc.; compared with traditional central processing units, GPU has more computing cores and higher memory bandwidth, and can process large amounts of data and parallel tasks at the same time, thereby greatly improving computing efficiency. Therefore, GPU plays a vital role in modern computing and provides strong support for various applications that require high-performance computing.

[0072] (2) Cluster: A group of interconnected computers or servers that work together to achieve a common goal and appear as a unified system. Cluster technology enables multiple computers or servers to work together to provide more powerful performance, higher availability, and better resource management.

[0073] (3) Large models: usually refer to artificial intelligence models that are extremely large in scale, have many parameters, and have powerful language understanding and generation capabilities; these models are built based on deep learning technology and learn language grammar, semantics, pragmatics and other knowledge and patterns by training on massive amounts of data.

[0074] (4) Training: refers to the process of optimizing and adjusting artificial intelligence models with a large number of parameters using large amounts of data and powerful computing resources. In this process, the model learns from massive amounts of text, images, audio and other data, and continuously adjusts its internal parameters to capture patterns, regularities and features in the data.

[0075] (5) CE (Correctable Error): refers to a type of error that occurs in the GPU's video memory. This error can be corrected through a certain error correction mechanism and usually does not immediately cause system crashes or data loss. However, frequent occurrences of video memory CE errors may indicate potential problems with the video memory or related hardware, which may affect the performance and stability of the GPU.

[0076] (6) UCE (Uncorrectable Error): refers to an error that occurs in the GPU's video memory and cannot be repaired by the error correction mechanism. When a video memory UCE error occurs, it usually means that there is a serious problem with the video memory or related hardware, which may cause data corruption, system failure, and even affect the normal operation of the GPU. Unlike CE, UCE errors often indicate that the hardware failure has reached a relatively serious level and requires appropriate repair or replacement measures to resolve.

[0077] (7) PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard): an interface standard for connecting a computer motherboard and various external devices (such as graphics cards, network cards, storage devices, etc.). The server in this application can be connected to the GPU through the PCIe interface.

[0078] (8) Replay Error: A type of error that occurs in PCI communications. When data is transmitted on the PCIe bus, if the receiver fails to successfully receive or process a data packet, the sender may resend the data packet. If this retransmission occurs frequently or abnormally, it will be identified as a PCIe Replay Error. This error may indicate a problem with the PCIe link, such as signal interference, hardware failure, driver problem, or improper system configuration, which may affect the efficiency and stability of data transmission between devices.

[0079] (9) Kubernetes (K8s): An open source container orchestration platform that automates the deployment, scaling, and management of containerized applications; containers are a lightweight virtualization technology.

[0080] (10) Fielddiag: A field diagnostic tool for GPUs. It is a set of methods and procedures used to detect and evaluate whether the GPU is working properly in actual usage scenarios and whether there are potential problems or failures. It includes testing and analyzing the GPU's hardware performance, functional characteristics, compatibility, etc. to determine the health of the GPU and provide a basis for subsequent maintenance, optimization, or replacement decisions.

[0081] (11) NVQual: A set of testing tools used to perform performance evaluation, compatibility testing, stability testing, etc. on NVIDIA GPUs or related products to ensure that they meet specific standards and requirements.

[0082] (12) CUDA sample: usually refers to the sample program provided by NVIDIA CUDA (Compute Unified Device Architecture). CUDA is a parallel computing platform and programming model launched by NVIDIA. CUDA Sample contains a series of sample codes written in CUDA to demonstrate the various functions and features of CUDA, helping developers understand and learn how to use CUDA for parallel computing programming, covering image processing, numerical computing, machine learning and other fields. CUDA Sample is used in this application for GPU fault diagnosis.

[0083] (13) Cluster availability: refers to the ability of a server cluster to provide services normally, operate continuously and meet user needs within a period of time; it is usually measured by the ratio of normal operating time to total time. The higher the availability, the less time the server cluster is experiencing failures, downtime or unable to respond to requests. Specifically, server cluster availability takes into account multiple aspects, including hardware stability, software reliability, network connection continuity, fault recovery speed, load balancing effectiveness and system scalability. A high-availability server cluster can maintain a stable service level in the face of various internal and external factors (such as hardware failures, software errors, network attacks, traffic peaks, etc.), ensuring business continuity and a good user experience.

[0084] At present, the importance of large model training in the field of artificial intelligence is becoming increasingly prominent. With the continuous expansion of data scale and the continuous improvement of algorithm complexity, the demand for computing resources of large models is growing exponentially. Among them, GPU has become the key hardware support for large model training due to its powerful parallel computing capabilities.

[0085] However, during the training of large models, GPUs face many severe challenges and failures are difficult to completely avoid for the following reasons:

[0086] 1) High-intensity use: Large model training places great demands on GPU computing and runs at high load for a long time. This high-intensity and continuous work will cause the GPU hardware components to wear and age faster, which in turn greatly increases the probability of failure.

[0087] 2) Complex computing tasks: Large model training involves extremely complex mathematical operations and data processing, which places extremely high demands on the GPU's computing units and memory systems. When processing these complex tasks, problems such as calculation errors or memory failures may occur, leading to failures.

[0088] In the related art, when a GPU failure occurs in a GPU server cluster, the training task currently being executed by the cluster will stop immediately, and manual work is required to determine the cause of the failure, migrate the tasks, and repair the GPU. This method is inefficient and prolongs the failure time of the cluster, resulting in a significant reduction in the availability of the cluster.

[0089] In summary, how to effectively improve the stability and availability of GPUs in large model training, accurately predict GPU failures in advance and handle them in a timely manner, has become an important technical problem that needs to be solved urgently. At the same time, a complete automatic migration method for training tasks can also greatly shorten the failure time of the cluster and effectively improve the availability of the cluster.

[0090] In response to the above problems, an embodiment of the present application proposes a fault handling method. When a GPU is about to fail based on historical fault information, a spare node is scheduled to a resource pool, and the node that is about to fail is removed from the running node group in a timely manner to ensure the normal execution of the task. At the same time, the detected nodes that are about to fail are centrally repaired, and the repaired nodes are migrated to the spare node group to supplement them. This method realizes the automatic prediction and repair of GPU faults, effectively reduces the failure time of the GPU cluster, and improves the availability of the GPU cluster.

[0091] Figure 1 Schematic diagram of a fault handling system provided in an embodiment of the present application. The fault handling method described in the embodiment of the present application can be implemented based on the system. Specifically, the fault handling method can be applied to monitoring and scheduling equipment in the system, such as Figure 1 As shown, the system specifically includes monitoring and scheduling equipment and a GPU server cluster.

[0092] Optionally, each server node in the GPU server cluster is connected to the same switch ( Figure 1The switch is connected to the monitoring and scheduling device mentioned above, and the monitoring and scheduling device can obtain historical fault information from each server node through the switch to perform subsequent fault prediction and node scheduling.

[0093] In some embodiments, the monitoring and scheduling device may be implemented as a server, which includes a GPU fault warning module, a GPU fault location and repair module, and a cluster scheduling module;

[0094] Specifically, the functions of the above modules are implemented as follows:

[0095] The GPU fault prediction module is used to determine whether the GPU in the server node is about to fail by running the GPU fault information of each server node in the node group, such as CE and Replay Error; and when it is determined that it is about to fail, the server node is marked as a node to be repaired.

[0096] The GPU fault location and repair module is used to automatically repair each server node in the node group to be repaired based on a preset repair strategy. The repair strategy may include: at least one of: GPU firmware update, BIOS firmware update, and server restart; and after the automatic repair is completed, the repair effect of the server node is tested. If the test passes, it is scheduled to the standby node group based on the cluster scheduling module. If the test fails, it is scheduled to the faulty node group based on the cluster scheduling module.

[0097] The cluster scheduling module is used to perform the scheduling of server nodes between different node groups. In some embodiments, the cluster scheduling module can be implemented as K8s, which can provide powerful management and orchestration functions for containerized applications, realize efficient and reliable application deployment and management. As an important tool for modern cloud-native application development and deployment, the embodiments of the present application can realize the scheduling of server nodes between different node groups based on it.

[0098] In some embodiments, the GPU server cluster may include multiple groups, namely, a running node group, a standby node group, a node group to be repaired, and a faulty node group (optional); each server node in the GPU server cluster can predict and repair GPU faults in a timely manner by scheduling between multiple node groups. It should be noted that the GPU servers in the cluster are all integrated with corresponding GPUs, and optionally, the GPU and the corresponding server body can be connected via a PCIe bus.

[0099] Specifically, the functions of the above node groups are implemented as follows:

[0100] The running node group is used to store the server nodes that are currently executing related training tasks.

[0101] The standby node group is used to store the grouping of standby server nodes. When a server in the running node group fails, the server nodes in the standby node group can be scheduled to the running node group at any time to migrate the training tasks.

[0102] The node group to be repaired is used to store server nodes that are defined as nodes to be repaired based on fault information.

[0103] The faulty node group is used to perform fault repair and testing on the nodes stored in the node group to be tested, identify server nodes whose faults cannot be automatically repaired, and wait for replacement by relevant engineers.

[0104] The present application is further described in detail below in conjunction with the accompanying drawings. Figure 2 FIG. 1 is a flow chart of a fault handling method provided in an embodiment of the present application, and the specific implementation process of the method is as follows:

[0105] Step S201, periodically obtaining target fault information corresponding to each server node in the running node group of the target cluster;

[0106] The target fault information is: fault information within a preset historical time period related to a graphics processing unit GPU in a corresponding server node;

[0107] During implementation, the embodiment of the present application periodically monitors each server node in the running node group to obtain GPU fault information in the server node. It should be noted that the embodiment of the present application does not limit the above monitoring period and can be set based on needs during implementation.

[0108] In some embodiments, the target fault information obtained above refers to the historical fault information of the GPU in the server node, and the preset historical time period can be set based on demand during implementation, for example, within 10 days.

[0109] In some embodiments, the above-mentioned target fault information in the embodiments of the present application may include at least one type of sub-information. For different types of sub-information, the corresponding fault identification conditions are also different. Optionally, the above-mentioned target fault information may include at least one of the CE information, UCE information and link fault information of the GPU. The above-mentioned link failure refers to a communication failure between the GPU in the server node and the server body, for example, it may be a PCIe Replay Error.

[0110] Step S202, for any server node, when it is determined that the target fault information of any server node meets the preset conditions, the server nodes in the standby node group are scheduled to the running node group, and any server node is scheduled to the node group to be repaired;

[0111] In a possible implementation manner, it is determined that the target fault information of any server node meets the preset condition based on at least one of the following manners:

[0112] When it is detected that the target fault information indicates that an uncorrectable error exists in the GPU of any of the above server nodes, it is determined that the target fault information of any of the above server nodes meets a preset condition;

[0113] When it is detected that the target fault information indicates that a correctable error exists in a GPU in any of the above-mentioned server nodes, and the number of occurrences of the above-mentioned correctable error reaches a first preset threshold, it is determined that the target fault information of any of the above-mentioned server nodes meets a preset condition;

[0114] When it is detected that the target fault information indicates that a link fault exists in a GPU in any of the server nodes, and the number of occurrences of the link fault reaches a second preset threshold, it is determined that the target fault information of any of the server nodes meets a preset condition.

[0115] In some embodiments, in the embodiments of the present application, different fault identification conditions (i.e., the above-mentioned preset conditions) are set for different types of sub-information in the target fault information. Specifically, for UCE-related information included in the target fault information, when it is identified that the target fault information includes a UCE record corresponding to a GPU, since the UCE cannot be automatically repaired, it is directly determined that the server node is at risk of failure and is scheduled to the node group to be repaired; when the target fault information includes CE-related information, the historical number of occurrences of CE is obtained from the target fault information, and when the number reaches a first preset threshold, it is determined that the server node is at risk of failure, and the value of the first preset threshold can be set based on demand; when the target fault information includes link fault-related information (such as the number of occurrences of Replay Error), the historical number of occurrences of the link failure is obtained from the target fault information, and when the number reaches a second preset threshold, it is determined that the server node is at risk of failure, and the value of the second preset threshold can be set based on demand.

[0116] It should be noted that when the target fault information includes two or more types of sub-information in the above-mentioned CE, UCE and link fault information, when any type of sub-information meets its corresponding fault identification condition, it is determined that the corresponding server node has a fault and is scheduled to the node group to be repaired.

[0117] In some embodiments, the present application may further set identification conditions corresponding to different levels for the above-mentioned sub-information of the same type, and determine the fault risk level corresponding to the sub-information by matching the sub-information in the target fault information with the identification conditions of different levels corresponding to it; for example, for a scenario in which the target fault information includes CE-related information, a first CE sub-threshold (greater than the first preset threshold), a second CE sub-threshold (greater than the first CE sub-threshold), etc. may be further set, and after the historical number of occurrences of CE reaches the first preset threshold, when the historical number of occurrences of CE is between the first preset threshold and the first CE sub-threshold, the fault risk level is determined to be low; when the historical number of occurrences of CE is between the first CE sub-threshold and the second CE sub-threshold, the fault risk level is determined to be medium; when the historical number of occurrences of CE is greater than the second CE sub-threshold, the fault risk level is determined to be high. Furthermore, different repair strategies may be further set for different fault risk levels, and the specific setting method may be set based on demand, and the embodiments of the present application are not limited thereto.

[0118] In a possible implementation, before scheduling any server node to the node group to be repaired, the method of the present application further includes:

[0119] If any server node is in the task execution state, any server node will be marked as a node to be repaired;

[0120] At this time, any server node is dispatched to the node group to be repaired, including:

[0121] The status of the node to be repaired is monitored, and after determining that the task corresponding to the node to be repaired is completed, the node to be repaired is scheduled to the node group to be repaired.

[0122] In some embodiments, for any server node, after determining that the server node has a risk of failure based on the above preset rules, it is necessary to further determine whether the current server node is executing related tasks (such as model training, etc.). If the server node is in a task execution state, the server node is first marked (that is, marked as a node to be repaired), and the task allocation to the node to be repaired is stopped. When the task corresponding to the marked node is completed, it is scheduled to the node group to be repaired. At the same time, if the node to be repaired has a task that has not been started, when it is determined that the node has a risk of failure, the corresponding task that has not been started is assigned to the newly added server node in the running node group (that is, the node scheduled from the standby node group to the running node group).

[0123] It should be noted that the above-mentioned task execution end includes two possibilities: task execution is completed (ie, execution is successful), and task is stopped due to abnormal circumstances.

[0124] Step S203: According to a preset repair strategy, each server node in the node group to be repaired is repaired periodically, and the repaired server node is scheduled to the standby node group.

[0125] In a possible implementation, the above repair strategy includes at least one of the following:

[0126] Updating the version of the GPU of each server node in the node group to be repaired (hereinafter referred to as GPU firmware update);

[0127] Updating the basic input and output system BIOS of each server node in the node group to be repaired (hereinafter referred to as BIOS firmware update);

[0128] Restart each server node in the node group to be repaired (hereinafter referred to as server restart) respectively.

[0129] In some embodiments, the monitoring and scheduling device may pre-store the latest version of the GPU firmware data packet and the latest version of the BIOS firmware data packet. When the server node is repaired according to the repair strategy, the corresponding device firmware is updated directly based on the pre-stored data packet.

[0130] In some embodiments, when the above-mentioned repair strategy includes multiple items, the embodiment of the present application does not limit the order of GPU firmware update and BIOS firmware update, but if server restart is included at the same time, the server restart has the lowest priority, that is, GPU firmware update or BIOS firmware update is executed first.

[0131] In a possible implementation, after periodically repairing each server node in the node group to be repaired, before scheduling the repaired server node to the standby node group, the method of the present application further includes:

[0132] Perform different types of performance tests on the repaired server nodes and determine that the test results indicate that all levels of performance tests have passed;

[0133] Among them, different types of performance tests include at least one of the following: a first test for the GPU performance in the repaired server node, a second test for the interaction performance between the server and the GPU in the repaired server node, and a third test for the software function of the repaired server node.

[0134] In some embodiments, after the server node is repaired based on a preset repair strategy, the repair effect of the server node is also tested in the embodiments of the present application, that is, a performance test is performed on the repaired server node to determine whether it is successfully repaired; specifically, the test may include different types of tests, such as a test of GPU performance (that is, the first test, which can be optionally implemented through Field diag), a test of the connection between the server and the GPU (that is, the above-mentioned second test, which can be optionally implemented through NVQual), and a functional test of the application layer software (that is, the above-mentioned third test, which can be optionally implemented through CUDA Sample).

[0135] In the specific implementation, if the test results indicate that all types of tests have passed, it can be determined that the server node has been repaired successfully, and it can be scheduled to the spare node group for subsequent use; if the test results indicate that any type of test result has failed, it can be determined that the server node has not been repaired successfully, and it can be scheduled to the faulty node group for relevant technicians to repair or replace it.

[0136] In a possible implementation, if the different types of performance tests include a first test, a second test, and a third test, different types of performance tests are performed on the repaired server node, including:

[0137] Performing a first test, a second test, and a third test on the repaired server node in sequence;

[0138] If the test result indicates that any test result is a failure, the method further includes:

[0139] Stop the performance test on the repaired server node and schedule the repaired server node to the faulty node group.

[0140] In some embodiments, different types of tests in the embodiments of the present application have different corresponding priorities. The first test has the highest priority, the second test is second, and the third test has the lowest priority. That is, when executing the corresponding performance tests, they are executed in the order of the first test, the second test, and the third test. When the test result of one of the tests is failed, the subsequent performance tests are stopped, and it is directly determined that the repair has not been successful, and it is scheduled to the faulty node group.

[0141] In some embodiments, when dispatching a server node to a fault node group, a fault log corresponding to the server node can also be generated. The fault log can carry the corresponding test results, target fault information, and repair strategies, etc., to facilitate subsequent related technical personnel to perform fault repairs. At the same time, a reminder message can also be sent to relevant personnel, which can be a text message, a phone call, an alarm light, etc., so that relevant technical personnel can perform fault repairs in a timely manner.

[0142] The above method can detect the fault risk of server nodes periodically based on their target fault information, thereby timely discovering GPU faults and reducing the fault discovery time of the GPU server cluster; when determining the storage fault risk of the server node, the server nodes of the standby node group are scheduled to the running node group, and the faulty nodes are scheduled from the running node group to the node group to be repaired, which can achieve timely isolation of nodes that may have faults, timely isolation of faulty resources, and improve the availability of the GPU server cluster; at the same time, the server nodes of the node group to be repaired are periodically repaired and added to the standby node group, thereby achieving automatic repair of faults.

[0143] Figure 3 A schematic diagram of a GPU fault prediction process provided in an embodiment of the present application is shown below in combination with Figure 3 , the above GPU fault prediction process is described in detail with a specific example, and the process specifically includes:

[0144] Step S301, periodically obtaining target fault information corresponding to each server node;

[0145] The target fault information includes at least one of UCE information, CE information, and Replay Error information; the information includes the historical occurrence times of the corresponding fault.

[0146] It should be noted that the above-mentioned server node is any node located in the running node group.

[0147] Step S302, perform fault prediction based on the target fault information to determine whether the server node has a fault risk; if yes, execute step S303, otherwise return to execute step S301;

[0148] Step S303: Schedule the server node of the standby node group to the running node group, and mark the server node as a node to be repaired.

[0149] Step S304: determine that the task corresponding to the node to be repaired is completed, and schedule the node to be repaired to a node group to be repaired.

[0150] Figure 4 A schematic diagram of a GPU fault repair process provided in an embodiment of the present application is shown below in combination with Figure 4 The above GPU fault repair process is described in detail with a specific example. The process specifically includes:

[0151] Step S401, sequentially performing GPU firmware update, BIOS firmware update and server restart on the server node;

[0152] It should be noted that the above-mentioned server node is any node in the node group to be repaired.

[0153] Step S402, performing Field diag, NVQual and CUDA Sample tests on the server node in sequence;

[0154] Step S403, determine whether the test result indicates that all tests have passed; if yes, proceed to step S404, if no, proceed to step S405;

[0155] Step S404, dispatching the server node to the standby node group;

[0156] Step S405: dispatch the server node to the faulty node group.

[0157] Based on the same inventive concept, the present application also provides a fault handling device, see Figure 5 , the device comprises:

[0158] The acquisition module 501 is used to periodically acquire target fault information corresponding to each server node in the running node group of the target cluster; the target fault information is: fault information within a preset historical time period related to the graphics processing unit GPU in the corresponding server node;

[0159] The scheduling module 502 is used for scheduling the server nodes in the standby node group to the running node group and scheduling the server nodes to the node group to be repaired when determining that the target fault information of the server node meets the preset conditions for any server node;

[0160] The repair module 503 is used to periodically repair each server node in the node group to be repaired according to a preset repair strategy, and schedule the repaired server node to the standby node group.

[0161] In a possible implementation manner, the scheduling module 502 determines that the target fault information of any server node meets a preset condition based on at least one of the following methods:

[0162] When it is detected that the target fault information indicates that an uncorrectable error exists in the GPU of any of the server nodes, it is determined that the target fault information of any of the server nodes meets a preset condition;

[0163] When it is detected that the target fault information indicates that a correctable error exists in a GPU in any of the server nodes, and the number of occurrences of the correctable error reaches a first preset threshold, it is determined that the target fault information of any of the server nodes meets a preset condition;

[0164] When it is detected that the target fault information indicates that a link fault exists in a GPU in any of the server nodes, and the number of occurrences of the link fault reaches a second preset threshold, it is determined that the target fault information of any of the server nodes meets a preset condition.

[0165] In a possible implementation manner, the above repair strategy includes at least one of the following:

[0166] Updating the GPU version of each server node in the above-mentioned node group to be repaired respectively;

[0167] Updating the BIOS of each server node in the above-mentioned node group to be repaired;

[0168] Restart each server node in the above-mentioned node group to be repaired respectively.

[0169] In a possible implementation manner, after periodically repairing each server node in the node group to be repaired, before scheduling the repaired server node to the standby node group, the repair module 503 is further used to:

[0170] Performing different types of performance tests on the above-mentioned repaired server nodes, and determining that the test results indicate that all levels of performance tests have passed;

[0171] Among them, the above-mentioned different types of performance tests include at least one of the following: a first test for the GPU performance in the repaired server node, a second test for the interaction performance between the server and the GPU in the repaired server node, and a third test for the software function of the repaired server node.

[0172] In a possible implementation manner, if the different types of performance tests include a first test, a second test, and a third test, the repair module 503 is specifically configured to:

[0173] Performing a first test, a second test, and a third test on the repaired server node in sequence;

[0174] If the test results indicate that any of the tests fails, the method further includes:

[0175] Stop the performance test on the repaired server node, and schedule the repaired server node to the faulty node group.

[0176] In a possible implementation, the scheduling module 502 is further configured to:

[0177] If any of the above server nodes is in the task execution state, then any of the above server nodes is marked as a node to be repaired;

[0178] The status of the node to be repaired is monitored, and after it is determined that the task corresponding to the node to be repaired is completed, the node to be repaired is scheduled to the node group to be repaired.

[0179] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, and the electronic device can realize the function of the aforementioned fault handling device, referring to Figure 6 , the electronic device comprises:

[0180] At least one processor 601, and a memory 602 connected to the at least one processor 601. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6 In the example, the processor 601 and the memory 602 are connected via a bus 600. The bus 600 is Figure 6 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 600 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 601 can also be called a controller, and there is no limitation on the name.

[0181] In the embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 can execute the fault handling method discussed above by executing the instructions stored in the memory 602. The processor 601 can implement Figure 5 The functions of each module in the device shown.

[0182] Among them, the processor 601 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 602 and calling data stored in the memory 602, the various functions of the device and processing data, the device can be monitored as a whole.

[0183] In one possible design, the processor 601 may include one or more processing units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0184] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the fault handling method disclosed in the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.

[0185] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0186] By designing and programming the processor 601, the code corresponding to the application anomaly detection method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 2 The steps of the fault handling method of the embodiment shown are as follows: How to design and program the processor 401 is a technique well known to those skilled in the art and will not be described in detail here.

[0187] An embodiment of the present application also provides a computer-readable storage medium that stores computer-executable instructions required to execute the above-mentioned processor, which includes a program required to execute the above-mentioned processor.

[0188] In some possible implementations, various aspects of the fault handling method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the above-mentioned program product is run on an electronic device, the above-mentioned program code is used to enable the above-mentioned electronic device to execute the steps of the fault handling method according to various exemplary implementations of the present application described above in this specification.

[0189] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0190] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0194] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A fault handling method, characterized in that: The method comprises: Periodically obtaining target fault information corresponding to each server node in the running node group of the target cluster; the target fault information is: fault information within a preset historical time period related to a graphics processing unit GPU in the corresponding server node; For any server node, when it is determined that the target fault information of any server node meets the preset conditions, the server nodes in the standby node group are scheduled to the running node group, and the any server node is scheduled to the node group to be repaired; According to the preset repair strategy, each server node in the node group to be repaired is repaired periodically, and the repaired server node is scheduled to the standby node group.

2. The method according to claim 1, characterized in that: Determining that the target fault information of any server node meets a preset condition is based on at least one of the following methods: When it is detected that the target fault information indicates that an uncorrectable error exists in the GPU of any server node, determining that the target fault information of any server node meets a preset condition; When it is detected that the target fault information indicates that a correctable error exists in the GPU of any server node, and the number of occurrences of the correctable error reaches a first preset threshold, it is determined that the target fault information of any server node meets a preset condition; When it is detected that the target fault information indicates that a link fault exists in the GPU of any server node and the number of occurrences of the link fault reaches a second preset threshold, it is determined that the target fault information of any server node meets a preset condition.

3. The method according to claim 1, characterized in that The repair strategy includes at least one of the following: Updating the version of the GPU of each server node in the node group to be repaired respectively; Updating the version of the basic input and output system BIOS of each server node in the node group to be repaired respectively; Restart each server node in the node group to be repaired respectively.

4. The method according to claim 1, characterized in that: After each server node in the node group to be repaired is repaired periodically, before the repaired server node is scheduled to the standby node group, the method further includes: Performing different types of performance tests on the repaired server node, and determining that the test results indicate that all levels of performance tests have passed; Among them, the different types of performance tests include at least one of the following: a first test for the GPU performance in the repaired server node, a second test for the interaction performance between the server and the GPU in the repaired server node, and a third test for the software function of the repaired server node.

5. The method according to claim 4, characterized in that If the different types of performance tests include a first test, a second test, and a third test, then performing different types of performance tests on the repaired server node includes: performing a first test, a second test, and a third test on the repaired server node in sequence; If the test result indicates that any test result is a failure, the method further includes: Stop the performance test on the repaired server node, and schedule the repaired server node to the faulty node group.

6. The method according to any one of claims 1 to 5, characterized in that: Before scheduling any server node to the node group to be repaired, the method further includes: If any of the server nodes is in a task execution state, marking the any of the server nodes as a node to be repaired; The step of dispatching any server node to the node group to be repaired includes: The state of the node to be repaired is monitored, and after it is determined that the task corresponding to the node to be repaired is completed, the node to be repaired is scheduled to a node group to be repaired.

7. A fault handling device, characterized in that: The device comprises: An acquisition module, used for periodically acquiring target fault information corresponding to each server node in the running node group of the target cluster; the target fault information is: fault information within a preset historical time period related to a graphics processing unit GPU in the corresponding server node; A scheduling module is used for scheduling the server nodes in the standby node group to the running node group and scheduling the server nodes to the node group to be repaired when it is determined that the target fault information of any server node meets the preset conditions for any server node; The repair module is used to periodically repair each server node in the node group to be repaired according to a preset repair strategy, and schedule the repaired server node to the standby node group.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 6 when executing a computer program stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 6.

Images