GPU fault early warning processing method and computing device

Abstract

The embodiment of the invention provides a GPU fault early warning processing method and computing equipment, the method is applied to first computing equipment, and the method comprises the steps that early warning information of a target GPU is acquired; the early warning information is determined based on working feature information of the target GPU in a computing node; determining a check point adjustment strategy according to the early warning information; the check point adjustment strategy is a strategy used for adjusting the trigger frequency of a check point of a target GPU instance, and the target GPU instance is a GPU instance running a calculation task in the target GPU; and adjusting the trigger frequency of the check point of the target GPU instance according to the check point adjustment strategy. According to the method provided by the embodiment of the invention, the progress loss caused by the GPU fault can be effectively and reasonably reduced, the data loss caused by the GPU fault can also be reduced, the utilization efficiency of computing resources is improved, and the overall performance and stability of the system are also improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of computing devices, and more particularly to a GPU fault warning processing method and a computing device. Background Art

[0002] In the fields of high-performance computing and artificial intelligence, graphics processing units (GPUs) are widely used due to their powerful parallel computing capabilities. However, GPUs may fail during long-term, high-load operation. When a serious hardware failure occurs on a GPU, the computing tasks running on the GPU will be abandoned, which will not only cause computing task failure and data loss, but also loss of corresponding task progress.

[0003] In order to avoid losing all task progress after a failure, related technologies will periodically trigger checkpoints. When a checkpoint is triggered, the file running to the checkpoint can be stored in a pre-defined path. In this way, when a computing task is abnormally terminated due to a GPU failure or other reasons, after the user resubmits the task, it can be restored from the file corresponding to the checkpoint to continue processing the computing task.

[0004] However, this method cannot dynamically adjust the triggering of checkpoints in real time according to the actual working conditions of the GPU. Therefore, how to handle GPU failures more reasonably to improve the reliability and stability of the system is still a problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a GPU fault warning processing method and a computing device, which can reduce progress loss caused by GPU failure and improve the reliability and stability of the system.

[0006] In a first aspect, an embodiment of the present application provides a GPU fault warning processing method, which is applied to a first computing device, and the method includes: obtaining warning information of a target GPU; the warning information is determined based on working characteristic information of the target GPU in a computing node; according to the warning information, determining a checkpoint adjustment strategy; the checkpoint adjustment strategy is a strategy for adjusting the triggering frequency of checkpoints of a target GPU instance, the target GPU instance being a GPU instance in the target GPU that is running a computing task; according to the checkpoint adjustment strategy, adjusting the triggering frequency of checkpoints of the target GPU instance.

[0007] Beneficial effects of this implementation method: By obtaining the warning information of the target GPU, the embodiment of the present application can determine the potential failure risk in advance, so as to take preventive measures to reduce the probability of system crash or task failure. In addition, the trigger frequency of the checkpoint can be dynamically adjusted according to the warning information. When the potential failure risk is detected, the trigger frequency of the checkpoint can be increased to ensure that there are enough recovery points when a failure occurs, so that the task can be rolled back to a closer checkpoint after the failure, which can reduce the progress loss caused by GPU failure, and can also reduce the data loss and task interruption caused by GPU failure, improve the utilization efficiency of computing resources, and improve the overall performance and stability of the system.

[0008] In a possible implementation, the warning information includes a failure probability, and the failure probability indicates the probability of the target GPU failing within a target time period; determining a checkpoint adjustment strategy based on the warning information includes: determining the failure probability based on the warning information; determining, in the checkpoint adjustment strategy, a checkpoint adjustment strategy corresponding to the failure probability based on the failure probability; and the checkpoint adjustment strategy includes a trigger frequency of the adjusted checkpoint.

[0009] The beneficial effects of this implementation are as follows: by pre-saving multiple checkpoint adjustment strategies, the checkpoint adjustment strategy corresponding to the failure probability can be quickly determined, data processing efficiency is higher, flexibility is stronger, and the response speed to failure warnings can be improved.

[0010] In a possible implementation, the warning information includes a failure probability, and the failure probability indicates the probability of the target GPU failing within a target time period; determining the checkpoint adjustment strategy based on the warning information includes: determining the failure probability based on the warning information; determining the trigger frequency of the adjusted checkpoint based on the failure probability and a preset formula, to obtain the checkpoint adjustment strategy.

[0011] The beneficial effects of this implementation are as follows: by determining the checkpoint adjustment strategy by calculation, a corresponding checkpoint adjustment strategy can be formulated more accurately for different failure probabilities, which is more targeted and more applicable to the adjusted checkpoints, and can more effectively reduce the progress loss caused by GPU failures.

[0012] In a possible implementation, adjusting the triggering frequency of the checkpoint of the target GPU instance according to the checkpoint adjustment strategy includes: determining that the GPU instance in the target GPU that is running the computing task is the target GPU instance; based on the checkpoint adjustment strategy, constructing a new checkpoint task for the task instance corresponding to the target GPU instance, and starting the new checkpoint task.

[0013] The beneficial effects of this implementation are as follows: the checkpoint triggering frequency of the GPU instance running the computing task is dynamically and automatically adjusted according to the checkpoint adjustment strategy, which can respond to fault warnings more quickly, with higher targeting, reliability, resource utilization efficiency and automation.

[0014] In a second aspect, an embodiment of the present application provides a GPU fault warning processing method, which is applied to a second computing device, and the method includes: regularly obtaining working characteristic information of a target GPU; the working characteristic information is information related to the failure probability of the target GPU; calling a preset fault prediction model to perform prediction based on the working characteristic information to obtain the failure probability of the target GPU; based on the failure probability, outputting warning information; the warning information is used to adjust the trigger frequency of the checkpoint of the target GPU instance, and the target GPU instance is a GPU instance in the target GPU that is running a computing task.

[0015] The beneficial effects of this implementation are as follows: by predicting the failure probability of the target GPU to output early warning information, the checkpoint triggering frequency of the target GPU in the GPU instance running the computing task can be dynamically adjusted. When a potential failure risk is detected, the checkpoint triggering frequency can be increased to ensure that there are enough recovery points when a failure occurs, thereby reducing progress loss caused by GPU failure, data loss and task interruption caused by GPU failure, improving the utilization efficiency of computing resources, and improving the overall performance and stability of the system.

[0016] In a possible implementation, the operating characteristic information includes one or more of the core temperature, power consumption, number of correctable errors, fault records, operating time, maintenance date, and model of the target GPU.

[0017] Beneficial effects of this implementation: By comprehensively considering various working characteristic information of the target GPU, the accuracy of fault prediction can be improved.

[0018] In a possible implementation, the periodically acquiring the working characteristic information of the target GPU includes:

[0019] The working characteristic information of the target GPU is periodically queried in a chip for managing the GPU through a universal serial bus (USB) communication network to obtain the working characteristic information of the target GPU.

[0020] The beneficial effects of this implementation method are as follows: the USB communication network has high reliability and stability, which can ensure that packet loss or errors are not prone to occur during data transmission, thereby improving the reliability of obtaining working characteristic information. In addition, the USB communication network has low transmission delay and can quickly obtain working characteristic information.

[0021] In a possible implementation, outputting warning information based on the failure probability includes: determining whether the failure probability is greater than a warning threshold; if so, assembling the identification number of the target GPU and the failure probability into a simple network management protocol trap message to obtain warning information; and outputting the warning information through a user datagram protocol.

[0022] The beneficial effects of this implementation method are as follows: the Simple Network Management Protocol trap message is an active notification mechanism that can immediately send warning information when the failure probability is greater than a preset threshold, thereby improving response speed and real-time performance; in addition, the User Datagram Protocol is a connectionless transmission protocol with lower transmission overhead and higher transmission efficiency. Sending warning information through the User Datagram Protocol can reduce network burden and improve data transmission speed and efficiency.

[0023] In a possible implementation, the method further includes: updating the regularly acquired working characteristic information of the target GPU into a target database; and using the working characteristic information in the target database to retrain the preset fault prediction model.

[0024] The beneficial effects of this implementation are as follows: regularly retraining the model with new data can avoid model aging and obsolescence problems, and can also improve the accuracy of the model in predicting the probability of failure.

[0025] In a third aspect, an embodiment of the present application further provides a computing device, comprising a processor, wherein the processor is connected to a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement a method as described in any one of the first aspect, or to implement a method as described in any one of the second aspect.

[0026] Beneficial effects of this embodiment: When the processor executes the method described in any one of the first aspect or the method described in any one of the second aspect, the trigger frequency of the checkpoint is dynamically adjusted through early warning information, and the trigger frequency of the checkpoint can be increased during high failure risk periods, so that the task can be rolled back to a closer checkpoint after a failure, which can reduce progress loss caused by GPU failure, and can also reduce data loss and task interruption caused by GPU failure, improve the utilization efficiency of computing resources, and improve the overall performance and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 A schematic diagram of a system architecture for GPU fault warning processing provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of progress loss provided in an embodiment of the present application;

[0030] Figure 3 A flowchart of a GPU fault warning processing method provided in an embodiment of the present application;

[0031] Figure 4 An example diagram of the distribution of checkpoint frequencies in a checkpoint adjustment strategy provided in an embodiment of the present application;

[0032] Figure 5 A diagram showing an example of the distribution of checkpoint frequencies in another checkpoint adjustment strategy provided in an embodiment of the present application;

[0033] Figure 6 Another schematic diagram of progress loss provided in an embodiment of the present application;

[0034] Figure 7 A flowchart of another GPU fault warning processing method provided in an embodiment of the present application;

[0035] Figure 8 A schematic diagram of the structure of a GPU fault warning processing device provided in an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of the structure of another GPU fault warning processing device provided in an embodiment of the present application;

[0037] Fig.10 A schematic diagram of the structure of a computing device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments made by ordinary technicians in this field under the enlightenment of the embodiments belong to the scope of protection of the present application.

[0039] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] Figure 1 A schematic diagram of a system architecture for GPU fault warning processing provided in an embodiment of the present application. Figure 1 As shown, in the embodiment of the present application, the system for GPU fault warning processing includes a computing node and a task management device. Among them, the task management device is a device for scheduling and managing computing tasks, and the task management device includes at least a processor and other components. The processor can be a central processing unit (CPU) or other devices with processing capabilities. Among them, the computing node includes at least one physical GPU, which is used to execute the computing tasks issued by the task management device.

[0041] GPU has been widely used in artificial intelligence, high-performance computing and other fields due to its powerful parallel computing capabilities. In practical applications, a single physical GPU resource can be divided into multiple independent virtual GPUs through virtualization technology. Virtual GPUs are also called GPU instances. Each virtual GPU can be assigned to different users or tasks. This virtualization technology allows multiple users to share the same physical GPU while ensuring that each user's tasks do not interfere with each other. In an embodiment of the present application, the task scheduler of the application layer in the task management device can receive the computing tasks issued by the user, and distribute the computing tasks to several virtual GPU instances of the computing node according to the resource situation, and the GPU instance can also perform computing tasks based on the computing power resources of the physical GPU. These large-scale computing tasks usually have a long running time. A single computing task may take several hours or even days. During the task running process, the task may be interrupted or failed due to various system failures. At this time, the progress of the computing task will be lost.

[0042] According to data from relevant deep learning platforms, about 28% of system failures are GPU hardware failures. Especially under the dual influence of the increase in single GPU computing units and the increase in the number of computing cluster nodes, the probability of GPU failure is greater. Therefore, it is very important to study how to effectively minimize the impact of GPU failures on computing tasks.

[0043] In related technologies, the main method is to avoid the loss of all task progress after a single failure by periodically triggering checkpoints. A checkpoint can be defined as a specific time point or data snapshot that saves the current state during the calculation process. Specifically, a checkpoint can be used to record the intermediate state of a GPU instance when running a computing task, including memory content, register status, process status, etc. This record allows the system to restore the computing task from the most recent checkpoint when a failure occurs, rather than recalculating from scratch, thereby reducing computing time and resource waste.

[0044] For example, Figure 2 A schematic diagram of progress loss provided in an embodiment of the present application. Figure 2 As shown, in the periodic checkpoint triggering method, the intervals between checkpoints 1 to 3 are the same. When a failure occurs during the processing of the computing task between checkpoint 2 and checkpoint 3, the progress between checkpoint 2 and the failure point will be lost.

[0045] Understandably, the performance of saving checkpoints is limited by the elasticity characteristics of the entire system, such as cluster bandwidth, hard disk speed, etc. The smaller the cluster bandwidth and the lower the hard disk speed, the more system resources are occupied in the process of saving task files, and the greater the progress loss; moreover, under this static checkpoint mechanism, when a failure occurs, the task progress between the failure point and the checkpoint will inevitably still be lost. Therefore, the setting of checkpoints cannot be too dense or too sparse, and how to effectively reduce progress loss has become a problem that needs to be solved urgently.

[0046] To this end, the embodiment of the present application provides a GPU fault warning processing method, the scheme is as follows: the embodiment of the present application adopts a partial dynamic checkpoint mechanism to dynamically adjust the checkpoint triggering frequency according to the probability of future failures; specifically, a second computing device (e.g. Figure 1 The computing node in the target GPU periodically obtains the working characteristic information of the target GPU, calls the preset fault prediction model to predict the fault probability based on the working characteristic information, and then outputs a warning message according to the magnitude of the fault probability; and the first computing device (e.g. Figure 1The task management device in the target GPU can obtain the warning information of the target GPU and determine the checkpoint adjustment strategy based on the warning message, thereby dynamically adjusting the checkpoint trigger frequency. Through the dynamic checkpoint mechanism, the checkpoint trigger frequency is adjusted in real time according to the warning information of the target GPU, which can reduce the progress loss caused by GPU failure, thereby improving the computing resource utilization efficiency in the high-performance computing cluster.

[0047] Combination Figure 1 In the computing node, there can be a baseboard management controller (Baseboard Management Controller, BMC) for managing and processing node device information. The BMC can obtain the working characteristic information of the GPU through a dedicated network channel, and call a preset fault prediction model to predict based on the working characteristic information to obtain the failure probability of the target GPU. When it is determined that the failure probability of the GPU is greater than the preset threshold, the warning information is reported to the task management device through the dedicated network channel so that the task management device can adjust the trigger frequency of the checkpoint. Optionally, a task management system and a device management system can also be run in the task management device, and the task management system and the device management system can be integrated into a high-performance computing (High-Performance Computing, HPC) platform. Exemplarily, after the device management system in the task management device receives the warning information sent by the BMC of the computing node, it can determine the checkpoint adjustment strategy according to the warning information, and then the task management system in the task management device adjusts the trigger frequency of the checkpoint according to the checkpoint adjustment strategy. That is to say, the task of saving checkpoints can be managed by the task management system in the task management device. After adjusting the trigger frequency of the checkpoints, the task management system can send the task execution command to the computing node corresponding to the computing task through the network, and then the corresponding operating system in the computing node will perform the specific operation of saving checkpoints.

[0048] It should be noted that Figure 1 The computing node shown may also include other necessary electronic components such as a central processing unit (CPU), etc. Similarly, the task management device may also include other devices, which is not limited in the embodiments of the present application.

[0049] Figure 3 A flowchart of a GPU fault warning processing method provided in an embodiment of the present application. Figure 3 The GPU fault warning processing method shown can be applied to a first computing device, wherein the first computing device can be Figure 1 The task management device in the embodiment of the present application may be a server, etc. Figure 1The task management device in the task management device, more specifically, can be a processor in the task management device. The following is a detailed description using the execution subject as the task management device as an example. Figure 3 , which may specifically include the following steps:

[0050] S301, obtaining warning information of a target GPU.

[0051] The warning information is determined based on the working characteristic information of the target GPU in the computing node. The working characteristic information is information related to the failure probability of the target GPU. For example, the working characteristic information may include one or more of the core temperature, power consumption, number of correctable errors, failure records, operating time, maintenance date, model, etc. of the target GPU. Based on the working characteristic information of the target GPU, the failure probability of the target GPU can be determined, thereby obtaining the warning information.

[0052] It is understandable that during the operation of the target GPU, some working characteristic information of the target GPU, such as core temperature, power consumption, number of correctable errors, operating time, etc., will continue to change, and the failure probability of the target GPU will also continue to change accordingly. Therefore, in this application, the task management device needs to continuously obtain the early warning information of the target GPU in order to adjust the checkpoint trigger frequency according to the actual early warning information.

[0053] For example, in combination Figure 1 In the embodiment of the present application, the BMC in the computing node can output warning information based on the failure probability of the target GPU. For example, when the BMC in the computing node determines that the failure probability of the target GPU in the target time period is greater than a preset threshold, it will send the warning information of the target GPU to the task management device. Correspondingly, the task management device can receive the warning information of the target GPU sent by the BMC. The process of how the BMC in the computing node determines and outputs the warning information based on the working characteristic information of the target GPU is introduced in the subsequent embodiments and will not be repeated here.

[0054] S302: Determine a checkpoint adjustment strategy based on the warning information.

[0055] The checkpoint adjustment strategy is a strategy for adjusting the triggering frequency of the checkpoint of the target GPU instance, and the target GPU instance is a GPU instance in the target GPU that is running the computing task.

[0056] It can be understood that the target GPU is a physical GPU. After the target GPU is divided into multiple independent virtual GPUs through virtualization technology and multiple GPU instances are obtained, these GPU instances will be allocated to perform different computing tasks according to actual needs. The GPU instance that is allocated to the running computing task is the target GPU instance. For example, the resources of the target GPU are divided into three GPU instances A, B, and C. At this time, only the GPU instance with serial number A is running the computing task, and the GPU instances with serial numbers B and C are idle. In this case, only the GPU instance with serial number A is the target GPU instance.

[0057] Exemplarily, the task management device can determine a new checkpoint adjustment strategy based on the acquired warning information, so as to adjust the checkpoint of the computing task running in the target GPU with the new checkpoint adjustment strategy. Among them, the checkpoint adjustment strategy should at least include the trigger frequency of the adjusted checkpoint, so that the task management device can adjust the checkpoint trigger frequency according to the checkpoint adjustment strategy. The checkpoint adjustment strategy in the embodiment of the present application should meet the following requirements: reduce unnecessary checkpoint save operations when the GPU is running normally, thereby saving computing resources and storage space, and increase the checkpoint trigger frequency when a potential failure risk is detected to ensure that there are enough recovery points when a failure occurs to reduce progress loss.

[0058] In a possible embodiment, the early warning information includes a failure probability, and the failure probability indicates the probability of the target GPU failing within a target time period. During the operation of the target GPU, some working characteristic information of the target GPU, such as core temperature, power consumption, number of correctable errors, running time, etc., will continue to change. Therefore, the failure probability determined based on the working characteristic information of the target GPU will also continue to change. The determined failure probability can only be used to characterize the probability of the target GPU failing within a period of time. Optionally, the target time period can be the length of time that the working characteristic information of the target GPU affects the target GPU. For example, if the working characteristic information of the target GPU at the current moment will only affect the performance of the target GPU within the next hour at most, the target time period can be set to 1h, and the failure probability is only used to indicate the probability of the target GPU failing within the next hour. Optionally, the target time period can also be any time length set by the user according to actual needs, and the embodiment of the present application is not limited.

[0059] Optionally, determining a checkpoint adjustment strategy based on the warning information may include:

[0060] S1. Determine the failure probability based on the early warning information.

[0061] S2. According to the failure probability, in the checkpoint adjustment strategy, determine the checkpoint adjustment strategy corresponding to the failure probability, wherein the checkpoint adjustment strategy includes the adjusted checkpoint triggering frequency.

[0062] Exemplarily, a plurality of checkpoint adjustment strategies corresponding to different failure probabilities may be pre-saved in the checkpoint adjustment strategy file of the task management device, and each checkpoint adjustment strategy includes the corresponding adjusted checkpoint trigger frequency. Then, the task management device may parse the early warning information to obtain the probability of failure of the target GPU within the target time period, and then search the pre-saved checkpoint adjustment strategy file according to the failure probability to determine the checkpoint adjustment strategy corresponding to the failure probability as the strategy for adjusting the checkpoint this time. It can be understood that based on the adjusted checkpoint trigger frequency, the number of additional checkpoints that should be added within the target time interval and the interval between each additional checkpoint can be known, and then the task management device can adjust the checkpoint trigger frequency.

[0063] Optionally, in another possible embodiment, only one checkpoint adjustment strategy may be pre-saved in the checkpoint adjustment strategy file. In this case, the warning information may not include any additional warning description information. The warning information is directly used to indicate whether a checkpoint adjustment is required. The warning has only two states: on and off. The checkpoint adjustment strategy only needs to include the checkpoint intervals when the warning is triggered and not triggered. The checkpoint interval when the warning is not triggered is the original preset checkpoint interval, and the checkpoint interval when the warning is triggered is the adjusted checkpoint interval.

[0064] For example, Figure 4 This is an example diagram of the distribution of checkpoint frequencies in a checkpoint adjustment strategy provided in an embodiment of the present application. Figure 4 As shown in the figure, taking the preset threshold of the failure probability as 0.5 as an example, the trigger frequency of the checkpoint when no warning is triggered is 1. When the failure probability is greater than 0.5, the warning is triggered and the trigger frequency of the checkpoint becomes 2. By increasing the trigger frequency of the checkpoint, the progress loss caused by GPU failure can be reduced.

[0065] In another possible embodiment, the early warning information includes a failure probability, and determining the checkpoint adjustment strategy according to the early warning information may further include:

[0066] S10. Determine the failure probability based on the early warning information.

[0067] S20. Determine the trigger frequency of the adjusted checkpoint according to the failure probability and a preset formula to obtain a checkpoint adjustment strategy.

[0068] Exemplarily, the trigger frequency of the checkpoint can also be calculated according to a preset formula. After parsing the warning information and determining the failure probability, the task management device can bring the failure probability into the preset formula for calculating the trigger frequency of the adjusted checkpoint to calculate the adjusted checkpoint trigger frequency, thereby obtaining the checkpoint adjustment strategy.

[0069] The present application embodiment does not limit the preset formula, and the higher the probability of failure, the higher the calculated checkpoint frequency. Figure 5 This is another example diagram of the distribution of checkpoint frequencies in a checkpoint adjustment strategy provided in an embodiment of the present application. Figure 5 As shown in the figure, taking the example of requiring an early warning when the probability of failure is greater than 0.3, when the probability of failure is greater than 0.3, as the probability of failure increases, the triggering frequency of the checkpoint increases accordingly. Figure 5 , the preset formula may be y=20 / 7x+1 / 7. It should be noted that the preset formula may also be other formulas. More checkpoints may be added during a time period with a higher probability of failure, and fewer checkpoints may be added during a time period with a lower probability of failure, thereby more effectively reducing progress loss.

[0070] S303: Adjust the checkpoint triggering frequency of the target GPU instance according to the checkpoint adjustment strategy.

[0071] Exemplarily, after determining the checkpoint adjustment policy, the task management device may adjust the checkpoint triggering frequency of the target GPU instance based on the checkpoint adjustment policy.

[0072] In a possible embodiment, adjusting the checkpoint triggering frequency of the target GPU instance according to the checkpoint adjustment policy may include:

[0073] S100: Determine a GPU instance in a target GPU that is running a computing task as a target GPU instance.

[0074] S200. Based on the checkpoint adjustment strategy, construct a new checkpoint task for the task instance corresponding to the target GPU instance, and start the new checkpoint task.

[0075] Exemplarily, a single physical GPU can virtualize multiple GPU instances through virtualization technology. In practical applications, computing tasks may be assigned to one or more GPU instances for operation. When adjusting the checkpoint of the target GPU, only the GPU instance running the computing task can be adjusted to reduce the amount of data processing. Therefore, the task management device can first determine the GPU instance running the computing task in the target GPU, and then build a new checkpoint task for the task instance corresponding to the target GPU instance based on the checkpoint adjustment strategy. Among them, the task instance corresponding to the target GPU instance is a task instance for executing a new checkpoint task. It can be understood that in the embodiment of the present application, the task management device, as a device for managing computing tasks, can naturally determine the GPU instance running the computing task in the target GPU. Then, the task management device uses the instance name of the task instance to search for the old checkpoint task of the task instance in the task pool, stop the old checkpoint task, and start the new checkpoint task. Then, the corresponding file will be saved when the new checkpoint is executed. In this way, by adjusting the checkpoint of the GPU instance running the computing task of the target GPU, the progress loss can be effectively reduced.

[0076] Exemplarily, in an embodiment of the present application, snmptrapd and Slurm work scheduling tools can be deployed in a task management device. SNMP is the full name of Simple Network Management Protocol in English, and SNMP trap is a message format in which a managed device actively sends device events to a manager on SNMP. In an embodiment of the present application, it can be used for BMC to send early warning information to a task management device. The snmptrapd service under the Linux system can receive SNMP trap messages as a manager and execute corresponding command line programs. Therefore, after snmptrapd is deployed in the task management device, the task management device can monitor and receive SNMP trap messages from BMC through snmptrapd to obtain early warning information.

[0077] The Slurm job scheduling tool can complete the submission and modification of tasks through a series of command line operations. In the embodiment of the present application, the operation of triggering the checkpoint can also exist in the form of a slurm task, and a corresponding slurm instance is abstracted for each GPU instance using a configuration file. The instance name of the slurm instance can be set to be determined by the IP address of the BMC and the GPU instance number. For example, <ip> . <gid>The instance name of each slurm instance can be determined by using a format or hashing algorithm.

[0078] After receiving the warning information of the target GPU sent by the BMC, the task management device can not only parse out the identification number and failure probability of the target GPU, but also parse out the IP address of the BMC. Therefore, after determining the GPU instance running the computing task in the target GPU, the task management device can determine the instance name of the slurm instance corresponding to the GPU instance running the computing task based on the instance number of the GPU instance running the computing task and the IP address of the BMC. The instance name of the slurm instance can be used to search for the old checkpoint task of the task instance in the task pool, so that the old checkpoint task can be stopped and a new checkpoint task can be started.

[0079] It can be understood that, based on the checkpoint adjustment strategy, when constructing a new checkpoint task for the task instance corresponding to the GPU instance running the computing task, it can involve all the checkpoint tasks of the subsequent unfinished computing tasks, or only the checkpoint tasks of the target time interval. When only the checkpoint tasks of the target time interval are involved, a timer can also be executed in the task management device so that the new checkpoint task continues to execute the original checkpoint task after the target time interval is completed.

[0080] For example, Figure 6 Another schematic diagram of progress loss provided in the embodiment of the present application. Figure 6 As shown, based on the dynamic checkpoint mechanism of the embodiment of the present application, after adding an additional checkpoint on the basis of the original checkpoints 1 to 3, the progress of the loss becomes the progress between the additional checkpoint and the failure point. Figure 1 The lost progress shown can be effectively reduced by using the solution of the embodiment of the present application.

[0081] The GPU fault warning processing method provided by the embodiment of the present application is applied to a first computing device, including: obtaining warning information of a target GPU; the warning information is determined based on the working characteristic information of the target GPU in the computing node; according to the warning information, determining a checkpoint adjustment strategy, wherein the checkpoint adjustment strategy is a strategy for adjusting the trigger frequency of the checkpoint of the target GPU instance, and the target GPU instance is a GPU instance in the target GPU that is running a computing task; according to the checkpoint adjustment strategy, adjusting the trigger frequency of the checkpoint of the target GPU instance. Compared with the related art, the embodiment of the present application can determine the potential fault risk in advance by obtaining the warning information of the target GPU, so as to take preventive measures to reduce the probability of system crash or task failure. In addition, the trigger frequency of the checkpoint is dynamically adjusted according to the warning information, and the trigger frequency of the checkpoint can be increased when the potential fault risk is detected to ensure that there are enough recovery points when the fault occurs, so that it can fall back to a closer checkpoint after the fault, so as to reduce the progress loss caused by the GPU fault, and also reduce the data loss and task interruption caused by the GPU fault, improve the utilization efficiency of computing resources, and also improve the overall performance and stability of the system.

[0082] Figure 7 A flowchart of another GPU fault warning processing method provided in an embodiment of the present application. Figure 7 The GPU fault warning processing method shown is applied to a second computing device, wherein the second computing device can be Figure 1 The computing nodes in the , computing nodes can be servers, etc. Figure 1 , the execution subject of the embodiment of the present application may be Figure 1 The computing node in the computing node, more specifically, may be a baseboard management controller BMC in the computing node. The following is a detailed description using the baseboard management controller BMC as an example. Figure 7 , which may specifically include the following steps:

[0083] S701. Regularly obtain working characteristic information of a target GPU.

[0084] The working characteristic information is information related to the failure probability of the target GPU.

[0085] Exemplarily, the BMC may periodically obtain the working characteristic information of the target GPU at regular intervals, and then determine whether to trigger an early warning.

[0086] In a possible embodiment, periodically acquiring the operating characteristic information of the target GPU may include: periodically querying the operating characteristic information of the target GPU in a chip for managing the GPU through a universal serial bus (USB) communication network to obtain the operating characteristic information of the target GPU.

[0087] Exemplarily, in some computing devices, in order to better complete data processing, a chip for managing GPUs will be deployed in the device to process management requests from the BMC. For example, in some GPU modules commonly used in high-performance computing nodes, there are not only multiple GPU cores inside, but also chips for managing these multiple GPUs, where the chip for managing GPUs can also be a BMC chip. The BMC of the embodiment of the present application can establish a communication connection with the chip for managing GPUs through a Universal Serial Bus (USB) communication network, and then query the working feature information of the target GPU in the chip for managing GPUs.

[0088] Among them, the working characteristic information of the target GPU is used to predict the failure probability of the target GPU in the target time interval. In a possible embodiment, the working characteristic information may include one or more of the core temperature, power consumption, number of correctable errors, fault records, operating time, maintenance date, model, etc. of the target GPU. The higher the core temperature of the target GPU, the higher the probability of failure. Similarly, the greater the power consumption, the greater the number of correctable errors, the more fault records, the longer the operating time, the longer the maintenance date, and the older the model, the more likely the GPU will fail. In order to improve the prediction accuracy of the failure probability, the working characteristic information of the target GPU may include multiple of the above.

[0089] Optionally, in some other embodiments, the information related to the failure probability of the target GPU may also include other information, such as real-time data of the power supply voltage and current of the computing node, the speed of the cooling fan on the computing node, etc. This information may also affect the accuracy of the failure probability of the target GPU. The working characteristic information of the target GPU may also include this information, which is not limited to the embodiments of the present application.

[0090] S702: Call a preset fault prediction model to perform prediction based on the working characteristic information to obtain the fault probability of the target GPU.

[0091] Exemplarily, after obtaining the working characteristic information of the target GPU, the BMC can call a preset fault prediction model and input the working characteristic information of the target GPU into the preset fault prediction model for prediction. The preset fault prediction model can output a probability value of 0-1, which is used to indicate the probability of the target GPU failing in the target time interval.

[0092] The preset fault prediction model is a pre-trained model for predicting the failure probability of a GPU based on the working characteristic information of the GPU, for example, it may be a Gradient Boosting Decision Tree (GBDT) model.

[0093] In a possible embodiment, the model can be trained in advance and the model file of the preset fault prediction model can be exported and saved in the flash memory of the BMC. Before calling the preset fault prediction model to make a prediction based on the working characteristic information, the model file of the preset fault prediction model is first read from the flash memory so that the preset fault prediction model is saved in the target software library, and then the fault probability is predicted.

[0094] Exemplarily, if the BMC supports the xgboost software library, the target software library can be the Xgboost software library, which can provide a multi-language API for training and reasoning GBDT models. Before training the model, you can prepare multiple sets of GPU working feature information and fault time points in advance, then write a program, use the API of the xgboost software library to define the GBDT model, set the number of iterations, maximum tree depth and other model parameters, and then read the working feature information of multiple sets of GPUs, call the API for training until the number of iterations or the maximum tree depth is reached, and obtain the trained model, and then save the trained model as a model file and deploy it to the flash memory of the BMC. When the BMC starts, the model file of the GBDT model is read from the flash memory and saved in the xgboost software library, so that the preset fault prediction model can be called later to make predictions based on the working feature information and obtain the failure probability of the GPU.

[0095] In a possible embodiment, in order to improve the prediction accuracy of the preset fault prediction model, the regularly acquired working characteristic information of the target GPU can also be updated to the target database for retraining the preset fault prediction model. The target database can be a remote lightweight database such as SQLite, and maintenance personnel can regularly / irregularly use new data to retrain the model to avoid aging and obsolescence of the model.

[0096] S703: Output warning information based on the failure probability.

[0097] The warning information is used to adjust the trigger frequency of the checkpoint of the target GPU instance. The target GPU instance is the GPU instance in the target GPU that is running the computing task. For details on how to adjust the trigger frequency of the checkpoint of the target GPU instance based on the warning information, please refer to Figure 3 The description of the illustrated embodiment will not be repeated here.

[0098] Exemplarily, after determining the failure probability of the target GPU, the BMC may determine and output corresponding warning information according to the magnitude of the failure probability. For example, when it is determined that the failure probability is greater than the warning threshold, the BMC outputs warning information for instructing the task management device to adjust the checkpoint trigger frequency; when it is determined that the failure probability is less than the warning threshold, the BMC outputs warning information for instructing the task management device to maintain the original checkpoint trigger frequency.

[0099] In a possible embodiment, outputting warning information based on the failure probability may include:

[0100] S01. Determine whether the failure probability is greater than a warning threshold.

[0101] Exemplarily, whether to trigger an early warning can be determined based on the size of the failure probability. If the BMC determines that the failure probability is greater than the early warning threshold, step S02 can be executed. If the BMC determines that the failure probability is less than or equal to the early warning threshold, step S701 can be returned to execute. The embodiment of the present application does not limit the size of the preset threshold.

[0102] Exemplarily, when the BMC determines that the failure probability is greater than the warning threshold, it may generate and send warning information to the task management device so that the task management device adjusts the checkpoint triggering frequency of the target GPU instance according to the warning information.

[0103] S02. Assemble the identification number and failure probability of the target GPU into a simple network management protocol trap message to obtain early warning information.

[0104] S03. Output warning information via the User Datagram Protocol.

[0105] Exemplarily, in order to facilitate the task management device to adjust the checkpoint of the target GPU instance according to the early warning information, the identification number of the target GPU and the failure probability as the early warning description information can be included in the early warning information. The SNMP trap message can carry several custom binding variables (binding). When the SNMP trap is used as the early warning method, the BMC can assemble the identification number and the failure probability of the target GPU into the SNMP trap message to obtain the early warning information, and then send the early warning information to the task management device through the User Datagram Protocol (UDP). In the embodiment of the present application, after snmptrapd is deployed in the task management device, the task management device can monitor and receive the SNMP trap message from the BMC through snmptrapd to obtain the early warning information, so that the checkpoint adjustment strategy can be determined according to the early warning information, and the trigger frequency of the checkpoint of the target GPU instance can be adjusted according to the checkpoint adjustment strategy.

[0106] It can be understood that the loss time = the time lost between the fault point and the checkpoint when the GPU fails + the time lost in the checkpoint saving operation; efficiency improvement = (related technology loss time - improved technology loss time) / related technology loss time. When the accuracy and recall rate of the preset fault prediction model are higher, the task management device can adjust the checkpoint trigger frequency to more accurately reduce the progress loss.

[0107] For example, if the prior probability of GPU failure is 0.001% / min, the model runs once every 720 minutes, the prediction window (that is, the target time interval) is 720 minutes, and saving the checkpoint will lose 1 minute. If a checkpoint is saved every fixed time T=360 minutes, when the failure probability is higher than the preset threshold such as 0.5, taking the additional checkpoint interval as t as an example, the fault prediction model with different precision and recall rates is simulated 10,000 times, and the data in Tables 1 and 2 can be obtained.

[0108] Table 1

[0109]

[0110] Table 2

[0111]

[0112]

[0113] Combining Table 1 and Table 2, it can be seen that if the precision and recall are high, that is, when the prediction results of the fault prediction model are more accurate, more checkpoints can be added and the intervals between checkpoints can be reduced to effectively reduce the loss time; if the precision and recall are low, fewer checkpoints should be added to avoid losing too much time due to the checkpoint saving operation.

[0114] The GPU fault warning processing method provided in the embodiment of the present application is applied to the second computing device, and the method includes: regularly obtaining the working characteristic information of the target GPU; the working characteristic information is information related to the failure probability of the target GPU; calling a preset fault prediction model to predict based on the working characteristic information to obtain the failure probability of the target GPU; outputting warning information based on the failure probability; the warning information is used to adjust the trigger frequency of the checkpoint of the target GPU instance, and the target GPU instance is the GPU instance in the target GPU that is running the computing task. By predicting the failure probability of the target GPU and adjusting the trigger frequency of the checkpoint of the target GPU instance, the progress loss caused by GPU failure can be reduced, and the data loss and task interruption caused by GPU failure can also be reduced, the utilization efficiency of computing resources can be improved, and the overall performance and stability of the system can also be improved.

[0115] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0116] Figure 8 Schematic diagram of a GPU fault warning processing device provided in an embodiment of the present application. The GPU fault warning processing device 80 provided in an embodiment of the present application is applied to a first computing device, such as Figure 8 As shown, the GPU fault warning processing device 80 provided in the embodiment of the present application includes an acquisition unit 801, a determination unit 802 and an adjustment unit 803.

[0117] The acquisition unit 801 is used to acquire the warning information of the target GPU, wherein the warning information is determined based on the working characteristic information of the target GPU in the computing node.

[0118] The determination unit 802 is used to determine a checkpoint adjustment strategy according to the warning information, wherein the checkpoint adjustment strategy is a strategy for adjusting the triggering frequency of the checkpoint of the target GPU instance, and the target GPU instance is a GPU instance in the target GPU that is running the computing task.

[0119] The adjustment unit 803 is used to adjust the checkpoint triggering frequency of the target GPU instance according to the checkpoint adjustment strategy.

[0120] In a possible implementation, the warning information includes a failure probability, where the failure probability indicates a probability that a target GPU fails within a target time period; the determining unit 802 is specifically configured to:

[0121] Determine the probability of failure based on early warning information;

[0122] According to the failure probability, in the checkpoint adjustment strategy, a checkpoint adjustment strategy corresponding to the failure probability is determined; the checkpoint adjustment strategy includes a triggering frequency of the adjusted checkpoint.

[0123] In a possible implementation, the warning information includes a failure probability, where the failure probability indicates a probability that a target GPU fails within a target time period; the determining unit 802 is specifically configured to:

[0124] Determine the probability of failure based on early warning information;

[0125] According to the failure probability and the preset formula, the triggering frequency of the adjusted checkpoint is determined to obtain the checkpoint adjustment strategy.

[0126] In a possible implementation, the adjusting unit 803 is specifically configured to:

[0127] Determine the GPU instance that is running the computing task in the target GPU as the target GPU instance;

[0128] Based on the checkpoint adjustment strategy, a new checkpoint task is constructed for the task instance corresponding to the target GPU instance, and the new checkpoint task is started.

[0129] The device provided in this embodiment can be used to perform the above Figure 3 The implementation principle and technical effect of the method in the illustrated embodiment are similar and will not be described in detail here.

[0130] Fig. 9 Schematic diagram of another GPU fault warning processing device provided in an embodiment of the present application. The GPU fault warning processing device 90 provided in an embodiment of the present application is applied to a second computing device, such as Fig. 9 As shown, the GPU fault warning processing device 90 provided in the embodiment of the present application includes an acquisition unit 901, a prediction unit 902 and a processing unit 903.

[0131] The acquisition unit 901 is used to periodically acquire the working characteristic information of the target GPU; the working characteristic information is information related to the failure probability of the target GPU.

[0132] The prediction unit 902 is used to call a preset fault prediction model to perform prediction based on the working characteristic information to obtain the fault probability of the target GPU.

[0133] The processing unit 903 is used to output warning information based on the failure probability, wherein the warning information is used to adjust the trigger frequency of the checkpoint of the target GPU instance, and the target GPU instance is the GPU instance in the target GPU that is running the computing task.

[0134] In a possible implementation, the operating characteristic information includes one or more of the core temperature, power consumption, number of correctable errors, fault records, operating time, maintenance date, and model of the target GPU.

[0135] In a possible implementation, the acquiring unit 901 is specifically configured to:

[0136] The working characteristic information of the target GPU is periodically queried in a chip for managing the GPU through a universal serial bus (USB) communication network to obtain the working characteristic information of the target GPU.

[0137] In a possible implementation, the processing unit 903 is specifically configured to:

[0138] Determine whether the failure probability is greater than the warning threshold;

[0139] If yes, the identification number and failure probability of the target GPU are assembled into a simple network management protocol trap message to obtain warning information;

[0140] Output warning information via User Datagram Protocol.

[0141] In a possible implementation, the acquiring unit 901 is further configured to:

[0142] The regularly acquired working characteristic information of the target GPU is updated into the target database; the working characteristic information in the target database is used to retrain the preset fault prediction model.

[0143] The device provided in this embodiment can be used to perform the above Figure 7 The implementation principle and technical effect of the method in the embodiment shown are similar and will not be described in detail here.

[0144] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, it can also be stored in the memory of the above device in the form of program code, and the functions of the above data processing modules can be called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0145] Fig.10 A schematic diagram of the structure of a computing device is provided for an embodiment of the present application. Fig.10 As shown, the computing device 100 includes a processor 1001 , and the processor 1001 is connected to a memory 1002 .

[0146] The memory 1002 is used to store computer-executable instructions. The memory 1002 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0147] The processor 1001 is used to execute the computer-executable instructions stored in the memory 1002 to implement the network configuration method described in the aforementioned method embodiment. The processor 1001 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0148] The computing device 100 may further include a communication interface 1003, so as to communicate and interact with an external device through the communication interface 1003. The external device may be, for example, a server or other computing devices.

[0149] In specific implementation, if the communication interface 1003, the memory 1002 and the processor 1001 are implemented independently, the communication interface 1003, the memory 1002 and the processor 1001 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0150] Optionally, in a specific implementation, if the communication interface 1003, the memory 1002 and the processor 1001 are integrated on a chip, the communication interface 1003, the memory 1002 and the processor 1001 can communicate through an internal interface.

[0151] The computing device provided in this embodiment may be a task management device or a baseboard management controller BMC or other devices in the above embodiments, which can execute the above Figure 3 or Figure 7 The technical solutions in the method embodiments shown have similar beneficial effects and will not be described in detail here.

[0152] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the technical solution provided by any of the aforementioned method embodiments.

[0153] An embodiment of the present application also provides a computer program product, including a computer program, which is used to implement the technical solution provided by the aforementioned method embodiment when executed by a processor.

[0154] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, rather than to limit them. Although the embodiments of the present application have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / gid> < / ip>

Claims

1. A GPU fault early warning processing method, applied to a first computing device, characterized in that: The method comprises: Acquire warning information of a target GPU; the warning information is determined based on working characteristic information of the target GPU in a computing node; Determine a checkpoint adjustment strategy according to the warning information; the checkpoint adjustment strategy is a strategy for adjusting the triggering frequency of the checkpoint of the target GPU instance, and the target GPU instance is a GPU instance in the target GPU that is running the computing task; According to the checkpoint adjustment policy, the checkpoint triggering frequency of the target GPU instance is adjusted.

2. The method according to claim 1, characterized in that The early warning information includes a failure probability, where the failure probability indicates a probability that the target GPU fails within a target time period; Determining a checkpoint adjustment strategy according to the early warning information includes: Determining a failure probability based on the warning information; According to the failure probability, in the checkpoint adjustment strategy, a checkpoint adjustment strategy corresponding to the failure probability is determined; the checkpoint adjustment strategy includes a triggering frequency of the adjusted checkpoint.

3. The method according to claim 1, characterized in that: The early warning information includes a failure probability, where the failure probability indicates a probability that the target GPU fails within a target time period; Determining a checkpoint adjustment strategy according to the early warning information includes: Determining a failure probability based on the warning information; According to the failure probability and a preset formula, the triggering frequency of the adjusted checkpoint is determined to obtain the checkpoint adjustment strategy.

4. The method according to any one of claims 1 to 3, characterized in that The step of adjusting the checkpoint triggering frequency of the target GPU instance according to the checkpoint adjustment strategy includes: Determine a GPU instance in the target GPU that is running a computing task as a target GPU instance; Based on the checkpoint adjustment strategy, a new checkpoint task is constructed for the task instance corresponding to the target GPU instance, and the new checkpoint task is started.

5. A GPU fault warning processing method, applied to a second computing device, characterized in that: The method comprises: Periodically acquiring operating characteristic information of a target GPU; the operating characteristic information being information related to a failure probability of the target GPU; Calling a preset fault prediction model to perform prediction based on the working characteristic information to obtain a fault probability of the target GPU; Based on the failure probability, output warning information; the warning information is used to adjust the triggering frequency of the checkpoint of the target GPU instance, and the target GPU instance is a GPU instance in the target GPU that is running a computing task.

6. The method according to claim 5, characterized in that The working characteristic information includes one or more of the core temperature, power consumption, number of correctable errors, fault records, operating time, maintenance date, and model of the target GPU.

7. The method according to claim 5, characterized in that The periodically obtaining the working characteristic information of the target GPU includes: The working characteristic information of the target GPU is periodically queried in a chip for managing the GPU through a universal serial bus (USB) communication network to obtain the working characteristic information of the target GPU.

8. The method according to claim 5, characterized in that The outputting of warning information based on the failure probability includes: Determining whether the failure probability is greater than a warning threshold; If yes, assembling the identification number of the target GPU and the failure probability into a simple network management protocol trap message to obtain early warning information; The warning information is output via the User Datagram Protocol.

9. The method according to any one of claims 5 to 8, characterized in that: The method further comprises: The regularly acquired working characteristic information of the target GPU is updated into the target database; the working characteristic information in the target database is used to retrain the preset fault prediction model.

10. A computing device, characterized in that The computing device includes a processor connected to a memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 4, or to implement the method according to any one of claims 5 to 9.

Images