Method and device for predicting failure of GPU, electronic equipment and storage medium

Abstract

The present disclosure relates to a method and device for predicting a faulty GPU, an electronic device and a storage medium. The method comprises obtaining parameter information of each GPU in a plurality of GPUs to obtain a parameter information set; inputting the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model; and determining a faulty GPU from the plurality of GPUs according to the prediction result. The present disclosure can improve the prediction accuracy of the faulty GPU.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, and in particular, to a method and device for predicting a faulty GPU, an electronic device, and a storage medium. BACKGROUND

[0002] With the rapid popularization of intelligent terminals, graphic processing units (GPUs) are increasingly used in general computing fields. A GPU with several hundred computing cores can achieve a tera floating-point operation per second (TFLOPS) in computing power. In the field of general computing, the powerful floating-point operation capability of a GPU is incomparable to that of a central processing unit (CPU), and the general computing capability of a GPU can make up for the deficiency of the parallel computing capability of a CPU.

[0003] In related technologies, in order to facilitate understanding of the state of each GPU in a GPU cluster, a daemon process is deployed on each node of the GPU, and the daemon process collects GPU information such as the model, temperature, power consumption, use time, and use state of the GPU, and then displays the GPU information. When it is determined that a GPU has failed according to the GPU information, an alarm is triggered.

[0004] However, this method only triggers an alarm when a GPU fails, and informs the user that the GPU has failed. Thereafter, the user replaces the GPU or migrates the program of the faulty GPU to another GPU for execution, which affects the normal operation of the business. SUMMARY

[0005] This section is provided to introduce the general concepts of the present disclosure in a simplified form, which will be described in detail in the following detailed description. This section does not intend to identify key or essential features of the claimed technical solutions, nor is it intended to be used to limit the scope of the claimed technical solutions.

[0006] In a first aspect, the present disclosure provides a method for predicting a faulty GPU, the method comprising:

[0007] obtaining parameter information of each GPU in a plurality of GPUs to obtain a parameter information set;

[0008] inputting the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model;

[0009] determining a faulty GPU from the plurality of GPUs according to the prediction result.

[0010] In a second aspect, the present disclosure provides a device for predicting a faulty GPU, the device comprising:

[0011] a parameter information set obtaining module, configured to obtain parameter information of each GPU in a plurality of GPUs to obtain a parameter information set;

[0012] a prediction result obtaining module, configured to input the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model;

[0013] a prediction module, configured to determine a faulty GPU from the plurality of GPUs according to the prediction result.

[0014] In a third aspect, the present disclosure provides a computer readable medium having a computer program stored thereon, wherein the program, when executed by a processing device, implements the steps of the method of the first aspect.

[0015] In a fourth aspect, the present disclosure provides an electronic device comprising:

[0016] a storage device having one or more computer programs stored thereon;

[0017] one or more processing devices configured to execute the one or more computer programs in the storage device to implement the steps of the method of the first aspect.

[0018] The technical solution provided by the present disclosure comprises obtaining parameter information of each GPU in a plurality of GPUs to obtain a parameter information set; inputting the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model, so that the prediction results of a plurality of different models for the failure rate of a GPU can be obtained. Finally, a faulty GPU is determined from the plurality of GPUs according to the prediction result, so that the faulty GPU can be predicted more accurately before the GPU fails by combining the prediction results of a plurality of prediction models, to remind the user to handle it in time and avoid affecting the normal operation of the business.

[0019] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent by describing in detail the following specific embodiments with reference to the attached drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale. In the drawings:

[0021] Figure 1is a flowchart of a method of predicting a failed GPU according to an example embodiment.

[0022] Figure 2 is a flowchart of a method of predicting a failed GPU according to another example embodiment.

[0023] Figure 3 is a flowchart of a method of predicting a failed GPU according to Figure 2 is a schematic diagram of a method of predicting a failed GPU based on a cascade architecture model according to an example embodiment.

[0024] Figure 4 is a flowchart of a method of predicting a failed GPU according to yet another example embodiment.

[0025] Figure 5 is a flowchart of a method of predicting a failed GPU according to Figure 4 is a schematic diagram of a method of predicting a failed GPU based on a cascade architecture model according to an example embodiment.

[0026] Figure 6 is a block diagram of a device of predicting a failed GPU according to an example embodiment.

[0027] Figure 7 is a structural schematic diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While several embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art. It should be understood that the drawings and embodiments are only for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "comprising" and variations thereof as used herein are used inclusively, i.e., "including, but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related terms are defined in the description that follows.

[0031] It should be noted that the terms "first", "second", and the like in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0032] It should be noted that the terms "one", "multiple" in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that "one or more" should be understood unless otherwise explicitly indicated in the context.

[0033] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0034] In the related art, in order to predict the failure of the GPU in advance, a detection process is usually deployed on the node of the GPU to extract the state parameters of the CPU from the node, and then the state parameters of the GPU are compared with the preset average state parameters, and finally according to the comparison result, it is determined whether the GPU will fail.

[0035] However, this prediction method is too single, and the average state parameters cannot truly reflect the failure standard of the GPU, so it cannot guarantee the accuracy of the prediction result.

[0036] In view of the above problems, the present disclosure provides a failure GPU prediction method and device, electronic equipment and storage medium, which can improve the prediction accuracy of the failure GPU.

[0037] The professional terms involved in the present disclosure are explained as follows:

[0038] Sliding window: The sliding window algorithm is to perform the required operation on a given array or string of a certain window size. This technique can convert the nested loops in a part of the problem into a single loop, so it can reduce the time complexity.

[0039] One-dimensional convolutional neural network (1D-CNN): The input of the one-dimensional convolutional neural network is a vector and a convolution kernel, and the output is also a vector. Usually, the length of the input vector is much larger than the length of the convolution kernel.

[0040] Multilayer Perceptron (MLP) is also called Artificial Neural Network (ANN). In addition to the input and output layers, it can have multiple hidden layers. The simplest MLP contains only one hidden layer, i.e. a three-layer structure. The length of the output vector depends on the padding scheme of the convolution operation, the output of the equal-width convolution.

[0041] Gradient Boosting (GB) model, which is to fit a gradient boosting model using the negative gradient of the loss function at the value of the current model as an approximation of the residual in the boosting tree algorithm.

[0042] Figure 1 is a flow chart of a method for predicting a faulty GPU according to an exemplary embodiment, as shown in Figure 1 The method can include the following steps:

[0043] 110, obtaining parameter information of each GPU in the plurality of GPUs to obtain a parameter information set.

[0044] Exemplarily, the method for predicting a faulty GPU provided by the embodiment can be applied to an electronic device, which can include a terminal device, a server, etc. The electronic device can establish a communication link with each of a plurality of devices to be tested having a GPU. The following describes the electronic device as a server.

[0045] In some embodiments, the server can receive parameter information of the GPU uploaded by each of the plurality of devices to be tested in real time, obtain parameter information of the plurality of GPUs, and then integrate the parameter information of the plurality of GPUs into a parameter information set, wherein the server receives the parameter information of the plurality of GPUs in the parameter information set at the same time. For example, the server receives the parameter information of the plurality of GPUs from 9:00 to 12:00 as a parameter information set, and the parameter information of each GPU in the parameter information set is received at 9:00 to 12:00.

[0046] The parameter information of the GPU can represent the state of the GPU. Optionally, the parameter information includes but is not limited to temperature, power consumption, GPU utilization, GPU memory utilization, etc.

[0047] 120, inputting the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model.

[0048] Exemplarily, for example, the server is configured with a plurality of pre-trained prediction models, for example, the prediction models include a first prediction model and a second prediction model, the server can input the parameter information set into the first prediction model and the second prediction model, respectively, and then obtain a first prediction result output by the first prediction model according to the parameter information set and a second prediction result output by the second prediction model according to the parameter information set. The prediction result includes a failure rate corresponding to each GPU in the parameter information set.

[0049] 130, determining a faulty GPU from the plurality of GPUs according to the prediction result.

[0050] According to the above example, the server can filter out the first fault GPU satisfying the preset condition according to the first prediction result, for example, filter out the GPU with a failure rate greater than a first probability threshold (such as 80%) in the first prediction result as the first fault GPU. Then filter out the second fault GPU satisfying the second preset condition according to the second prediction result, for example, filter out the GPU with a failure rate greater than a second probability threshold (such as 90%) in the second prediction result as the second fault GPU, and finally determine the fault GPU as the same GPU in the first fault GPU and the second fault GPU. The first probability threshold and the second probability threshold can be the same or different, which is not limited here.

[0051] As can be seen, in the embodiment, by obtaining the parameter information of each GPU in the plurality of GPUs, the parameter information set is obtained; then the parameter information set is input into the plurality of pre-trained prediction models to obtain the prediction result corresponding to each prediction model, so that the prediction results of the plurality of different models for the failure rate of the GPU can be obtained. Finally, the fault GPU is determined from the plurality of GPUs according to the prediction result, so that the fault GPU can be predicted before the GPU fails, combining the prediction results of the plurality of prediction models, to remind the user to handle in time and avoid affecting the normal operation of the business. Since the determination of the fault GPU considers the prediction results of different prediction models, the factors considered are more comprehensive, making the prediction more accurate.

[0052] Figure 2 is a flow chart of a fault GPU prediction method according to another example embodiment, as shown in Figure 2 The method can include the following steps:

[0053] 210, obtaining the parameter information of each GPU in the plurality of GPUs to obtain a parameter information set.

[0054] The specific implementation of step 210 can refer to step 110, which will not be repeated here.

[0055] 220, inputting the parameter information set into a first prediction model to obtain a first prediction result corresponding to the first prediction model, and the first prediction result includes a first failure rate corresponding to each GPU.

[0056] For example, the parameter information set includes parameter information 1 corresponding to GPU1, parameter information 2 corresponding to GPU2, parameter information 3 corresponding to GPU3, and parameter information 10 corresponding to GPU10. The server can input the parameter information 1 to the first prediction model to obtain the first failure rate a1 of GPU1, the first failure rate a2 of GPU2, and the first failure rate a10 of GPU10 output by the first prediction model.

[0057] 230. Selecting, from the plurality of GPUs, first target GPUs according to the first failure rates corresponding to each of the GPUs.

[0058] In some embodiments, the implementation of step 230 can include:

[0059] Determining the first m GPUs with the largest first failure rates in the first prediction results as the first target GPUs.

[0060] For example, the server can sort the first failure rates a1, a2, a3,..., a10 in descending order, and then select the first m first failure rates corresponding to the GPUs with the largest first failure rates as the first target GPUs. For example, m = 3, the first three largest first failure rates are a2, a4 and a6, and then the first target GPUs include GPU2, GPU4 and GPU6. Wherein m is a positive integer, and m can be determined according to the total number of GPUs, for example, m can be equal to the total number of GPUs multiplied by k1%.

[0061] In some embodiments, the implementation of step 230 can include:

[0062] Determining the GPUs with first failure rates greater than or equal to a first failure rate threshold in the first prediction results as the first target GPUs. Optionally, the first failure rate threshold can be a user-defined setting, or can be the average of all first failure rates.

[0063] 240. Inputting the parameter information of the first target GPUs into a second prediction model to obtain a second prediction result corresponding to the second prediction model, the second prediction result including a second failure rate corresponding to each of the GPUs in the first target GPUs.

[0064] For example, the server can input the parameter information a2 of GPU2, the parameter information a4 of GPU4 and the parameter information a6 of GPU6 into the second prediction model, and obtain the second failure rate b2 corresponding to GPU2, the second failure rate b4 corresponding to GPU4 and the second failure rate b6 corresponding to GPU6 output by the second prediction model.

[0065] 250. Determining, from the first target GPUs, a failure GPU according to the second failure rate corresponding to each of the GPUs in the first target GPUs.

[0066] In some embodiments, the implementation of step 250 can include:

[0067] Determining the first n first target GPUs with the largest second failure rates in the second prediction results as the failure GPUs.

[0068] For example, the server can sort the second failure rates b2, b4, b6 in descending order, and then select the first n second failure rates as the second target GPUs. For example, n = 1, and the first second failure rate is b4, then the server can determine GPU4 corresponding to the second failure rate b4 as the failure GPU.

[0069] Then the second target GPUs include GPU5 and GPU2. n is a positive integer, which can be determined according to the total number of GPUs, for example, n can be equal to the total number of GPUs multiplied by k2%. k1 and k2 can be the same or different, which is not limited here.

[0070] In some other embodiments, the specific implementation of step 250 can include:

[0071] The first target GPU with a second failure rate greater than or equal to a second failure rate threshold in the second prediction result is determined as the failure GPU. Optionally, the second failure rate threshold can be user-defined or the average of all second failure rates.

[0072] Optionally, the first prediction model is a 1D-CNN model, and the second prediction model is an MLP model, or the first prediction model is an MLP model, and the second prediction model is a 1D-CNN model. Both the first prediction model and the second prediction model are trained according to the plurality of parameter information samples of the GPUs, and can output the failure rate of the GPU according to the input parameter information of the GPU.

[0073] For example, in actual application, as shown in Figure 3 The 1D-CNN model and the MLP model can form a cascading architecture model. The server can input the parameter information set into the 1D-CNN model first, and the 1D-CNN model predicts the top k1% of the first target GPUs with the highest scores, and then inputs the parameter information of the first target GPU into the MLP model for prediction, and the top k2% of the GPUs with the highest scores of the MLP model are determined as the failure GPUs. The prediction score can be positively correlated with the failure rate.

[0074] In some embodiments, the number of cascaded models in the cascading architecture can not be limited, and the cascading order of different models can not be limited. For example, a third prediction model can be cascaded on the basis of the first prediction model and the second prediction model described above. The second target GPU can be obtained according to the above manner, and the parameter information of the second target GPU is input into the third prediction model. The GPU with the highest prediction score output by the third prediction model is taken as the faulty GPU. By analogy, a plurality of same or different models can be cascaded to predict the faulty GPU in this way.

[0075] It can be understood that the training of the prediction model in the present embodiment can be end-to-end training, that is, a plurality of models are trained together.

[0076] In the present embodiment, the GPU fault timing model based on the cascading technology proposed can improve the prediction accuracy by an average of 11.1% according to experimental data, thereby effectively improving the prediction accuracy.

[0077] Figure 4 is a flowchart of a method for predicting a faulty GPU according to yet another exemplary embodiment, as shown in Figure 4 The method can include the following steps:

[0078] 310. Obtain parameter information of each GPU in a plurality of GPUs to obtain a parameter information set.

[0079] The specific implementation of step 310 can refer to step 110, and thus will not be described here.

[0080] In some embodiments, the specific implementation of step 310 can include obtaining parameter information of each GPU in a plurality of GPUs according to a preset period to obtain a parameter information set, wherein the period length of the preset period is greater than or equal to a specified length.

[0081] For example, the specified length is 1 day, and the preset period can be to predict once every 3 days. Then, the pre-trained prediction model will obtain the parameter information of each GPU in a plurality of GPUs for 3 days to obtain a parameter information set, and then make a prediction based on the parameter information set.

[0082] Considering that the prediction model can determine the occurrence of a GPU fault according to a parameter information set obtained in 1 day, it is certain that the occurrence of a GPU fault can be determined according to a parameter information set obtained in 3 days. After all, a GPU will fail within 1 day, and it will certainly fail within 3 days. In the present embodiment, by making the period length of the preset period greater than or equal to the specified length, the prediction time length can be effectively increased, and the prediction model accuracy can be improved by an average of 4.1% according to experimental data, thereby effectively improving the prediction accuracy of the prediction model.

[0083] In some embodiments, before step 310, the method can further comprise: obtaining a parameter information sample of the GPU; and performing sliding sampling on the parameter information sample through a preset sliding window to obtain a first sample; and training a model based on the first sample to obtain a plurality of prediction models.

[0084] For example, the server can obtain a plurality of parameter information samples of the historical collected parameter information of the plurality of GPUs, arrange the plurality of parameter information samples in a specified manner, perform sliding sampling on the parameter information samples through a preset sliding window, for example, the window size of the sliding window is a preset size (such as 3), and the sliding direction is a preset direction (such as from left to right), so that each 3 parameter information samples can be sampled once, thereby obtaining a first sample, and then inputting the first sample into a specified different model (such as a 1D-CNN model, an MLP model, etc.) for training to obtain a plurality of prediction models.

[0085] Optionally, the specified manner can be random arrangement or arrangement according to time sequence, which is not limited herein.

[0086] In this embodiment, by performing sliding sampling on the training sample through the preset sliding window, the number of positive samples can be increased by about 60 times, thereby improving the prediction accuracy of the trained prediction model.

[0087] 320, inputting the parameter information set into the first prediction model to obtain a first prediction result corresponding to the first prediction model, the first prediction result comprising a first failure rate corresponding to each GPU.

[0088] For example, the parameter information set comprises parameter information 1 corresponding to GPU 1, parameter information 2 corresponding to GPU 2, parameter information 3 corresponding to GPU 3, and parameter information 10 corresponding to GPU 10. The server can input the parameter information 1 to the parameter information 10 into the first prediction model to obtain the first failure rate a1 of GPU 1, the first failure rate a2 of GPU 2, and the first failure rate a10 of GPU 10 output by the first prediction model.

[0089] 330, inputting the parameter information set into the second prediction model to obtain a second prediction result corresponding to the second prediction model, the second prediction result comprising a second failure rate corresponding to each GPU.

[0090] For example, the server can also input the parameter information 1 to the parameter information 10 into the second prediction model to obtain the second failure rate b1 of GPU 1, the second failure rate b2 of GPU 2, and the second failure rate b10 of GPU 10 output by the second prediction model.

[0091] 340. Input the parameter information set into the third prediction model to obtain the third prediction result corresponding to the third prediction model. The third prediction result includes the third failure rate corresponding to each GPU.

[0092] Following the example above, the server can also input parameter information 1 to parameter information 10 into the third prediction model to obtain the third failure rate c1 of GPU1, the third failure rate c2 of GPU2, ... and the third failure rate c10 of GPU10 output by the third prediction model.

[0093] 350. Select the first target GPU from multiple GPUs based on the first failure rate corresponding to each GPU.

[0094] In some implementations, the top m GPUs with the highest first failure rate in the first prediction results are identified as the first target GPUs.

[0095] For example, if m=4, the first failure rate in the first prediction results is sorted from largest to smallest as follows: a2, a3, a4, a10... Then the first target GPU can be determined as GPU2, GPU3, GPU4, GPU10.

[0096] 360. Select a second target GPU from multiple GPUs based on the second failure rate corresponding to each GPU.

[0097] In some implementations, the top n GPUs with the highest second failure rates in the second prediction results are identified as the second target GPUs.

[0098] For example, if n=4, the second failure rates in the second prediction results are sorted from largest to smallest as follows: b1, b3, b2, b7... Then the second target GPUs can be determined as GPU1, GPU3, GPU2, GPU7.

[0099] 370. Select a third target GPU from multiple GPUs based on the third failure rate corresponding to each GPU.

[0100] In some implementations, the top k GPUs with the highest third failure rates in the third prediction results are identified as the third target GPUs.

[0101] For example, if k=3, the third failure rate in the third prediction results is sorted from largest to smallest as c3, c5, c6, c9..., then the third target GPU can be determined as GPU3, GPU5, GPU6, GPU9.

[0102] 380. Identify the same GPU among the first target GPU, the second target GPU, and the third target GPU as the faulty GPU.

[0103] Based on the above examples, the GPU 3 can be determined as the faulty GPU.

[0104] wherein m, n, k are positive integers, and m, n, k can be the same or different, which is not limited herein. Optionally, m, n, k can be determined according to the total number of GPUs, and specifically can be the total number of GPUs k%.

[0105] Optionally, the first prediction model is a 1D-CNN model, the second prediction model is an MLP model, and the third prediction model is a GB model.

[0106] For example, in actual application, as shown in Figure 5 The 1D-CNN model, the MLP model and the GB model can form a parallel architecture model, and the server can input the parameter information set into the 1D-CNN model, the MLP model and the GB model, and then take the intersection of the top k1% GPUs in the prediction scores of the 1D-CNN model, the MLP model and the GB model as the faulty GPU. The prediction score of the GPU can be positively correlated with the failure rate of the GPU.

[0107] In some embodiments, the parallel architecture model can include at least two prediction models, and the specific number is not limited herein. For example, the parallel architecture model can further include four or more prediction models, and the plurality of prediction models are different from the prediction model, so as to determine the faulty GPU by combining the decisions of multiple models.

[0108] In this embodiment, the parallel architecture model is proposed to predict the faulty GPU, and the experimental data verifies that the accuracy can be improved by 13% on average, thereby effectively improving the prediction accuracy.

[0109] It can be understood that in this embodiment, the parallel architecture model can be obtained by individually training each prediction model based on the first sample.

[0110] In some embodiments, the method further includes:

[0111] If the use time of the currently used prediction model is longer than the preset time, a new parameter information sample of the GPU is obtained, and the new parameter information sample is slidingly sampled by a preset sliding window to obtain a second sample. The model is trained based on the second sample to obtain a plurality of prediction models.

[0112] For example, if the preset time length is 15 days, the server can reacquire a new parameter information sample of the GPU every 15 days, and perform sliding sampling on the new parameter information sample through a preset sliding window to obtain a second sample, and then perform retraining of the model based on the second sample to obtain a plurality of new prediction models, so that the plurality of prediction models are streaming models rather than a fixed static model.

[0113] In this embodiment, by using the streaming model technology, the experimental data verification can improve the prediction accuracy of the prediction model by an average of 24.7% compared with the fixed model.

[0114] Figure 6 is a block diagram of a prediction device for a faulty GPU according to an example embodiment, as shown in Figure 6 The device 500 can include a parameter information set acquisition module 510, a prediction result acquisition module 520, and a prediction module 530. Wherein:

[0115] The parameter information set acquisition module 510 is configured to acquire the parameter information of each GPU in the plurality of GPUs to obtain a parameter information set.

[0116] The prediction result acquisition module 520 is configured to input the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model.

[0117] The prediction module 530 is configured to determine a faulty GPU from the plurality of GPUs according to the prediction result.

[0118] In some embodiments, the plurality of prediction models includes a first prediction model and a second prediction model, and the prediction result acquisition module 520 includes:

[0119] The first prediction result determination submodule is configured to input the parameter information set into the first prediction model to obtain a first prediction result corresponding to the first prediction model, and the first prediction result includes a first failure rate corresponding to each GPU.

[0120] The first target GPU determination submodule is configured to select a first target GPU from the plurality of GPUs according to the first failure rate corresponding to each GPU.

[0121] The second prediction result determination submodule is configured to input the parameter information of the first target GPU into the second prediction model to obtain a second prediction result corresponding to the second prediction model, and the second prediction result includes a second failure rate corresponding to each GPU in the first target GPU.

[0122] Correspondingly, the prediction module 530 includes:

[0123] The fault GPU determination submodule is configured to determine, from the first target GPUs, a fault GPU according to a second failure rate corresponding to each GPU in the first target GPUs.

[0124] In some embodiments, the first target GPU determination submodule is further configured to determine, as the first target GPUs, the first m GPUs with the largest first failure rates in the first prediction result, where m is a positive integer.

[0125] The fault GPU determination submodule is further configured to determine, as the fault GPUs, the first n target GPUs with the largest second failure rates in the second prediction result, where n is a positive integer.

[0126] In some embodiments, the first prediction model is a 1D-CNN model, and the second prediction model is an MLP model; or the first prediction model is an MLP model, and the second prediction model is a 1D-CNN model.

[0127] In some embodiments, the prediction model includes a first prediction model, a second prediction model, and a third prediction model, and the prediction result acquisition module 520 is specifically configured to:

[0128] input the set of parameter information into the first prediction model to obtain a first prediction result corresponding to the first prediction model, the first prediction result including a first failure rate corresponding to each GPU.

[0129] input the set of parameter information into the second prediction model to obtain a second prediction result corresponding to the second prediction model, the second prediction result including a second failure rate corresponding to each GPU.

[0130] input the set of parameter information into the third prediction model to obtain a third prediction result corresponding to the third prediction model, the third prediction result including a third failure rate corresponding to each GPU.

[0131] Correspondingly, the prediction module 530 is specifically configured to select, from the plurality of GPUs, first target GPUs according to the first failure rate corresponding to each GPU; select, from the plurality of GPUs, second target GPUs according to the second failure rate corresponding to each GPU; select, from the plurality of GPUs, third target GPUs according to the third failure rate corresponding to each GPU; and determine, as the fault GPUs, the same GPUs in the first target GPUs, the second target GPUs, and the third target GPUs.

[0132] In some embodiments, the prediction result acquisition module 520 is specifically further configured to determine, as the first target GPUs, the first m GPUs with the largest first failure rates in the first prediction result; determine, as the second target GPUs, the first n GPUs with the largest second failure rates in the second prediction result; and determine, as the third target GPUs, the first k GPUs with the largest third failure rates in the third prediction result, where m, n, and k are positive integers.

[0133] In some embodiments, the first prediction model is a 1D-CNN model, the second prediction model is an MLP model, and the third prediction model is a GB model.

[0134] In some embodiments, the parameter information set obtaining module 510 is specifically configured to obtain the parameter information of each of the plurality of GPUs according to a preset period, to obtain the parameter information set, where a period length of the preset period is greater than or equal to a specified length.

[0135] In some embodiments, the apparatus 500 further includes:

[0136] The sample obtaining module is configured to obtain a parameter information sample of the GPU.

[0137] The preprocessing module is configured to perform sliding sampling on the parameter information sample through a preset sliding window, to obtain a first sample.

[0138] The first model training module is configured to perform model training based on the first sample, to obtain a plurality of prediction models.

[0139] In some embodiments, the apparatus 500 further includes:

[0140] The second sample obtaining module is configured to, if a use length of the currently used prediction model is above a preset length, obtain a new parameter information sample of the GPU, and perform sliding sampling on the new parameter information sample through a preset sliding window, to obtain a second sample.

[0141] The second model training module is configured to perform model training based on the second sample, to obtain a plurality of prediction models.

[0142] Reference will be made to the following description Figure 7 which shows a structural schematic diagram of an electronic device 600 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Multimedia Players), vehicle terminal (such as vehicle navigation terminal) and the like, and fixed terminals such as digital TV, desktop computers and the like. Figure 7 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0143] As Figure 7As shown, the electronic device 600 can include a processing device (e.g., a central processor, a GPU, etc.) 601 that can perform various suitable actions and processes in accordance with programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage device 608. Various programs and data needed in the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other by a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0144] Generally, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication devices 609 can allow the electronic device 600 to exchange data with other devices wirelessly or through wires. Although Figure 7 The electronic device 600 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0145] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 609, or installed from the storage devices 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0146] It is noted that the aforementioned computer-readable medium of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a computer-readable program code transmitted by a computer-readable storage medium or carried by a carrier wave in a baseband or as part of a carrier wave. Such a propagated computer-readable signal medium can take various forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to wire, cable, RF (radio frequency), or the like, or any suitable combination of the foregoing.

[0147] In some embodiments, the electronic device can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communications of any form or medium (e.g., a communications network). Examples of communications networks include local area networks ("LAN"), wide area networks ("WAN"), internetworks (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future developed networks.

[0148] The aforementioned computer-readable medium can be contained in the aforementioned electronic device; or can exist separately without being assembled into the electronic device.

[0149] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: obtain parameter information of each GPU in the plurality of GPUs to obtain a parameter information set; input the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model; and determine a faulty GPU from the plurality of GPUs according to the prediction result.

[0150] Alternatively, the computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: obtain parameter information of each GPU in the plurality of GPUs to obtain a parameter information set; input the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model; and determine a faulty GPU from the plurality of GPUs according to the prediction result.

[0151] Computer program code for carrying out operations of the present disclosure can be written in any of one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0152] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0153] The modules described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0154] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0155] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of a program of a processor, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] The above description is merely exemplary of the present disclosure and the application of the principles thereof and the scope of the disclosure is not limited to the specific embodiments described herein, but only by the claims that follow. For example, the features described above and in the following description are not limited to the combination stated in the respective paragraphs. Rather, each feature can be used alone or in combination with one or more other features. It is therefore contemplated that the features described herein can be used in any combination. It is further contemplated that the features described herein can be used in any combination with any other features described herein or known in the art.

[0157] Moreover, while operations can be depicted in the drawings in a particular, chronological order, this should not be understood as requiring or implying that such operations be performed in the time order illustrated. On the contrary, it is recognized that such operations can be performed in any order, or some operations can be performed in parallel, that the illustrated embodiments can take place over different time periods than that illustrated, and that the illustrated embodiments can take place at different scales than that shown. Furthermore, it is recognized that some activities can take place on a daily, weekly, monthly, yearly, or other periodic basis, or on a continuous or intermittent basis. It is further recognized that such activities can take place on a computer system, a computer network, or on any other suitable device or combination of devices.

[0158] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With respect to the devices in the above-described embodiments, in which various modules perform operations, the specific manner in which the various modules perform the operations has been described in detail in the embodiments relating to the method. Here, no detailed explanation will be given.

Claims

1. A method of predicting a failing GPU, the method comprising: The method comprises the following steps: obtaining parameter information of each GPU in a plurality of GPUs to obtain a parameter information set; inputting the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model; determining a faulty GPU from the plurality of GPUs according to the prediction result; the plurality of prediction models comprise a first prediction model and a second prediction model, and the inputting of the parameter information set into the plurality of pre-trained prediction models to obtain the prediction result corresponding to each prediction model comprises: inputting the parameter information set into the first prediction model to obtain a first prediction result corresponding to the first prediction model, wherein the first prediction result comprises a first failure rate corresponding to each GPU; selecting a first target GPU from the plurality of GPUs according to the first failure rate corresponding to each GPU; inputting the parameter information of the first target GPU into the second prediction model to obtain a second prediction result corresponding to the second prediction model, wherein the second prediction result comprises a second failure rate corresponding to each GPU in the first target GPU; the determining of the faulty GPU from the plurality of GPUs according to the prediction result comprises: determining the faulty GPU from the first target GPU according to the second failure rate corresponding to each GPU in the first target GPU.

2. The method of claim 1, wherein, the selecting of the first target GPU from the plurality of GPUs according to the first failure rate corresponding to each GPU comprises: determining the first m GPUs with the largest first failure rate in the first prediction result as the first target GPU, wherein m is a positive integer; the determining of the faulty GPU from the first target GPU according to the second failure rate corresponding to each GPU in the first target GPU comprises: determining the first n first target GPUs with the largest second failure rate in the second prediction result as the faulty GPU, wherein n is a positive integer.

3. The method of claim 1, wherein, the first prediction model is a 1D-CNN model, and the second prediction model is an MLP model; alternatively, the first prediction model is an MLP model, and the second prediction model is a 1D-CNN model.

4. The method according to any one of claims 1 to 3, characterized in that, the obtaining of the parameter information of each GPU in the plurality of GPUs to obtain the parameter information set comprises: obtaining the parameter information of each GPU in the plurality of GPUs according to a preset period to obtain the parameter information set, wherein the period length of the preset period is greater than or equal to a specified length.

5. The method according to any one of claims 1 to 3, characterized in that, before the obtaining of the parameter information of each GPU in the plurality of GPUs to obtain the parameter information set, the method further comprises: obtaining a parameter information sample of the GPU; sliding sampling the parameter information sample through a preset sliding window to obtain a first sample; training a model based on the first sample to obtain a plurality of prediction models.

6. The method of claim 5, wherein, the method further comprises: if the use length of the currently used prediction model is greater than or equal to a preset length, obtaining a new parameter information sample of the GPU and sliding sampling the new parameter information sample through the preset sliding window to obtain a second sample; training a model based on the second sample to obtain the plurality of prediction models.

7. A prediction apparatus of a failure GPU, characterized by, The device comprises: a parameter information set obtaining module, configured to obtain parameter information of each GPU in a plurality of GPUs to obtain a parameter information set; a prediction result obtaining module, configured to input the parameter information set into a plurality of pre-trained prediction models to obtain a prediction result corresponding to each prediction model; a prediction module, configured to determine a faulty GPU from the plurality of GPUs according to the prediction result; the plurality of prediction models comprise a first prediction model and a second prediction model, and the prediction result obtaining module comprises: a first prediction result determining submodule, configured to input the parameter information set into the first prediction model to obtain a first prediction result corresponding to the first prediction model, the first prediction result comprising a first failure rate corresponding to each GPU; a first target GPU determining submodule, configured to select a first target GPU from the plurality of GPUs according to the first failure rate corresponding to each GPU; a second prediction result determining submodule, configured to input parameter information of the first target GPU into the second prediction model to obtain a second prediction result corresponding to the second prediction model, the second prediction result comprising a second failure rate corresponding to each GPU in the first target GPU; the prediction module comprises: a faulty GPU determining submodule, configured to determine a faulty GPU from the first target GPU according to the second failure rate corresponding to each GPU in the first target GPU.

8. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processing device to implement the steps of the method in any one of claims 1-6.

9. An electronic device, comprising: comprise: a storage device having one or more computer programs stored thereon; one or more processing devices configured to execute the one or more computer programs in the storage device to implement the steps of the method in any one of claims 1-6.

Images