Method and apparatus for determining abnormal processing unit, and non-volatile storage medium
By obtaining the operating data of the graphics processor, using preset warning rules and exception judgment models, and combining multiple judgment methods, the warning rules are identified and updated in real time, the problem that GPU faults cannot be identified in time during large-scale model training is solved, and the efficiency and accuracy of the training task are improved.
Patent Information
- Application Number
- PCT/CN2024/121939
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-12-14
- Filing Date
- 2024-09-27
- Publication Date
- 2025-08-21
AI Technical Summary
In the prior art, GPU faults cannot be identified in time during the training of large models, resulting in interruption of training tasks and low efficiency, and the fault detection methods are single and low accuracy, which affects training effects and waste of resources.
By obtaining the operating data of the graphics processor, using preset warning rules and exception judgment models, combining multiple judgment methods such as expert rules, threshold judgment, distribution analysis and machine learning algorithms, we can identify and update the warning rules in real time to predict possible faulty graphics processors.
It realizes accurate identification of exceptions during the training task execution process, improves the efficiency of the training task, reduces fault recovery time and resource waste, and enhances the accuracy and adaptability of prediction.
Smart Images
Figure CN2024121939_21082025_PF_FP_ABST
Abstract
Description
Method and device for determining exception handler and non-volatile storage medium
[0001] Related applications
[0002] This application claims priority to Chinese patent application number 202311723709X, filed on December 14, 2023, entitled “Method, device and non-volatile storage medium for determining an exception handler,” the entire text of which is hereby incorporated by reference. Technical Field
[0003] The present application relates to the field of data processing, and in particular to a method and device for determining an abnormal processor and a non-volatile storage medium. Background Art
[0004] Training large models typically requires a large number of GPUs (Graphics Processing Units). GPU errors during training can severely impact training results. Current techniques for addressing GPU errors typically wait until model training is interrupted before troubleshooting the GPUs in use. This can lead to delays in identifying faulty GPUs and negatively impacting training efficiency.
[0005] To address the above-mentioned problems, no effective solutions have been proposed so far.
[0006] Summary of the Invention
[0007] In a first aspect, an embodiment of the present application provides a method for determining an abnormal processor, comprising: obtaining operating data of each graphics processor in a target system, and determining a first abnormal graphics processor based on preset warning rules and the operating data of each graphics processor; determining a graphics processor set in the target system corresponding to a target training task, and determining a second abnormal graphics processor in the graphics processor set based on the operating data of the graphics processors in the graphics processor set; training a preset abnormality judgment model based on the operating data of the first abnormal graphics processor and the second abnormal graphics processor to obtain a target abnormality judgment model, and updating the warning rules based on the target abnormality judgment model, wherein the target abnormality judgment model is used to determine a third abnormal graphics processor in the graphics processor.
[0008] In some embodiments, the step of determining the first abnormal graphics processor based on preset warning rules and the operating data of the graphics processor includes: determining the warning data type based on the preset warning rules; determining the warning operating data corresponding to the warning data type in the operating data, and the abnormality judgment rule corresponding to the warning operating data; judging the warning operating data based on the abnormality judgment rule, thereby determining the first abnormal graphics processor.
[0009] In some embodiments, the early warning operation data includes first-category early warning operation data. The first-category early warning operation data is data used to determine whether a graphics processor is faulty based on a preset threshold value. The step of determining the first abnormal graphics processor by judging the early warning operation data based on an abnormality judgment rule includes determining a preset threshold value corresponding to the first-category early warning operation data, and determining the first abnormal graphics processor from each graphics processor based on a comparison result of the first-category early warning operation data corresponding to each graphics processor with the preset threshold value.
[0010] In some embodiments, the warning operation data also includes second-category warning operation data, and the second-category warning operation data is data for determining whether a graphics processor is faulty based on the distribution of the second-category warning operation data of each graphics processor in the target system. The warning operation data is judged according to an abnormality judgment rule, and the step of determining the first abnormal graphics processor includes: determining the abnormal value interval corresponding to the second-category warning operation data based on the distribution of the second-category warning operation data of each graphics processor in the target system, and determining that the graphics processor corresponding to the second-category warning data in the abnormal value interval is the first abnormal graphics processor.
[0011] In some embodiments, the step of determining a second abnormal graphics processor in a graphics processor set based on operating data of graphics processors in the graphics processor set includes: determining an operating data set corresponding to the graphics processor set, wherein the operating data set includes operating data corresponding to each graphics processor in the graphics processor set at multiple data collection time points, the multiple data collection time points are time points before a current detection time point, and the current detection time point is a time point at which the second abnormal graphics processor is determined in the graphics processor set based on the operating data of the graphics processors in the graphics processor set; determining a weight of each operating data based on the data collection time point corresponding to each operating data, wherein the closer the data collection time point is to the current detection time point, the greater the weight; determining detection data corresponding to each graphics processor based on the weight of the operating data and the operating data set; and determining the second abnormal graphics processor in the graphics processor set based on each detection data.
[0012] In some embodiments, the step of determining a second abnormal graphics processor in the graphics processor set based on each detection data includes: determining a target abnormal partitioning tree model; processing the detection data using the target abnormal partitioning tree model to determine abnormal detection data corresponding to each graphics processor in the graphics processor set; determining an abnormality score for each graphics processor based on the abnormal detection data corresponding to each graphics processor; and determining the second abnormal graphics processor in the graphics processor set based on each abnormality score.
[0013] In some embodiments, determining the second abnormal GPU in the set of GPUs based on each abnormality score includes: determining a GPU corresponding to an abnormality score greater than an abnormality threshold as the second abnormal GPU.
[0014] In some embodiments, the target abnormality judgment model includes a first base learner, a second base learner, a third base learner and a meta-learner, wherein the first base learner includes a support vector machine, the second base learner includes a gradient boosting decision tree, the third base learner includes a fully connected neural network, and the meta-learner includes a logistic regression model.
[0015] In some embodiments, the step of updating the warning rules based on the target abnormality judgment model includes: determining the importance of each type of operating data based on the second basic learner; updating the warning rules based on the importance, wherein updating the warning rules includes: updating the type of operating data in the warning rules.
[0016] In a second aspect, an embodiment of the present application further provides a device for determining an abnormal processor, comprising: a first processing module for acquiring operating data of each graphics processor in a target system, and determining a first abnormal graphics processor based on preset warning rules and the operating data of each graphics processor; a second processing module for determining a set of graphics processors in a target system corresponding to a target training task, and determining a second abnormal graphics processor in the graphics processor set based on the operating data of the graphics processors in the graphics processor set; a third processing module for training a preset abnormality judgment model based on the operating data of the abnormal graphics processor to obtain a target abnormality judgment model, and updating the warning rules based on the target abnormality judgment model, wherein the target abnormality judgment model is used to determine a third abnormal graphics processor in the graphics processor.
[0017] In a third aspect, an embodiment of the present application further provides a non-volatile storage medium, in which a program is stored, wherein a method for determining an abnormality handler is controlled to execute on a device where the non-volatile storage medium is located when the program is running.
[0018] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a memory and a processor, the processor being configured to run a program stored in the memory, wherein a method for determining an abnormal processor is executed when the program is running.
[0019] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0021] FIG1 is a schematic structural diagram of a target system provided according to an embodiment of the present application;
[0022] FIG2 is a schematic structural diagram of an early warning system provided according to an embodiment of the present application;
[0023] FIG3 is a flow chart of a method for determining an exception handler according to an embodiment of the present application;
[0024] FIG4 is a schematic diagram of the structure of an abnormality judgment model provided according to an embodiment of the present application;
[0025] FIG5 is a schematic diagram of a training and verification process of an abnormality judgment model provided according to an embodiment of the present application;
[0026] FIG6 is a schematic structural diagram of a device for determining an abnormality handler according to an embodiment of the present application;
[0027] FIG7 is a schematic structural diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.
[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0030] Currently, GPU errors during training are a significant factor affecting the results of large model training. GPUs play a key role in large-scale deep learning training, handling the enormous amount of computation and data. However, due to the complexity of training and the long runtimes involved, GPU errors often become a significant factor hindering the training process.
[0031] First, training large models typically takes a long time, potentially hours or even days. During this process, the GPU needs to run continuously at high load, which increases the risk of GPU failure. The GPU is a highly complex and sensitive hardware component, and any failure or error can disrupt the entire training process.
[0032] Secondly, large models typically have a large number of parameters and complex computational operations. This places higher demands on GPUs, such as memory capacity, computational performance, and parallel processing capabilities. If the GPU cannot meet these requirements, problems such as memory overflow, computational errors, or performance degradation may occur. This is especially true when GPU anomalies occur, which increases the probability of memory overflow, computational errors, or performance degradation.
[0033] Furthermore, large-scale deep learning training often requires constant adjustment of hyperparameters and network structures. This requires frequent GPU configuration and debugging. Human intervention or configuration errors can also lead to GPU errors, such as incorrect hyperparameter settings and incorrect video memory allocation.
[0034] Finally, large-scale training often requires parallel computing on multiple GPUs, which involves communication and synchronization between GPUs. If these operations are incorrect or unstable, data transmission errors or performance degradation may occur.
[0035] As can be seen, failure to detect GPU errors in advance can lead to the following consequences: First, large model training may be interrupted, requiring manual intervention to retrain the model, resulting in low training efficiency. Second, GPU errors may cause training model anomalies, preventing the model from achieving the expected results. Third, GPU failures can lead to insufficient resources, resulting in project failure and the need for urgent GPU procurement.
[0036] In summary, the normal operation of GPUs is crucial to the completion of large model training. To ensure the normal operation of the training process, timely identification of abnormal GPUs has become an urgent problem to be solved.
[0037] Currently, related technologies for identifying abnormal GPUs typically perform troubleshooting on the GPU used for training after the model training task is interrupted, lacking the ability to provide early warning. Furthermore, if there are sufficient nodes after a failure, the interrupted task can be redistributed to a new node, and training can be resumed from the interruption point based on checkpoints. However, if there are insufficient nodes, the node where the faulty graphics card is located must be repaired or replaced. After hardware troubleshooting, repair, or replacement of the faulty graphics card, the interrupted training task can be restarted on the original node.
[0038] Because related technologies cannot promptly identify abnormal GPUs during training, they can lead to the following training issues: First, during distributed training, because each GPU operates independently, a GPU failure may occur without disrupting the entire model training process. This can affect model training performance during the period of the GPU failure and cause checkpoints during that period to be contaminated by the hardware failure. This requires manual troubleshooting by algorithm developers, who can then select a suitable, unaffected checkpoint to restart training. Second, certain GPU failures cannot be directly indicated by GPU metrics. For example, some failures can only be detected by abnormal GPU power consumption over a period of time, requiring personnel with extensive practical experience to identify the anomaly. Third, related technologies often detect GPU failures with a delay, making it impossible to predict the potential failures or the number of faulty devices. Procuring and replacing equipment and scheduling other training tasks after a failure occurs wastes significant time and disrupts the normal training schedule. In inference services, the lack of early warning can also lead to online service interruptions, impacting business operations. Fourth, the fault detection methods used in related technologies are relatively simple, mostly relying solely on rules or statistical models, without integrating and adaptively adjusting multiple detection methods. Due to insufficient training data in the early stages, relying solely on models resulted in low accuracy. Similarly, individual rules cannot cover all possible scenarios, resulting in poor scalability. Fifth, related technologies do not link fault detection to training tasks, mostly using certain statistical values to identify abnormal GPUs. However, some statistical indicators, such as GPU utilization and power usage, are highly correlated with specific training tasks.
[0039] In summary, the current GPU fault detection mechanism, which mainly detects interruptions in training or online inference tasks, suffers from insufficient intelligence, low accuracy, and poor timeliness. This results in subsequent fault recovery requiring a large amount of work from professionals, leading to delays in training plans and waste of training resources during the recovery period.
[0040] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.
[0041] According to an embodiment of the present application, a method embodiment of a method for determining an exception handler is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0042] An embodiment of the present application provides a target system as shown in FIG1 , which is used to run the above-mentioned abnormal processor determination method. As shown in FIG1 , the target system includes an early warning system 10 and a plurality of graphics processors 12. The early warning system is used to execute the following abnormal processor determination method: obtaining the operating data of each graphics processor in the target system, and determining the first abnormal graphics processor based on the preset early warning rules and the operating data of each graphics processor; determining the graphics processor set in the target system corresponding to the target training task, and determining the second abnormal graphics processor in the graphics processor set based on the operating data of the graphics processors in the graphics processor set; training the preset abnormal judgment model based on the operating data of the first abnormal graphics processor and the second abnormal graphics processor to obtain a target abnormal judgment model, and updating the early warning rules based on the target abnormal judgment model, wherein the target abnormal judgment model is used to determine the third abnormal graphics processor in the graphics processor.
[0043] Figure 2 is a structural diagram of the early warning system 10 provided according to an embodiment of the present application. As shown in Figure 2, the early warning system 10 includes a rule statistics module 20, a task anomaly detection module 22 and an integrated model module 24. Among them, the rule statistics module 20 is used to analyze the operating data of the GPU according to the screened feature types, expert rules, etc., so as to determine the abnormal graphics processor in the target system. The task anomaly detection module 22 is used to judge the abnormal graphics processor for all graphics processors involved in a specific training task. The integrated model module 24 is used to train the abnormality prediction model based on the operating data of the abnormal graphics processor determined by other modules, and determine the abnormal graphics processor in the target system through the abnormal prediction model. In addition, the integrated model module 24 is also used to perform feature importance analysis based on the training results of the abnormal prediction model, and feed back the analysis results to the rule statistics module 20.
[0044] In the above operating environment, an embodiment of the present application provides a method for determining an exception handler, as shown in FIG3 , the method comprising the following steps:
[0045] Step S302: Obtain the operating data of each graphics processor in the target system, and determine the first abnormal graphics processor according to the preset warning rules and the operating data of each graphics processor. The target system includes multiple graphics processors.
[0046] Specifically, GPU metrics can be used to obtain the operating data of each graphics processor, which may include temperature, power consumption, core frequency, memory usage, video memory usage, etc.
[0047] In one embodiment, the above-mentioned step S302 includes: determining the warning data type according to a preset warning rule; determining the warning operation data corresponding to the warning data type in the operation data, and the abnormality judgment rule corresponding to the warning operation data; judging the warning operation data according to the abnormality judgment rule, thereby determining the first abnormal graphics processor.
[0048] As an optional embodiment, the early warning operation data includes first-category early warning operation data and second-category early warning operation data, wherein the first-category early warning operation data is data used to determine whether a graphics processor is faulty based on a preset threshold, and the second-category early warning operation data is data used to determine whether a graphics processor is faulty based on the distribution of the second-category early warning operation data of each graphics processor in the target system. For the first-category early warning operation data, judging the early warning operation data based on an abnormality judgment rule to thereby determine a first abnormal graphics processor includes: determining a preset threshold corresponding to the first-category early warning operation data, and determining the first abnormal graphics processor in each graphics processor based on a comparison result of the first-category early warning operation data corresponding to each graphics processor with the preset threshold;
[0049] For the second-category warning operation data, the warning operation data is judged according to the abnormality judgment rule, so as to determine the step of the first abnormal graphics processor, including: determining the abnormal value interval corresponding to the second-category warning operation data based on the distribution of the second-category warning operation data of each graphics processor in the target system, and determining that the graphics processor corresponding to the second-category warning data located in the abnormal value interval is the first abnormal graphics processor.
[0050] Specifically, the preset warning rules may include an expert rule base and feature importance determined according to the training results of the target anomaly prediction model. When judging whether the graphics processor is faulty according to the preset warning rules, it can be grouped according to the GPU model and a threshold can be set respectively. If it exceeds or falls below the threshold, an immediate warning will be issued. For screening features that do not have a clear threshold for the time being, the 3sigma principle can be used. That is, assuming that this part of the numerical monitoring data satisfies the normal distribution, then the distance from the average value is The probability of a value other than The probability of occurrence is extremely low. If any important feature value of a GPU falls within this range, an immediate warning is issued. Subsequently, the threshold table is continuously updated using real GPU failure data and corresponding model groupings. In other words, the preset warning rules are updated based on the training results of the target anomaly prediction model and the operating data of abnormal GPUs identified through other methods.
[0051] Step S304: determining a GPU set in the target system corresponding to the target training task, and determining a second abnormal GPU in the GPU set based on the operating data of the GPUs in the GPU set;
[0052] In one embodiment, the above-mentioned step S304 includes: determining an operation data set corresponding to the graphics processor set, wherein the operation data set includes operation data corresponding to each graphics processor in the graphics processor set at multiple data collection time points, the multiple data collection time points are time points before the current detection time point, and the current detection time point is the time point when the second abnormal graphics processor in the graphics processor set is determined based on the operation data of the graphics processors in the graphics processor set; determining the weight of each operation data based on the data collection time point corresponding to each operation data, wherein the closer the data collection time point is to the current detection time point, the greater the weight; determining the detection data corresponding to each graphics processor based on the weight of the operation data and the operation data set; and determining the second abnormal graphics processor in the graphics processor set based on each detection data.
[0053] Specifically, when determining a GPU set based on the target training task and performing anomaly detection on the GPUs within the set, the system first determines whether the number of GPUs used by the training task meets the minimum threshold for entry into the module. If so, the system then groups the GPUs for each large model training task and pulls real-time monitoring information on the GPU resources used during training.
[0054] Furthermore, because frames closer to the next algorithm run have greater reference value, data collected at different time points can be weighted using multi-frame numerical statistics based on exponential time decay. For example, exponentially weighted decay statistics such as [frame]_0·e^(-kt0)+[frame]_1·e^(-kt1)+[frame]_2·e^(-kt2)... can be used as input for the anomaly detection algorithm. Here, t is the number of seconds since the most recent monitoring data was retrieved. This fully utilizes monitoring data from multiple time frames between algorithm runs, while also giving greater weight to data closer to the next test. The K value is a decay coefficient set by the operator and can be adjusted in practice.
[0055] As an optional embodiment, the step of determining a second abnormal graphics processor in the graphics processor set based on each detection data includes: determining a target abnormal partitioning tree model; processing the detection data through the target abnormal partitioning tree model to determine abnormal detection data corresponding to each graphics processor in the graphics processor set; determining an abnormality score of each graphics processor based on the abnormal detection data corresponding to each graphics processor; and determining a second abnormal graphics processor in the graphics processor set based on each abnormality score.
[0056] As an optional implementation, given that in actual detection, the detection algorithm runs much more frequently than the monitoring data is collected. To ensure full utilization of monitoring data, for numerical data, all monitoring data between two detection algorithm runs can be pulled. For non-numerical feature data, this is equivalent to extracting multiple frames (description information) of the feature.
[0057] After processing the input data, the isolation forest algorithm is used to detect outliers. After the GPU monitoring data grouped by task passes through multiple random anomaly partitioning trees, the anomaly score of each GPU can be calculated using the following anomaly score function: s(x) = 2 -E(h(x)) c(n)=2(H(n-1))-(2(n-1) / n)
[0058] In the above formula, s(x) is the anomaly score, and H(i) is the harmonic function, which can be estimated as H(i) = ln(i) + 0.5772156649. c(n) is the average path length for a given sample n and is used to normalize the path length h(x) for sample x. E(h(x)) represents the average height of all random anomaly partitioning trees, or the expected value of the path length. After determining the anomaly score for each GPU, a threshold can be set for the score. GPUs with anomaly scores exceeding the threshold can be warned of failure.
[0059] Specifically, the Isolation Forest algorithm is a machine learning algorithm used to detect outliers. It randomly partitions a dataset to construct an isolation tree and determines whether a data point is an outlier by calculating the path length of the tree. Because the Isolation Forest algorithm can detect outliers quickly and efficiently, it can be used to determine whether there are outliers in the GPU's operating data.
[0060] Step S306: Train the preset abnormality judgment model based on the operating data of the first abnormal graphics processor and the second abnormal graphics processor to obtain a target abnormality judgment model, and update the warning rules based on the target abnormality judgment model, wherein the target abnormality judgment model is used to determine the third abnormal graphics processor in the graphics processor.
[0061] It should be noted that the difference between the first abnormal graphics processor, the second abnormal graphics processor and the third abnormal graphics processor is only the difference in the determination method.
[0062] In the technical solution provided in step S306, the target abnormality judgment model includes a first basic learner, a second basic learner, a third basic learner and a meta-learner, wherein the first basic learner includes a support vector machine, the second basic learner includes a gradient boosting decision tree, the third basic learner includes a fully connected neural network, and the meta-learner includes a logistic regression model.
[0063] Specifically, the target anomaly judgment model, shown in Figure 4, consists of multiple base learners and a meta-learner. The base learners include the first base learner, SVM (Support Vector Machine), the second base learner, GBDT (Gradient Boosted Decision Tree), and the third base learner, DNN (Fully Connected Neural Network). SVM can transform nonlinear problems into linear problems using kernel functions, thereby constructing an efficient classifier. It also has strong generalization capabilities and is particularly advantageous for solving small sample size problems. Furthermore, the constructed classifier is determined only by support vectors, effectively avoiding overfitting.
[0064] The kernel function can be set by the staff. For example, the sigmoid kernel function can be selected. The kernel function formula is as follows: k(x,y)=tanh(αx T y+c)
[0065] The sigmoid kernel function has lower computational overhead than more complex kernel functions and is particularly effective for binary classification problems. The parameters α and c are user-adjustable parameters that control the kernel shape and range. T represents the matrix transpose.
[0066] GBDT itself is a boosting algorithm (ensemble learning algorithm), which means that a final strong learner can be obtained by weighted combination of all weak learners, which is highly interpretable and scalable. In addition, GBDT can obtain the importance of each feature through feature importance. Therefore, the feature importance ranking obtained from GBDT can be used to increase the important features of the expert rule component. DNN has strong nonlinear fitting capabilities and can map complex nonlinear relationships. By integrating these three basic learners and using LR (logistic regression) as the meta-learner model, it is possible to combine the advantages of multiple models and improve the model's generalization ability.
[0067] When adding important features to the expert rules, it is possible to add features that were not previously included in the expert rules and were determined by GBDT, or to adjust the weights of each feature in the expert rules during anomaly judgment, so that more important features have a greater impact on the anomaly judgment results.
[0068] At the same time, this module also uses the interpretability of the GBDT model to obtain more important features through feature importance analysis, providing important features for setting rules and thresholds for the rule warning module.
[0069] In some embodiments of the present application, the step of updating the warning rules based on the target abnormality judgment model includes: determining the importance of each type of operating data based on the second basic learner; updating the warning rules based on the importance, wherein updating the warning rules includes: updating the type of operating data in the warning rules.
[0070] During model training, the operating data of the first and second abnormal GPUs can be fed back and annotated. Furthermore, since abnormal GPU data is relatively rare, the monitoring data of a single abnormal GPU within a certain period before the abnormality, after manual observation, can be used as abnormal data and incorporated into the training data. This allows for the acquisition of a large amount of training data with an imbalance of positive and negative samples.
[0071] In addition, for a certain amount of historical unlabeled monitoring data, the task anomaly detection module in the early warning system can be used to label the unlabeled data according to the training tasks corresponding to the data, and then the labeled data can be used to train the target anomaly prediction model. This achieves the technical effect of fully utilizing historical data.
[0072] The training process for the target anomaly prediction model is shown in Figure 5. During the training process, the target anomaly prediction model can be trained and evaluated using n-fold cross-validation. Specifically, n-fold cross-validation involves dividing the original training data into n subsets, with n-1 subsets used as training data and the remaining subset used as validation data. During the training phase, each base learner in the target anomaly prediction model is trained on the n-1 training subsets, and the trained base learners in the target anomaly prediction model are used to make predictions on the remaining subsets. These steps are repeated multiple times to obtain n prediction results from each base learner. These results are then combined into a new training dataset, which serves as the input for the meta-learner. The meta-learner is trained on this dataset, and finally, the training results of the entire ensemble learner are verified on the global test subset.
[0073] By adopting the model training method provided in the embodiments of the present application, the training data can be fully utilized, and the generalization ability of the model can be better evaluated, thereby reducing the risk of overfitting.
[0074] By acquiring the operating data of each graphics processor in the target system, and determining the first abnormal graphics processor based on the preset warning rules and the operating data of each graphics processor; determining the graphics processor set in the target system corresponding to the target training task, and determining the second abnormal graphics processor in the graphics processor set based on the operating data of the graphics processors in the graphics processor set; training the preset abnormality judgment model based on the operating data of the first abnormal graphics processor and the second abnormal graphics processor to obtain the target abnormality judgment model, and updating the warning rules based on the target abnormality judgment model, wherein the target abnormality judgment model is used to determine the third abnormal graphics processor in the graphics processor, jointly predicting the abnormal graphics processor that may fail in multiple ways, and correcting the preset warning rules according to the prediction results, the purpose of accurately determining the abnormal graphics processor that may fail during the execution of the training task is achieved, thereby achieving the technical effect of improving the efficiency of the training task, and further solving the technical problem of low training task efficiency caused by the inability to pre-determine the graphics processor that may fail during the execution of the training task in the related technology.
[0075] In summary, the method for determining an exception handler provided in the embodiments of the present application has the following advantages over related technologies:
[0076] First, multiple judgment methods are used to comprehensively predict the health status of the processor, making the prediction more accurate;
[0077] Second, we fully utilize the characteristics of the training task and take advantage of the fact that the data of GPUs performing the same training task are more similar to each other to filter out abnormal data. We then determine the abnormal score of the GPU based on the abnormal data, thereby more accurately identifying the abnormal GPU.
[0078] Third, the preset warning rules are continuously updated during the identification process, effectively avoiding missed detections;
[0079] Fourth, the health status of the graphics processor can be predicted without sufficient training data, and training data can be obtained based on the prediction results to train the target abnormality prediction model.
[0080] An embodiment of the present application provides a device for determining an abnormal processor, and FIG6 is a schematic diagram of the structure of the device. As can be seen from FIG6, the device includes: a first processing module 60, which is used to obtain the operating data of each graphics processor in the target system and determine the first abnormal graphics processor based on preset warning rules and the operating data of each graphics processor; a second processing module 62, which is used to determine the graphics processor set in the target system corresponding to the target training task, and determine the second abnormal graphics processor in the graphics processor set based on the operating data of the graphics processors in the graphics processor set; a third processing module 64, which is used to train a preset abnormality judgment model based on the operating data of the abnormal graphics processor to obtain a target abnormality judgment model, and update the warning rules based on the target abnormality judgment model, wherein the target abnormality judgment model is used to determine the third abnormal graphics processor in the graphics processor.
[0081] In some embodiments of the present application, the first processing module 60 is also used to: determine the warning data type based on preset warning rules; determine the warning operation data corresponding to the warning data type in the operation data, and the abnormality judgment rules corresponding to the warning operation data; judge the warning operation data based on the abnormality judgment rules, thereby determining the first abnormal graphics processor.
[0082] In some embodiments of the present application, the early warning operation data includes a first type of early warning operation data and a second type of early warning operation data, wherein the first type of early warning operation data is data for determining whether a graphics processor is faulty based on a preset threshold value, and the second type of early warning operation data is data for determining whether a graphics processor is faulty based on the distribution of the second type of early warning operation data of each graphics processor in the target system; the first processing module 60 is further used to: for the first type of early warning operation data, determine the preset threshold value corresponding to the first type of early warning operation data, and determine the first abnormal graphics processor in each graphics processor based on the comparison result of the first type of early warning operation data corresponding to each graphics processor and the preset threshold value; for the second type of early warning operation data, determine the abnormal value interval corresponding to the second type of early warning operation data based on the distribution of the second type of early warning operation data of each graphics processor in the target system, and determine that the graphics processor corresponding to the second type of early warning data located in the abnormal value interval is the first abnormal graphics processor.
[0083] In some embodiments of the present application, the second processing module 62 is further used to: determine an operation data set corresponding to a graphics processor set, wherein the operation data set includes operation data corresponding to each graphics processor in the graphics processor set at multiple data collection time points, the multiple data collection time points are time points before a current detection time point, and the current detection time point is a time point for determining a second abnormal graphics processor in the graphics processor set based on the operation data of the graphics processors in the graphics processor set; determine a weight of each operation data based on the data collection time point corresponding to each operation data, wherein the closer the data collection time point is to the current detection time point, the greater the weight; determine the detection data corresponding to each graphics processor based on the weight of the operation data and the operation data set; and determine the second abnormal graphics processor in the graphics processor set based on each detection data.
[0084] In some embodiments of the present application, the second processing module 62 is further used to: determine a target abnormal partitioning tree model; process detection data through the target abnormal partitioning tree model to determine abnormal detection data corresponding to each graphics processor in the graphics processor set; determine the abnormality score of each graphics processor based on the abnormal detection data corresponding to each graphics processor; and determine a second abnormal graphics processor in the graphics processor set based on each abnormality score.
[0085] In some embodiments, the second processing module 62 is further configured to: determine a graphics processor corresponding to an abnormality score greater than an abnormality threshold as the second abnormal graphics processor.
[0086] In some embodiments of the present application, the target abnormality judgment model includes a first base learner, a second base learner, a third base learner and a meta-learner, wherein the first base learner includes a support vector machine, the second base learner includes a gradient boosting decision tree, the third base learner includes a fully connected neural network, and the meta-learner includes a logistic regression model.
[0087] In some embodiments of the present application, the third processing module 64 is further used to: determine the importance of each type of operating data based on the second basic learner; update the warning rules based on the importance, wherein updating the warning rules includes: updating the type of operating data in the warning rules.
[0088] It should be noted that the various modules in the above-mentioned abnormality handler determination device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.
[0089] An embodiment of the present application provides a non-volatile storage medium, in which a program is stored. When the program is running, the device where the non-volatile storage medium is located is controlled to execute the following abnormal processor determination method: obtaining operating data of each graphics processor in a target system, and determining a first abnormal graphics processor based on preset warning rules and the operating data of each graphics processor; determining a graphics processor set in the target system corresponding to a target training task, and determining a second abnormal graphics processor in the graphics processor set based on the operating data of the graphics processors in the graphics processor set; training a preset abnormality judgment model based on the operating data of the first abnormal graphics processor and the second abnormal graphics processor to obtain a target abnormality judgment model, and updating the warning rules based on the target abnormality judgment model, wherein the target abnormality judgment model is used to determine a third abnormal graphics processor in the graphics processors.
[0090] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 7 shows a hardware structure block diagram of an electronic device for implementing a method for determining an abnormal processor. As shown in Figure 7, the electronic device 70 may include one or more (702a, 702b, ..., 702n are used in the figure to illustrate) processors 702 (the processor 702 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 704 for storing data, and a transmission module 706 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that the structure shown in Figure 7 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the electronic device 70 may also include more or fewer components than shown in Figure 7, or have a configuration different from that shown in Figure 7.
[0091] It should be noted that the one or more processors 702 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the electronic device 70. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0092] The memory 704 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for determining the exception handler in the embodiment of the present application. The processor 702 executes various functional applications and data processing by running the software programs and modules stored in the memory 704, that is, implementing the above-mentioned method for determining the exception handler. The memory 704 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 704 may further include a memory remotely located relative to the processor 702, and these remote memories may be connected to the electronic device 70 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0093] Transmission device 706 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of computer terminal 70. In one embodiment, transmission device 706 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 706 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0094] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 70 .
[0095] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0096] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0097] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0098] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0100] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for determining an exception handler, comprising: Acquire operating data of each graphics processor in the target system, and determine a first abnormal graphics processor based on a preset warning rule and the operating data of each graphics processor; Determining a graphics processor set in the target system corresponding to a target training task, and determining a second abnormal graphics processor in the graphics processor set based on operating data of the graphics processors in the graphics processor set; A preset abnormality judgment model is trained based on the operating data of the first abnormal graphics processor and the second abnormal graphics processor to obtain a target abnormality judgment model, and the warning rule is updated based on the target abnormality judgment model, wherein the target abnormality judgment model is used to determine a third abnormal graphics processor in the graphics processor.
2. The method for determining an exception handler according to claim 1, wherein: The step of determining the first abnormal graphics processor according to the preset warning rules and the operating data of the graphics processor includes: Determine the warning data type according to the preset warning rules; Determining, in the operating data, early warning operating data corresponding to the early warning data type and an abnormality judgment rule corresponding to the early warning operating data; The warning operation data is judged according to the abnormality judgment rule, thereby determining the first abnormal graphics processor.
3. The method for determining an exception handler according to claim 2, wherein: The early warning operation data includes first-category early warning operation data, which is data used to determine whether the graphics processor is faulty based on a preset threshold. The step of judging the early warning operation data based on the abnormality judgment rule to determine the first abnormal graphics processor includes: The preset threshold corresponding to the first type of warning operation data is determined, and the first abnormal graphics processor is determined in each of the graphics processors according to a comparison result between the first type of warning operation data corresponding to each of the graphics processors and the preset threshold.
4. The method for determining an exception handler according to claim 2, wherein: The early warning operation data further includes second-category early warning operation data, wherein the second-category early warning operation data is data for determining whether the graphics processor is faulty based on a distribution of the second-category early warning operation data of each graphics processor in the target system; The step of judging the warning operation data according to the abnormality judgment rule to thereby determine the first abnormal graphics processor includes: Based on the distribution of the second-category warning operation data of each of the graphics processors in the target system, the abnormal value interval corresponding to the second-category warning operation data is determined, and the graphics processor corresponding to the second-category warning data located in the abnormal value interval is determined to be the first abnormal graphics processor.
5. The method for determining an exception handler according to claim 1, wherein: The step of determining the second abnormal GPU in the GPU set according to the operating data of the GPUs in the GPU set includes: Determining an operating data set corresponding to the graphics processor set, wherein the operating data set includes operating data corresponding to each graphics processor in the graphics processor set at multiple data collection time points, the multiple data collection time points being time points before a current detection time point, and the current detection time point being a time point at which the second abnormal graphics processor in the graphics processor set is determined based on the operating data of the graphics processors in the graphics processor set; Determining a weight of each operating data according to the data collection time point corresponding to each operating data, wherein the closer the data collection time point is to the current detection time point, the greater the weight; Determining detection data corresponding to each graphics processor according to the weight of the operating data and the operating data set; The second abnormal graphics processor is determined in the graphics processor set according to each of the detection data.
6. The method for determining an exception handler according to claim 5, wherein: The step of determining the second abnormal GPU in the GPU set according to each of the detection data includes: Determine the target anomaly partitioning tree model; Processing the detection data through the target anomaly partitioning tree model to determine anomaly detection data corresponding to each of the graphics processors in the graphics processor set; Determining an anomaly score of each graphics processor based on the anomaly detection data corresponding to each graphics processor; The second abnormal GPU is determined in the set of GPUs according to each of the abnormality scores.
7. The method for determining an exception handler according to claim 6, wherein: The determining the second abnormal GPU in the set of GPUs according to each abnormality score includes: A graphics processor corresponding to an abnormality score greater than an abnormality threshold is determined as the second abnormal graphics processor.
8. The method for determining an exception handler according to claim 1, wherein: The target abnormality judgment model includes a first basic learner, a second basic learner, a third basic learner and a meta-learner, wherein the first basic learner includes a support vector machine, the second basic learner includes a gradient boosting decision tree, the third basic learner includes a fully connected neural network, and the meta-learner includes a logistic regression model.
9. The method for determining an exception handler according to claim 8, wherein: The step of updating the warning rule according to the target abnormality judgment model includes: determining the importance of each type of the operating data according to the second base learner; The warning rule is updated according to the importance, wherein updating the warning rule includes: updating the operation data type in the warning rule.
10. A device for determining an exception handler, comprising: a first processing module, configured to obtain operating data of each graphics processor in the target system, and determine a first abnormal graphics processor based on a preset warning rule and the operating data of each graphics processor; a second processing module, configured to determine a graphics processor set in the target system corresponding to a target training task, and determine a second abnormal graphics processor in the graphics processor set based on operating data of the graphics processors in the graphics processor set; a third processing module, configured to train a preset abnormality judgment model based on the operating data of the abnormal graphics processor to obtain a target abnormality judgment model, and update the warning rule based on the target abnormality judgment model, wherein the target abnormality judgment model is used to determine a third abnormal graphics processor in the graphics processor.
11. A non-volatile storage medium having a program stored therein, wherein: When the program is running, the device where the non-volatile storage medium is located is controlled to execute the method for determining the exception handler according to any one of claims 1 to 9.
12. An electronic device comprising: A memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the method for determining an abnormality handler according to any one of claims 1 to 9 is executed when the program is run.