Hardware status monitoring method and monitoring system during model training

Abstract

The embodiment of the present application provides a method and system for monitoring the hardware status during the model training process, and the monitoring method includes: for each first execution stage of the first training task, during the process of the tested hardware executing the first execution stage, the expected resource utilization range of the tested hardware during the execution of the first execution stage is determined by checking the resource utilization model corresponding to the first execution stage; when it is monitored that the actual resource utilization of the tested hardware is not within the expected resource utilization range, it is determined that the tested hardware is abnormal. By applying the embodiment of the present application, accurate monitoring of the operating status of the tested hardware can be achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and system for monitoring hardware status during model training. Background Art

[0002] In the process of model training through servers and other devices, in order to ensure the stability and reliability of the training process, it is necessary to monitor the status of key hardware in the training equipment. In the monitoring process, the resource usage in the hardware is generally used as a monitoring indicator. When the resource usage of the hardware is abnormal, it generally means that there is a problem with the training task.

[0003] In a related technology, a fixed resource usage threshold is set for the hardware under test. When the actual resource usage of the hardware under test is lower or higher than the preset threshold, the training is considered abnormal. The problem with this solution is that the types of operations that the hardware under test needs to participate in when performing training tasks are diverse, so the amount of resources required by the hardware under test in the normal execution of training tasks is also variable, making it difficult to set a reasonable resource usage threshold.

[0004] In another related technology, historical data of the hardware under test executing training tasks is collected, and the resource usage range of the hardware under test in the process of executing training tasks is generated based on the historical resource usage of the hardware under test, and the obtained resource usage range is expanded in time series to determine whether there are any abnormalities in the resource usage of the device under test during the subsequent execution of training tasks. The problem with this solution is that although the model training process has a strong regularity, due to various unstable factors such as delays and fluctuations in the training process, the entire model training process is not a strictly periodic behavior. Therefore, the resource usage range of the hardware under test in the future period given based on historical resource usage is often inaccurate, making it difficult to achieve effective abnormal state monitoring.

[0005] In summary, there is an urgent need to propose a technical solution that can achieve accurate status monitoring for the key hardware of training equipment. Summary of the invention

[0006] The purpose of the embodiment of the present application is to provide a method and system for monitoring the hardware status during the model training process, so as to accurately monitor the operating status of the tested hardware. The specific technical solution is as follows:

[0007] In a first aspect, the present application provides a method for monitoring hardware status during model training, comprising:

[0008] For each first execution stage of the first training task, during the process of the hardware under test executing the first execution stage, determining an expected resource usage range of the hardware under test during the process of executing the first execution stage by using a resource usage verification model corresponding to the first execution stage;

[0009] When it is monitored that the actual resource usage rate of the tested hardware is not within the expected resource usage rate range, it is determined that the tested hardware is abnormal.

[0010] In a second aspect, the present application provides a method for establishing a resource utilization rate verification model, comprising:

[0011] Obtaining each first execution phase included in a first training task assigned to the hardware under test;

[0012] For each of the first execution stages, determining an expected resource usage curve of the hardware under test during execution of the first execution stage;

[0013] Based on the expected resource usage rate curve corresponding to each first execution stage, generating the expected resource usage rate range of the hardware under test during the execution of each first execution stage, and obtaining the resource usage rate verification model corresponding to each first execution stage;

[0014] The resource usage rate checking model is used to monitor whether there is any abnormality in the actual resource usage rate of the tested hardware during the execution of the corresponding first execution phase.

[0015] In a third aspect, the present application provides a hardware status monitoring system during a model training process, including a device under test, a training task scheduling component, a model building component, a monitoring management component, and a monitoring execution component;

[0016] The training task scheduling component is used to send the training task to the device under test, and send a stage notification message to the monitoring management component when monitoring that the training task enters each first execution stage;

[0017] The device under test includes a usage rate collection component and at least one hardware under test; the hardware under test is used to perform training tasks, and the usage rate collection component is used to collect resource usage rates of each hardware under test in the device under test;

[0018] The model building component is used to determine the resource utilization verification model corresponding to each tested hardware of the tested device, and report the determined resource utilization verification model to the monitoring management component for storage; the resource utilization verification model corresponding to each of the tested hardware includes: the resource utilization verification model corresponding to each first execution stage included in the training task;

[0019] The monitoring management component is used to send a resource usage verification model corresponding to the current first execution stage of each of the tested hardware to the monitoring execution component in response to the stage notification message issued by the training task scheduling component;

[0020] The monitoring execution component is used to obtain the resource usage rate collected by the usage rate collection component for each hardware under test, determine the expected resource usage rate range of each hardware under test through the resource usage rate verification model issued by the monitoring management component, and monitor the resource usage rate of each hardware under test through the determined expected resource usage rate range; when it is monitored that the resource usage rate of any hardware under test is not within the expected resource rate usage range, it is determined that the hardware under test has an abnormality.

[0021] Beneficial effects of this application:

[0022] The hardware status monitoring method and monitoring system during the model training process provided in the present application divide the process of the tested hardware executing the training task into multiple execution stages. For each first execution stage in the training task, the resource utilization verification model corresponding to the first execution stage is used to determine the expected resource utilization range of the tested hardware during the execution of the first execution stage. The actual resource utilization of the tested hardware during the first execution stage is targetedly monitored through the determined expected resource utilization range.

[0023] Therefore, in actual application scenarios, even if the actual duration of each execution stage experienced by the hardware under test in the process of executing the training task may fluctuate due to actual factors, since the present application determines the corresponding expected resource utilization range through the resource utilization verification model corresponding to the execution stage each time the training task enters a new execution stage, it is possible to ensure that the expected resource utilization range of the current application is always compatible with the current execution stage of the training task being executed by the hardware under test, thereby enabling accurate monitoring of the operating status of the hardware under test.

[0024] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0026] Figure 1 It is a schematic diagram of technical solution 1 in the related art;

[0027] Figure 2 It is a schematic diagram of technical solution 2 in the related art;

[0028] Figure 3 A flowchart of a method for monitoring hardware status during model training provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the model training process and the corresponding theoretical resource utilization rate curve provided in the embodiment of the present application;

[0030] Figure 5 A schematic diagram of a resource utilization rate verification model provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of monitoring the operating status of the tested hardware based on the resource usage rate verification model provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of regularizing a historical resource usage rate curve provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of an initial resource utilization rate curve in the iterative calculation phase provided by an embodiment of the present application;

[0034] Fig. 9 A schematic diagram of an expected resource utilization rate curve in the iterative calculation phase provided by an embodiment of the present application;

[0035] Fig.10 A schematic diagram of a processing flow of an iterative calculation phase in the related art;

[0036] Fig.11 A schematic diagram of an expected resource utilization rate curve provided in an embodiment of the present application;

[0037] Fig.12 A schematic diagram of correcting an expected resource utilization rate curve provided in an embodiment of the present application;

[0038] Fig.13 A schematic diagram of a process for establishing a resource utilization rate verification model provided in an embodiment of the present application;

[0039] FIG14( a ) is a schematic diagram of a hardware status monitoring system during a model training process provided by an embodiment of the present application;

[0040] FIG14( b ) is another schematic diagram of a hardware status monitoring system during a model training process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0042] As the scale of artificial intelligence models grows and more training data is used in the model training process, the training of a large number of artificial intelligence models is basically carried out through distributed training. Distributed training means splitting the training task for the model to be trained into multiple subtasks, and distributing these subtasks to different node devices in the server cluster for execution. Generally speaking, the node devices in the server cluster generally include servers for performing calculations and switches for realizing data exchange between servers.

[0043] When a node device performs a model training task, a variety of hardware may be involved in the training process. Taking the server as an example, the hardware involved in the training process in each server includes computing acceleration cards such as GPU (Graphic Processing Unit), network card, CPU (Central Processing Unit) and memory.

[0044] During the model training process, in order to ensure the stability and reliability of the training process, it is usually necessary to monitor the operating status of each key hardware of each node device in the server cluster. The monitoring indicator is generally the resource usage of the hardware, and for a single hardware, it may be necessary to monitor the resource usage of multiple different types of resources. For example, for a single GPU, it may be necessary to monitor the usage of its computing resources and communication resources. When abnormal hardware resource usage is detected, generally speaking, the related training tasks have failed. It is necessary to promptly investigate the cause of the failure, isolate the fault and perform recovery to minimize the time that training is interrupted due to the failure.

[0045] It can be seen that quickly discovering faults by performing status detection on the hardware in the node device is crucial to ensuring the normal execution of the training task. The following is an explanation of two technical solutions for monitoring the operating status of the hardware in the node device in the related art:

[0046] Technical Solution 1: Set a fixed resource utilization threshold. When the hardware resource utilization is lower or higher than the preset threshold, the training is considered abnormal.

[0047] by Figure 1As an example, the actual resource usage collected for a piece of hardware during a certain period of time is shown, with the horizontal axis representing time T and the vertical axis representing resource usage. A fixed maximum resource usage and a fixed minimum resource usage are set for the hardware, which together constitute the resource usage range of the shaded portion of the diagram. When the actual resource usage collected exceeds the resource usage range shown in the diagram, it is considered that the training is abnormal.

[0048] However, in actual application scenarios, the resource usage characteristics of hardware are not fixed during the execution of training tasks. Taking the usage of network traffic resources as an example, there are usually multiple iterations in a training task, and each iteration involves a calculation process and a data synchronization process between node devices. During the calculation process, the traffic between cards is almost zero, and during the data synchronization process, short-term high-bandwidth burst traffic will occur. Therefore, if the usage of network traffic resources is monitored according to Technical Solution 1, it is impossible to effectively determine whether it is an abnormal state when the traffic usage rate is 0.

[0049] Technical Solution 2: Collect historical data on hardware resource usage, generate a historical resource usage range for the hardware during the training task based on the historical resource usage of the hardware, and add a certain offset to generate a subsequent resource usage range as an abnormality judgment condition. The resource usage range used as an abnormality judgment condition is extended in time series to determine whether there is an abnormality in the resource usage of the device under test during the subsequent training task.

[0050] by Figure 2 For example, the horizontal axis is time T, and the vertical axis is resource utilization, where two different shaded areas are shown. One part of the shaded area is the historical resource utilization range based on the historical resource utilization of the hardware. By adding a certain offset on this basis, the resource utilization range indicated in the other shaded area of ​​the figure as the abnormal judgment condition can be obtained. By extending the abnormal judgment condition in time series, the subsequent resource utilization of the hardware can be monitored. When the actual resource utilization of the hardware exceeds the resource utilization range represented by the abnormal judgment condition, the training is considered to be abnormal.

[0051] However, in actual application scenarios, although the model training process has a strong regularity, due to various unstable factors such as delays and fluctuations in the training process, the entire model training process is not a strictly periodic behavior. Therefore, the resource usage rate range given as an abnormal judgment condition based on historical resource usage is often inaccurate. For example, if the preset duration of a single iterative calculation is 15 seconds, and the historical resource situation collected by the hardware is also a model parameter update after 15 seconds of calculation, however, in a subsequent training cycle, it may be that the model parameter update is started after 16 seconds of iterative calculation. If technical solution 2 is adopted, the historical resource usage during the model parameter update process may be used to determine whether there is an abnormality in the resource usage of the last 1 second of the 16-second iterative calculation. Since the amount of resources required for the model parameter update process and the iterative calculation process is different, it is easy to misjudge the abnormal state.

[0052] Therefore, in the process of model training, how to accurately monitor the abnormal status of key hardware of node devices is still a technical difficulty. In view of this, the embodiment of the present application provides a method for monitoring the hardware status in the process of model training, see Figure 3 , the method comprising:

[0053] Step 101: For each first execution stage of the first training task, during the process of the hardware under test executing the first execution stage, the expected resource utilization range of the hardware under test during the execution of the first execution stage is determined by a resource utilization verification model corresponding to the first execution stage.

[0054] The hardware under test specifically refers to the hardware in the device that executes the training task and needs to participate in the training task. For example, the hardware under test can be the GPU, CPU, etc. in the server. The first training task can be understood as the model training task assigned to the device where the hardware under test is located, and the first execution stage can be any execution stage in the first training task.

[0055] In an embodiment of the present application, the entire execution process of the training task will be specifically divided into multiple execution stages, and for each of the execution stages, a resource utilization verification model applicable to the execution stage will be configured. Among them, the resource utilization verification model corresponding to an execution stage is used to give the expected resource utilization range of the hardware under test during the execution of the execution stage. The expected resource utilization range can be specifically understood as: when there is no abnormality in the execution of the training task, the reasonable resource utilization range of the hardware under test during the execution of the execution stage. It should be understood that each execution stage actually has a certain execution time, so the resource utilization verification model corresponding to an execution stage should actually give: the expected resource utilization range corresponding to each time point in the execution of the hardware under test during the execution of the execution stage, and the expected resource utilization range corresponding to different time points may be different.

[0056] Specifically, in the process of the hardware under test actually executing the training task, it will go through multiple training cycles, and each training cycle will include several execution stages. The various execution stages involved in the whole process can be divided into multiple different categories of execution stages, and the expected resource utilization range of the hardware under test in the process of executing the same type of execution stage is consistent (the execution stages can be divided according to actual needs, for example, in the embodiments later in this application, two types of execution stages are included: iterative calculation stage, model and training state preservation stage). In one example, a resource utilization verification model can be determined for each type of execution stage, and the resource utilization verification model is used to give the expected resource utilization range of the hardware under test in the process of executing this type of execution stage. When the hardware under test enters an execution stage each time subsequently, the specific category of the execution stage can be determined, and the resource utilization verification model determined for the execution stage of this category is used as the resource utilization verification model corresponding to the execution stage.

[0057] Exemplarily, for each execution stage, the historical resource usage of the tested hardware when executing the training task can be obtained, and the corresponding resource utilization rate verification model can be set in combination with the historical data. Alternatively, the corresponding resource utilization rate verification model can be obtained by theoretical calculation in combination with the actual amount of resources required for the execution stage and the amount of resources that the tested hardware can provide. The specific method for determining the resource utilization rate verification model is not limited in the embodiments of the present application.

[0058] In practical applications, the entire process of executing a model training task can usually be divided into multiple cycles, and the training process in each cycle is basically the same. Thus, in an embodiment of the present application, the training process in a cycle can be divided into multiple execution stages in combination with actual needs, and corresponding resource utilization verification models can be designed for the multiple execution stages. The designed resource utilization verification model can be subsequently applied to the corresponding execution stages of each cycle. In addition, those skilled in the art will understand that in the entire process of executing a model training task, a variety of different types of operation processes will be involved, such as calculation processes, model parameter update processes, model saving processes, etc. Thus, in one example, a cycle can be divided into multiple execution stages according to the types of operations involved in the training process in combination with the actual characteristics of the set training scheme.

[0059] Step 102: When it is monitored that the actual resource usage rate of the hardware under test is not within the determined expected resource usage rate range, it is determined that an abnormality exists in the hardware under test.

[0060] After establishing the resource usage rate verification model corresponding to each execution stage included in the training task, the status monitoring of the hardware under test can be realized based on the established resource usage rate verification model. In the embodiment of the present application, the corresponding resource usage rate verification model is selected as the target monitoring model according to the execution stage of the training task being executed by the hardware under test, and the resource usage of the hardware under test is monitored through the target monitoring model to see if there is any abnormality.

[0061] In an embodiment of the present application, in order to ensure timely switching of the target monitoring model, the execution stage of the training task can be monitored during the execution of the training task by the hardware under test. When it is monitored that the training task enters an execution stage, the resource utilization verification model corresponding to the execution stage is used as the target monitoring model.

[0062] During a certain execution phase of the tested hardware executing the training task, when it is monitored that the actual resource usage of the tested hardware is not within the expected resource usage range given by the target monitoring model, it is considered that the tested hardware is abnormal. As mentioned above, the resource usage verification model corresponding to an execution phase actually needs to give the expected resource usage ranges corresponding to each time point in the execution phase. Therefore, in the embodiment of the present application, when the tested hardware is executing to the time point t in the current execution phase, it is specifically determined whether the tested hardware is abnormal based on the expected resource usage range corresponding to the time point t in the execution phase given by the target monitoring model.

[0063] For example: Assume that a cycle in which the hardware under test executes a certain training task is divided into three execution stages: stage 1, stage 2, and stage 3. A resource utilization verification model A is designed to give the expected resource utilization range of the hardware under test during the execution of stage 1, a resource utilization verification model B is used to give the expected resource utilization range of the hardware under test during the execution of stage 2, and a resource utilization verification model C is used to give the expected resource utilization range of the hardware under test during the execution of stage 3. Then, during the process of the hardware under test executing the training task, when the training task enters stage 1, the resource utilization verification model A is used as the target monitoring model, and the resource utilization verification model A is used to monitor whether there is any abnormality in the resource utilization of the hardware under test. In the specific monitoring process, when the hardware under test executes to the nth second of stage 1, if the actual resource utilization collected is within the expected resource utilization range given by the resource utilization verification model A for the nth second of stage 1, it is considered that there is no abnormality in the operating status of the hardware under test; if the actual resource utilization collected is not within the expected resource utilization range given by the resource utilization verification model A for the nth second of stage 1, it is considered that there is an abnormality in the operating status of the hardware under test; similarly, when the training task enters the subsequent stage 2, the resource utilization verification model B is switched to the new target monitoring model; when the training task further enters the subsequent stage 3, the resource utilization verification model B is switched to the new target monitoring model.

[0064] In one example, when an abnormality is detected in the hardware under test, the device where the hardware under test is located can be notified to stop executing the training task, and a corresponding reminder message can be generated to remind relevant responsible personnel to handle the abnormality.

[0065] When the above method provided by the present application is applied to the application scenario of distributed training, in order to ensure the stable operation of the training process, one or more tested hardware that needs to be monitored in each node device in the server cluster can be specifically determined. For each piece of tested hardware, the various execution stages included in the training task that needs to be executed are determined, and a resource utilization verification model corresponding to each execution stage is designed, and the operating status of the tested hardware is monitored in combination with the above method provided in the embodiment of the present application. In addition, as explained in the foregoing, for some of the tested hardware, it may be necessary to monitor the usage of multiple types of resources used by it. In this case, it is necessary to configure a resource utilization verification model corresponding to each execution stage for each type of resource, and monitor each type of resource separately. For example, if the hardware under test is a GPU, and a cycle of a training task executed by the GPU is divided into three execution stages, namely stage 1, stage 2 and stage 3, then when it is necessary to monitor both the computing resource usage and the communication resource usage of the GPU, a resource rate usage model A1, a resource usage rate verification model B1 and a resource usage rate verification model C1 can be configured for the computing resource usage, and the expected computing resource usage ranges of the GPU during the execution stages 1-3 are respectively given by the resource usage rate verification models A1-C1, and the computing resource usage of the GPU is monitored based on the resource usage rate verification models A1-C1; and a resource rate usage model A2, a resource usage rate verification model B2 and a resource usage rate verification model C2 are configured for the communication resource usage, and the expected communication resource usage ranges of the GPU during the execution stages 1-3 are respectively given by the resource usage rate verification models A2-C2, and the communication resource usage of the GPU is monitored based on the resource usage rate verification models A2-C2.

[0066] The hardware status monitoring method during the model training process provided in the embodiment of the present application, for each first execution stage in the training task, will determine the expected resource utilization range of the hardware under test during the execution of the first execution stage through the resource utilization verification model corresponding to the first execution stage, and use the determined expected resource utilization range to perform targeted monitoring of the actual resource utilization of the hardware under test during the first execution stage.

[0067] Therefore, in actual application scenarios, even if the actual duration of each execution stage experienced by the hardware under test in the process of executing the training task may fluctuate due to actual factors, since the present application determines the corresponding expected resource utilization range through the resource utilization verification model corresponding to the execution stage each time the training task enters a new execution stage, it is possible to ensure that the expected resource utilization range of the current application is always compatible with the current execution stage of the training task being executed by the hardware under test, thereby enabling accurate monitoring of the operating status of the hardware under test.

[0068] In one embodiment of the present application, the entire execution process of the training task is divided into two types of execution stages: an iterative calculation stage and a model and training state preservation stage. Depending on the preset training scheme, the number of iterative calculation stages and the number of model and training state preservation stages included in the training task are not fixed, but according to general rules, the training task will include multiple training cycles, and the training cycle includes one or more iterative calculation stages and one model and training state preservation stage.

[0069] In the actual model training process, the training samples are generally divided into multiple batches, and only one batch of training samples is input into the model for iterative calculation each time, and the model parameters are updated based on the calculation results of this batch, and then the next batch of training samples is input into the updated model for calculation. In an embodiment of the present application, the process of inputting a batch of training samples into the model for calculation and updating the model parameters based on the calculation results of the batch is called an iterative calculation phase. In actual applications, the operations involved in an iterative calculation phase may include calculations (such as forward calculation operations), communications (such as operations to obtain data required for calculations from other devices), and other operations.

[0070] In addition, in the actual model training process, a checkpoint is usually set after a certain number of iterative calculation stages, so that the model state and model parameters can be periodically saved through the checkpoint as a backup in case of model training failure. At each checkpoint, other training-related information such as model parameters will be saved. In the embodiment of the present application, the process of saving these training-related information at a checkpoint is called a model and training state saving stage.

[0071] In actual applications, the resource usage of the tested hardware in each iterative calculation stage is basically the same, and the resource usage of the tested hardware in each model and training state preservation stage is also basically the same. Taking this into account, in an embodiment of the present application, two resource rate usage models can be set only for the entire training process of the training task: one is a first model configured for the iterative calculation stage, which is used to give the expected resource usage range of the tested hardware in the process of executing a single iterative calculation stage; the other is a second model configured for the model and training state preservation stage, which is used to give the expected resource usage range of the tested hardware in the process of executing a single model and training state preservation stage.

[0072] During the process of the hardware under test executing the first training task, when the first training task enters any iterative calculation stage, the first model will be used to determine the pre-fetch resource usage range of the hardware under test during the iterative calculation process, and the determined pre-fetch resource usage range will be used to monitor the operating status of the hardware under test during the iterative calculation stage; when the first training task enters any model and training state preservation stage, the second model will be used to determine the pre-fetch resource usage range of the hardware under test during the model and training state preservation stage, and the determined pre-fetch resource usage range will be used to monitor the operating status of the hardware under test during the model and training state preservation stage.

[0073] Therefore, in the embodiment of the present application, only based on the first model and the second model of the resource utilization rate verification model, the operation status of the tested hardware in the whole process of executing the first training task can be fully monitored. The data of the resource utilization rate verification model to be configured is small, and the method is easy to implement.

[0074] It is worth noting that when a training task includes multiple consecutive iterative calculation stages within a training cycle, although the resource utilization verification models used in these multiple iterative calculation stages are all the first model, the operating status monitoring of the hardware under test in these multiple iterative calculation stages is still significantly different from the technical solution 2 in the related technology mentioned above.

[0075] As mentioned above, due to various unstable factors in the model training process, the entire model training process is not a strictly periodic behavior. For example, if the execution time of a single iterative calculation phase is set to 15 seconds, but in the process of the hardware under test actually executing the training task, the actual execution time of each single iterative calculation phase will not be fixed at 15 seconds, but may be 15.3 seconds, 15.8 seconds, etc., and the actual execution time of different iterative calculation phases may be different.

[0076] If the second technical solution in the related technology is adopted, then if a resource usage verification model is established based on the historical resource usage of the tested hardware in training cycle a, and the first iterative calculation phase of training cycle a is only executed for 14.5 seconds, and the resource usage of the tested hardware in cycle b is monitored based on the established resource usage verification model, and the tested hardware is executed for 15.5 seconds in the first iterative calculation phase of cycle b, then the resource usage of the tested hardware at the beginning of the second iterative calculation phase of training cycle a will be used to judge whether there is an abnormality in the resource usage of the tested hardware at the end of the first iterative calculation phase of training cycle b, which is easy to misjudge. Moreover, as time goes by, the misalignment between the execution phase of the resource usage data used as the judgment standard in the resource usage model in the second technical solution and the actual execution phase of the training task executed by the tested hardware will continue to accumulate.

[0077] In the embodiment of the present application, if the first model is established based on the situation that the execution time of a single iterative calculation stage is 15 seconds, then when monitoring the process of the hardware under test executing a certain training cycle, even if the first iterative calculation stage is only executed for 14.5 seconds, after entering the second iterative calculation stage, the first model will be used again to determine the pre-fetch resource usage range of the hardware under test in the second iterative calculation process, and the resource usage of the hardware under test will be monitored on one side. In other words, for the second execution stage, the expected resource usage range of the hardware under test at time point t in the iterative calculation stage given by the first model will also be used to determine whether there is an abnormality in the actual resource usage of the hardware under test at time point t in the second execution stage, and the problem mentioned in the previous paragraph will not occur. Therefore, in the embodiment of the present application, even if the unstable factors existing in the training process cause a certain degree of deviation in the resource usage of the hardware under test at specific time points in different execution stages, since the target monitoring model will be reconfirmed when entering each execution stage, it is equivalent to re-aligning the timing between the actual model training process and the resource utilization verification model when entering each execution stage. The timing deviation problem existing in the above-mentioned technical solution 2 will be constrained within a very small range, which can be overcome by reasonably setting the interval size given by the expected resource utilization range.

[0078] The following is an example: In this example, the idea of ​​tensor parallelism and pipeline parallelism is adopted to split the training process of a certain artificial intelligence model into eight training tasks, and the eight training tasks are assigned to GPU0-GPU7 for processing. The training process in one training cycle is as follows: Figure 4 As shown in (a) in . For ease of understanding, the training process is briefly described first.

[0079] Specifically, pipeline parallelism refers to dividing the AI ​​model between layers, dividing the entire model into multiple linearly connected parts in sequence, and placing different parts on different devices for training. Due to the sequential dependency between different parts, for a batch of data, the data of the batch is first input into the first part of the sequence for forward calculation, and then the calculation result of the first part is input into the second part of the sequence for forward calculation, and so on; the back propagation process is to first perform back propagation of the gradient through the last part of the sequence, and then further back propagate through the second to last part of the sequence, and so on.

[0080] Tensor parallelism refers to dividing the parameters within the layer of the artificial intelligence model into multiple parallel blocks, and placing different blocks on different devices for training. For a batch of data, the batch of data is divided into multiple sub-batches with the same number of blocks, and different sub-batches are calculated through different blocks, and then the model parameters are uniformly updated.

[0081] for Figure 4 The training process shown in (a) in the figure is to divide the entire model into four parts that are linearly connected in sequence according to the idea of ​​pipeline parallelism, and further divide the model parameters of each part into two parallel blocks according to the idea of ​​tensor parallelism. The two blocks of the first part are placed in GPU0 and GPU1 respectively, the two blocks of the second part are placed in GPU2 and GPU3 respectively, the two blocks of the third part are placed in GPU4 and GPU5 respectively, and the two blocks of the fourth part are placed in GPU6 and GPU7 respectively. The F in the figure refers to the forward calculation process, B refers to the back propagation process, the first subscript of F and B refers to the number of the model part divided based on the pipeline parallel idea, the second subscript refers to the number of the block in each part, update refers to the model parameter update process, and save refers to the model save process of the checkpoint.

[0082] Taking the communication resource usage of GPU6 as an example, in an iterative calculation phase, it needs to calculate the communication resource usage of GPU6 in the forward calculation process F. 3,0 Receive the output data of the previous level of pipeline before, and 3,0 Then, the back propagation process B starts with the output result of GPU7 3,0 , and in B 3,0 After that, the gradient is further propagated to the previous pipeline and also needs to participate in the model parameter update process; and GPU6 also needs to participate in the model and training state preservation stage. Therefore, the theoretical communication resource utilization curve of GPU6 in a training cycle can be obtained as follows Figure 4As shown in (b) (the resource utilization curve is used to generate an expected resource utilization range to obtain a resource utilization verification model. The method of generating an expected resource utilization range based on the resource utilization curve will be described later), the horizontal axis is time T and the vertical axis is communication resource utilization.

[0083] If based on Figure 4 The entire curve (b) shows a resource utilization verification model suitable for a complete training cycle. When the communication resource usage of GPU6 is monitored by the resource utilization verification model, the following problems will be encountered: First, the resource utilization verification model has a large amount of data and occupies more resources; second, the resource utilization verification model contains a large amount of repeated information, for example, the data content corresponding to the two iterative execution stages is actually the same; third, it cannot adapt to the parameter changes of the checkpoint setting cycle, for example, Figure 4 In the case of (a), a checkpoint is set every two iterative calculation stages. If, in the subsequent training process, the checkpoint cycle is changed to set a checkpoint every three iterative calculation stages, the model can no longer be used to monitor the subsequent training process.

[0084] According to the embodiment of the present application, it is only necessary to generate a first model for a single iterative calculation stage based on the resource utilization curve of part ① of the diagram, and to generate a second model for a single model and a training state saving stage based on the resource utilization curve of part ② of the diagram. The two resource utilization verification models with smaller data volumes, the first model and the second model, are used to implement the operating status monitoring of GPU6 during the entire training process, which not only helps to reduce the amount of data, but also can adapt to parameter changes in the checkpoint setting cycle.

[0085] In one embodiment of the present application, the resource usage rate verification model corresponding to each execution stage in the training task is obtained based on the following method:

[0086] Step 201: Determine an expected resource usage curve of the hardware under test during each first execution phase of executing a first training task.

[0087] The expected resource usage curve corresponding to an execution phase can be specifically understood as: when the training task is executed without exception, the reasonable resource usage curve of the hardware under test during the execution of the execution phase. The form of the expected resource usage curve can be found in Figure 4 Schematic diagram of (b).

[0088] For example, the expected resource usage rate curve of a stage can be obtained by analyzing the actions that the tested hardware needs to participate in during a stage of execution through theoretical calculation. Figure 4The training process shown in (a) is used to determine the expected resource usage curves of GPU1-GPU6. Alternatively, the historical resource usage curves of the tested hardware in a type of execution phase can be collected by collecting historical data, and the expected resource usage curves corresponding to the type of execution phase can be generated based on the historical resource usage curves.

[0089] Step 202: For each first execution stage, based on the expected resource utilization curve of the tested hardware during the execution of the first execution stage, a preset offset range is added to each data point on the expected resource utilization curve to obtain a resource utilization verification model corresponding to the first execution stage.

[0090] As mentioned above, the model will be affected by various unstable factors during the training process, that is, the actual resource utilization rate of the tested hardware will fluctuate due to various actual situations. Therefore, when monitoring the actual resource utilization rate of the tested hardware, the actual resource utilization rate collected may not be able to fully match the expected resource utilization rate curve of the corresponding stage, but will appear near the expected resource utilization rate curve. In order to avoid misjudgment, in an embodiment of the present application, for each type of execution stage, a certain degree of deviation will be made with the expected resource utilization rate curve of the execution stage as the center, and the expected resource utilization rate range of the execution stage will be obtained, and the resource utilization rate verification model of the execution stage will be obtained.

[0091] The process of performing the offset with the expected resource utilization rate curve as the center can also be understood as: for each point on the expected resource utilization rate curve, a preset offset range is added with the point as the center to obtain the corresponding expected resource utilization rate range. Specifically, the preset offset amount can include two parts: one is the offset range of resource utilization rate ( Figure 4 (b) the offset range in the vertical direction), and (c) the offset range in time ( Figure 4 (b) The offset range in the horizontal direction).

[0092] After the resource utilization verification model is derived, when the actual resource utilization of the hardware under test is subsequently monitored through the resource utilization verification model, a certain sampling frequency can be set to regularly collect the actual resource utilization of the hardware under test to determine whether there is an abnormality in the hardware under test. For the currently collected sampling point, when the actual resource utilization represented by the sampling point is not within the determined expected resource utilization range, it indicates that there is an abnormality in the operating state of the hardware under test. In the embodiment of the present application, it can also be understood as follows: when the offset between a sampling point and the first data point on the expected resource utilization curve exceeds the preset offset range, it indicates that there is an abnormality in the operating state of the hardware under test, wherein the first data point is a point on the expected resource utilization curve that is closest to the first sampling point.

[0093] In an embodiment of the present application, a preset offset range can be set in combination with historical experience. Alternatively, a preset offset range can be set by means of a statistical method. For example, for a certain type of execution phase, the historical resource usage of the tested hardware in multiple such execution phases can be counted, and the preset offset range can be set according to the actual offset between the historical resource usage data points in different execution phases.

[0094] For an example of this: see Figure 5 , assuming that the expected resource usage curve of GPU6 in an iterative calculation phase is as shown in the figure, then after the expected resource usage curve is offset, the expected resource usage range obtained is specifically the shaded part of the figure. In the subsequent periodic sampling of the actual resource usage of GPU6, when the operating status of GPU6 is monitored in combination with the expected resource usage range, Figure 6 In other words, if the current sampling point is within the expected resource usage range, it indicates that the GPU 6 is operating normally; if the current sampling point is not within the expected resource usage range, it indicates that the GPU 6 is operating abnormally.

[0095] In the embodiment of the present application, after obtaining the expected resource utilization curve corresponding to each execution stage of the training task of the tested hardware, for each type of execution stage, the expected resource utilization range is obtained by offsetting with the expected resource utilization period corresponding to the execution stage, and the resource utilization range corresponding to the execution stage is obtained. By reasonably setting the interval size of the offset range used when offsetting, it is possible to ensure that the resource utilization verification model effectively adapts to the fluctuation phenomenon of the actual resource utilization during the model training process, and avoid misjudgment of abnormal status.

[0096] As mentioned above, the expected resource usage curve of the hardware under test at various execution stages of the training task can be determined by collecting historical data or theoretical calculation. The following is an explanation of the former solution:

[0097] Specifically, in the actual model training process, the entire training process for an artificial intelligence model may involve a very large number of training cycles, such as hundreds or thousands of training cycles. Therefore, in practical applications, in the early stage of model training, the historical resource utilization of the hardware under test in the previous or previous training cycles can be collected to obtain the historical resource utilization curve of the hardware under test, and on this basis, a resource utilization verification model corresponding to each execution stage of the training task is generated for the hardware under test. Then, in other subsequent training cycles, the operating status of the hardware under test is monitored based on the generated resource utilization verification model.

[0098] When determining each expected resource usage rate curve corresponding to the hardware under test by collecting historical data, the aforementioned step 201 specifically includes the following sub-steps:

[0099] Step 2011a: Collect a historical resource usage curve of the hardware under test in at least one historical training cycle of executing the first training task.

[0100] In an embodiment of the present application, a sampling frequency can be set based on actual needs, and the actual resource utilization of the hardware under test in at least one historical training cycle can be regularly sampled based on the sampling frequency, and a historical resource utilization curve can be determined based on the sampled data.

[0101] Step 212a: combining the execution stage information of each sampling point in the historical resource utilization rate curve, cutting the historical resource utilization rate curve to obtain the expected resource utilization rate curve corresponding to each first execution stage.

[0102] In practical applications, the current execution stage of the training task executed by the tested hardware can be recorded while sampling the actual resource usage rate of the tested hardware, so that the execution stage of each sampling point can be known. Alternatively, after each execution stage of the training task executed by the tested hardware enters, the first sampling point in the execution stage can be marked. Since each sampling point has a time sequence, the execution stage of each sampling point can be known based on the execution stage of the marked sampling point.

[0103] Therefore, in an embodiment of the present application, after obtaining the historical resource utilization curve of the hardware under test, the historical resource utilization curve is cut according to the execution stage information of each sampling point in the example resource utilization curve (i.e., the execution stage to which the sampling point belongs), so as to obtain the expected resource utilization curve corresponding to the hardware under test in various training stages.

[0104] Exemplarily, when the historical resource utilization data of the tested hardware in a historical training cycle is sampled, the historical resource utilization curve of the tested hardware in a historical training cycle can be obtained. By cutting the historical resource utilization curve in combination with the execution stage information of each sampling point, the expected resource utilization curve corresponding to each type of execution stage of the tested hardware can be directly obtained. If a historical training cycle contains multiple execution stages of the same type (for example, a training cycle may contain multiple iterative calculation stages), then after cutting, each of the multiple execution stages of the same type will correspond to a historical resource utilization curve. In this case, a historical resource utilization curve can be selected as the expected resource utilization curve corresponding to this type of execution stage, or the expected resource utilization curve corresponding to this type of execution stage can be obtained by averaging the sampling points in multiple historical resource utilization curves.

[0105] When sampling the historical resource utilization data of the hardware under test in multiple historical training cycles, after cutting the historical resource utilization curves of the multiple historical training cycles, each type of execution stage will correspond to multiple segments of historical resource utilization curves. For each type of execution stage, you can refer to the description in the previous paragraph and select a segment of historical resource utilization curve from the multiple segments of historical resource utilization curves corresponding to the type of execution stage as the corresponding expected resource utilization curve. Alternatively, you can also average the sampling points in the multiple segments of historical resource utilization curves corresponding to the type of execution stage to obtain the corresponding expected resource utilization curve.

[0106] In one embodiment of the present application, after collecting a historical resource usage curve of the hardware under test in at least one historical training cycle of executing the first training task, the method further includes:

[0107] The historical resource utilization rate curve is regularized to obtain a regularized historical resource utilization rate curve; the regularized historical resource utilization rate curve only includes vertical line segments and horizontal line segments.

[0108] In actual applications, the historical resource utilization rate of the hardware under test is collected to obtain the historical resource utilization rate curve of the hardware under test in at least one training cycle. Since the actual resource utilization rate collected may fluctuate due to various practical factors, the resource utilization rate represented by the sampling points in the historical resource utilization rate curve also fluctuates. However, during the execution of the training task, when the hardware under test is performing a certain calculation process (such as the forward calculation process and the reverse propagation process), the resource utilization rate of the hardware under test in this calculation process is basically stable, and after completing a calculation process, the resources occupied by the calculation process will be directly released (after the calculation resources are released, the calculation resource utilization rate of the hardware under test will directly drop to zero).

[0109] Therefore, the fluctuation phenomenon of the data at each sampling point in the historical resource utilization curve obtained by collecting historical data does not conform to the actual resource usage characteristics of the tested hardware in the process of executing the training task. Therefore, in the embodiment of the present application, the collected historical resource utilization curve can be regularized so that the historical resource utilization curve after regularization only contains horizontal line segments and vertical line segments, which is more in line with the actual resource usage characteristics of the tested hardware. After completing the regularization, the expected resource utilization curve corresponding to the tested hardware in each execution stage can be generated based on the historical resource utilization curve after regularization, and then the resource utilization verification model corresponding to the tested hardware in each execution stage can be obtained, so that the operating status of the tested hardware can be more accurately monitored based on these resource utilization verification models.

[0110] Figure 7 An example is given, Figure 7 The horizontal axis of the middle curve is time, and the vertical axis is resource utilization. It can be seen that there is a fluctuation in resource utilization in the historical resource utilization curve before regularization: for example, the resource utilization will slowly increase to a peak value over time, or slowly decrease over time. In the regularized historical resource utilization curve obtained by regularizing the historical resource utilization curve, the maximum and minimum values ​​are converted into horizontal line segments, which is more in line with the actual resource utilization characteristics of the tested hardware during the training task.

[0111] In one example, a forward alignment algorithm may be used to regularize the historical resource usage curve.

[0112] The collected historical resource utilization rate data is recorded as (t1, p1), (t2, p2), (t3, p3), …, and an exemplary description of the process of achieving forward alignment is given. Among them, for each historical resource utilization rate data (ti, pi), ti is the time point when the utilization rate is collected, and pi is the resource utilization rate collected at the time point ti. If the collected historical resource utilization rate data is directly connected, the expression of the obtained historical resource utilization rate curve is: p(t) = p1, t ≤ t1; p(t) = (p2 - p1) × (t - t1) / (t2 - t1) + p1, t1 < t ≤ t2; p(t) = (p3 - p2) × (t - t2) / (t3 - t2) + p2, t2 < t ≤ t3; …. Among them, t represents time, and p(t) represents the resource utilization rate. Then, after forward alignment, the expression of the regularized historical resource utilization rate curve obtained is specifically: p(t) = p1, t ≤ t1; p(t) = p2, t1 < t ≤ t2; p(t) = p3, t2 < t ≤ t3; ….

[0113] The following describes a solution for determining the expected resource utilization rate curve corresponding to the DUT in each execution stage through theoretical calculation.

[0114] Specifically, during the training process of the model, there will be many calculation processes and non-calculation processes. In the embodiments of the present application, the calculation processes include the forward calculation process and the backpropagation process, and the non-calculation processes include processes other than the forward calculation process and the backpropagation process that will affect the resource utilization rate of the DUT during the execution of the training task, such as data interaction and synchronization between different hardware, parameter update, model state information saving, etc. Among them, the calculation processes constitute the main part of the entire model training process, and the non-calculation processes are often based on the calculation results of the calculation processes. Considering this feature, in an embodiment of the present application, the calculation processes involved in the execution process can be first analyzed theoretically to establish a preliminary resource utilization rate curve, and then the preliminary resource utilization rate curve can be corrected based on the non-calculation processes involved in the training process to obtain the corresponding expected resource utilization rate curve.

[0115] When determining the expected resource utilization rate curve corresponding to the DUT in each execution stage through theoretical calculation, step 201 described above specifically includes the following sub-steps:

[0116] Step 2011b: For each first execution stage included in the first training task, determine the calculation processes that the DUT needs to execute in this first execution stage, and the relative time periods within the execution duration of each calculation process in this first execution stage.

[0117] As pointed out in the previous article, the calculation process includes the forward calculation process and the back propagation process. In actual application scenarios, before training an artificial intelligence model, it is necessary to plan a specific training plan, for example: whether to train the model based on a pipeline parallel strategy, a tensor parallel strategy, or a data parallel strategy, how to divide the entire model process into different training tasks and assign them to different node devices, and the execution order relationship between different training tasks. After the training plan is determined, for each type of training stage in the process of the hardware under test executing the training task, the forward calculation process and back propagation process that the hardware under test needs to execute in this type of training process can be analyzed based on the training plan, as well as the relative time period of each forward calculation process and back propagation process within the execution duration of the execution stage.

[0118] Combination Figure 8 For example, Figure 8 Shows Figure 4 The execution flow in one iteration execution phase in the training cycle (a) is shown in Figure 1. By analyzing the execution flow, it can be concluded that GPU6 needs to execute at time t in one iteration execution phase. 0 To time t 1 Execute the forward calculation process F 3,0 , and at time t 1 to 2 Perform back propagation process B 3,0 .

[0119] In actual application scenarios, there are many possible training schemes for model training, and the actual training process will also be different. Figure 8 However, for each type of training phase, similar results can be analyzed based on the training program adopted. Figure 8 The upper part illustrates the training process, so that based on the training process, it is possible to derive the calculation process that the hardware under test needs to participate in during this type of execution phase, as well as the relative time period of each calculation process during this type of execution phase.

[0120] Step 2012b: Based on the relative time periods corresponding to the calculation processes, the theoretical resource usage of the calculation processes and the total resources of the hardware under test, an initial resource usage curve of the hardware under test during the execution of the first execution phase is calculated.

[0121] After analyzing and deriving the calculation processes that the hardware under test needs to participate in within a type of execution stage, as well as the relative time periods corresponding to each calculation process, the amount of calculation required to be performed by the hardware under test can be determined based on the specific calculation content performed by the hardware under test in each calculation process, and the theoretical resource occupancy of each calculation process can be obtained. On this basis, combined with the actual total amount of resources of the hardware under test, the theoretical resource occupancy rate of the hardware under test in each calculation process can be obtained, and the initial resource utilization curve corresponding to this type of execution stage can be obtained.

[0122] Taking computing resources as an example, for each computing process in the execution phase, the theoretical resource utilization rate of the hardware under test when executing the computing process can be obtained based on the amount of data required by the hardware under test and the actual computing power of the hardware under test, thereby obtaining the initial resource utilization rate curve corresponding to the execution phase.

[0123] Follow up step 2011b for Figure 8 The example of GPU6 performing the forward calculation process F 3,0 With the back propagation process B 3,0 The size of the data required for calculation and the actual computing power of GPU6 are used to obtain the initial resource utilization curve of the computing resources corresponding to GPU6 in the iterative calculation stage. The initial resource utilization curve is as follows: Figure 8 shown in the lower half of .

[0124] Step 2013b: Based on the non-computing processes that the hardware under test needs to execute in the first execution phase, and the relative time periods of the execution duration of each non-computing process in the first execution phase, the initial resource utilization curve is corrected to obtain the expected resource utilization curve of the hardware under test during the execution of the first execution phase.

[0125] Specifically, non-computing processes such as communication involved in the execution phase may cause a temporary interruption of the computing process, or the non-computing process itself needs to use resources, which will affect the resource usage of the tested hardware during a type of execution phase. Therefore, in an embodiment of the present application, after obtaining an initial resource utilization curve corresponding to a type of execution phase, the non-computing processes that the tested hardware needs to participate in during the execution phase can be analyzed in combination with a specific training process, as well as the relative time periods of various types of non-computing processes in the execution phase, and based on this, the initial resource utilization curve corresponding to the execution phase is corrected to obtain the expected resource utilization curve corresponding to the execution phase of the tested hardware.

[0126] Follow up step 2012b for Figure 8 For GPU6, combined with Figure 8The training process of the iterative calculation stage shown in the figure can be analyzed to show that GPU6 and GPU7 perform forward calculation process F respectively based on tensor parallelism. 3,0 With F 3,1 , so that after GPU6 completes the forward calculation process F 3,0 , it is necessary to synchronize the tensor parallel results with GPU7 to execute the subsequent back propagation process B 3,0 In this case, we can analyze the theoretical waiting time of GPU6 waiting for tensor parallel result synchronization, and based on this, Figure 8 The initial resource utilization curve shown in the figure is modified to obtain the expected resource utilization curve corresponding to GPU6 in the iterative calculation stage. Based on this, the expected resource utilization curve corresponding to GPU6 in the iterative calculation stage is obtained. Fig. 9 , it can be seen that Fig. 9 The expected resource utilization curve shown in Figure 8 Based on the initial resource utilization curve shown in , the resource utilization of GPU6 during the process of waiting for tensor parallel result synchronization is deducted.

[0127] In addition, in some application scenarios, when determining the corresponding expected resource utilization curve for a type of execution phase, the process of determining the initial resource utilization curve based on the calculation process may not be involved. For example, when determining the resource utilization curve of communication resources, the hardware under test may not need to use communication resources in the calculation process. In this case, after analyzing the training process, the relative time period of the non-computing process that the hardware under test needs to participate in can be analyzed in combination with the calculation process included in the training process, and the expected resource utilization curve corresponding to this type of execution phase can be determined on this basis. For example, in the previous article, Figure 4 As shown, for the communication resource usage of GPU6, in an iterative calculation phase, it can be concluded that GPU6 needs to spend more time in the forward calculation process F 3,0 Receive the output data of the previous level of pipeline before, and 3,0 Then, the back propagation process B starts with the output result of GPU7 3,0 , and in B 3,0 After that, the gradient is further propagated to the previous level of pipeline, and it also needs to participate in the update process of model parameters, so that the following can be obtained: Figure 4 The expected resource usage curve corresponding to the iterative computing phase of GPU6 shown in (b) is shown in FIG. 1. For another example, the model and training state saving phase does not involve the computing process. For this case, the corresponding expected resource usage curve can be directly determined based on the non-computing process.

[0128] In summary, regarding how to determine the expected resource utilization curve corresponding to the hardware under test at each execution stage of the training task, the embodiments of the present application provide two optional implementation methods: collecting historical data or theoretical calculation. In practical applications, a specific implementation method can be selected based on the actual characteristics of the training task to determine the expected resource utilization curve.

[0129] For example, if the overall training process is relatively simple (e.g. Figure 4 ), we can use theoretical calculation to get an accurate expected resource usage curve. However, in some application scenarios, the training process involved may be complicated. If we use theoretical calculation to determine the corresponding expected resource usage curve, the calculation process will be complicated. For example, Fig.10 Schematic training process, Fig.10 The training process of a single iterative calculation stage is specifically shown, and the corresponding training scheme is: based on the idea of ​​pipeline parallelism, the model to be trained is divided into four parts that are linearly connected in sequence, and the four parts are respectively arranged on four GPUs from GPU-a to GPU-d. In addition, a batch of training data is further divided into eight sub-batches, and the eight sub-batches are staggered in time sequence and input into the pipeline, and the forward calculation process and the back propagation process are staggered in time sequence. It can be seen that if the expected resource utilization curve of GPU-a to GPU-d in the iterative calculation stage is determined based on theoretical calculation, the calculation process will be very complicated. So for Fig.10 For example, the expected resource utilization curve of GPU-a to GPU-d in the iterative calculation phase can be determined by collecting historical data to simplify the process of determining the expected resource utilization curve, thereby simplifying the process of establishing a resource utilization verification model corresponding to the hardware under test.

[0130] And combined with the description in the previous text, it can be seen that when the expected resource utilization curve corresponding to the hardware under test in each execution stage is determined by collecting historical data or theoretical calculation, the time and period for the hardware under test to use specific hardware resources can be accurately obtained. Therefore, a resource utilization verification model corresponding to each execution stage is generated based on the expected resource utilization curve corresponding to each execution stage, and based on the generated resource utilization verification model, when the actual resource usage of the hardware under test in each execution stage is monitored, it helps to more finely monitor the resource usage status of the hardware under test, thereby shortening the fault discovery time. It has been verified experimentally that when monitoring the operating status of the hardware under test based on the technical solution one or technical solution two in the aforementioned related technology, the fault discovery time may be as long as ten minutes to twenty minutes, and when monitoring the operating status of the hardware under test based on the method provided in the embodiment of the present application, the fault discovery time can be shortened to tens of seconds to a few minutes.

[0131] In one embodiment of the present application, the first expected resource usage curve of the tested hardware during the first execution phase of the first training task is determined based on the following method:

[0132] Obtaining a second expected resource usage curve of the reference hardware during a second execution phase of executing the second training task; the second training task and the first training task are associated tasks for the same model to be trained, and the expected resource usage of the reference hardware during the second execution phase is consistent with the expected resource usage of the tested hardware during the first execution phase;

[0133] The total amount of resources of the tested hardware and the total amount of resources of the reference hardware are combined to generate a first expected resource usage curve based on the second expected resource usage curve.

[0134] The second execution stage executed by the reference hardware and the first execution stage executed by the tested hardware can be understood as the same training stage (e.g., iterative calculation stage, or model and training state preservation stage) experienced by the reference hardware and the tested hardware in the process of executing their respective assigned training tasks for the same artificial intelligence model. Specifically, in the actual training process of the artificial intelligence model, the training tasks executed by different hardware may be related to each other, and the expected resource usage of one hardware during a certain execution stage of a training task may be consistent with the expected resource usage of another hardware during the corresponding execution stage of another training task. In an embodiment of the present application, in response to this situation, after determining the expected resource utilization curve of one hardware (reference hardware) in an execution phase (for example, an iterative calculation phase or a model and training state preservation phase), the determined expected resource utilization curve can be directly applied to the other hardware (the hardware under test) to obtain the expected resource utilization curve of the other hardware in the corresponding execution phase (the iterative calculation phase or the model and training state preservation phase), without having to separately determine the corresponding expected resource utilization curve for the other hardware, thereby helping to simplify the process of determining the expected resource utilization curve, and further simplifying the process of establishing a resource utilization verification model corresponding to the hardware under test.

[0135] Specifically, when the expected resource usage curve determined for the reference hardware is applied to the hardware under test, although the resource usage of the two hardware is the same, the total amount of resources of the two hardware may be different, which may lead to a certain difference in the resource usage of the two hardware. Therefore, the expected resource usage curve of the reference hardware can be adjusted by combining the total amount of resources of the hardware under test and the total amount of resources of the reference hardware to obtain the expected resource usage curve of the hardware under test.

[0136] For example, if during the model training process, a switch is connected to a server, and the switch is dedicated to forwarding the data of the server to the outside, or forwarding the external data to the server, then it can be considered that the communication resource usage of the switch and the server during the training process is consistent. Therefore, after determining the expected resource rate curve corresponding to each execution stage of the training process of the server, the expected resource rate curve corresponding to each execution stage of the training process of the switch can be generated based on the expected resource rate curve corresponding to the server.

[0137] In one embodiment of the present application, after generating a first expected resource utilization curve based on the second expected resource utilization curve, when the first monitoring sampling frequency of the hardware under test is lower than the second monitoring sampling frequency of the reference hardware, the generated first expected resource utilization curve can also be corrected so that the corrected first expected resource utilization curve is more suitable for the hardware under test, and the expected resource utilization range corresponding to the hardware under test is subsequently determined based on the corrected first expected resource utilization curve.

[0138] Among them, a monitoring sampling frequency of hardware specifically refers to the sampling frequency of collecting the actual resource utilization rate of the hardware in the process of monitoring the operating status of the hardware through the resource utilization verification model. In actual applications, the sampling process of the actual resource utilization rate of the hardware also needs to occupy its internal resources, so it may affect the training tasks being performed by the hardware. Therefore, if a device itself has strong capabilities, when monitoring the operating status of the hardware in the device, even if a higher sampling frequency is set, it will not have a significant impact on the training tasks being performed by the device, so a relatively high sampling frequency can be set to detect abnormal situations in time. However, if a device itself has weak capabilities, when monitoring the operating status of the hardware in the device, if a higher sampling frequency is set, it may have a significant impact on the training tasks being performed by the device, so only a relatively low sampling frequency can be set. For example, for a server, the server only needs to pay attention to its own network card, and the CPU capability is relatively strong. In actual applications, a sampling frequency of 1Hz (Hertz) can be set to collect the actual communication resource usage of the server network card; while network devices such as switches need to monitor a large number of ports and have relatively weak CPU capabilities, and the set sampling frequency can only reach 0.1 cycles.

[0139] In an embodiment of the present application, if the monitoring sampling frequency set for the hardware under test is lower than the monitoring sampling frequency set for the reference hardware, then after generating the first expected resource utilization rate curve of the hardware under test based on the second expected resource rate curve of the reference hardware, the generated first expected resource utilization rate curve can be corrected to remove unnecessary details in the first expected resource utilization rate curve, so that the corrected first expected resource utilization rate curve can be more suitable for the hardware under test.

[0140] For example: Assume that the capability of the reference hardware is stronger than that of the tested hardware, the monitoring sampling frequency set for the reference hardware is also higher than the monitoring sampling frequency set for the tested hardware, and the first expected resource usage rate curve of the tested hardware generated based on the second expected resource rate curve of the reference hardware is as follows: Fig.11 As shown. It can be seen that the first expected resource utilization curve contains certain details, for example: there is a small depression between point M and point N on the curve, which means that the expected resource utilization of the hardware under test will briefly drop to zero in a certain time period between point M and point N. However, if a smaller sampling frequency is set for the hardware under test, for example, when the actual resource usage of the hardware under test is collected, the time difference between two adjacent sampling points may be greater than the time difference between point M and point N. In this case, the depression in the illustrated curve will not have a significant impact on the monitoring process of the hardware under test, so the illustrated curve can be corrected to remove the details, and the corrected curve can be actually applied to the hardware under test, which helps to simplify the expected resource utilization curve, and further helps to simplify the resource utilization verification model derived based on the expected resource utilization curve, saving the amount of data for the resource utilization verification model.

[0141] In an embodiment of the present application, the process of correcting the first expected resource utilization curve is specifically as follows: determining the mean processing time in combination with the second monitoring sampling frequency set for the hardware under test, and dividing the generated first expected resource utilization curve into multiple sub-curves with a time length equal to the mean processing time according to the mean processing time, performing mean processing on the data points in each sub-curve, and obtaining a corrected first expected resource utilization curve.

[0142] The purpose of this operation is to smooth out the differences between the specific expected resource utilization rates given by the first expected resource utilization rate curve in each mean processing time, so as to simplify the first expected resource utilization rate curve. The mean processing time can be determined in combination with historical experience and actual conditions. For example, if the monitoring sampling frequency set for the hardware under test is low, a relatively long mean processing time can be set; if the monitoring sampling frequency set for the hardware under test is high, a relatively short mean processing time can be set.

[0143] After determining the mean processing duration, the generated first expected resource utilization curve can be specifically divided into multiple sub-curves with a time length equal to the mean processing duration, and mean processing is performed on the data points in each sub-curve to obtain a revised first expected resource utilization curve. Mean processing is performed on the data points in a sub-curve, that is, the mean of the expected resource utilization represented by each data point in the sub-curve is determined, and the expected resource utilization given by the first expected resource utilization curve in the sub-curve is adjusted to the mean. In other words, the revised first expected resource utilization curve will only give a fixed expected resource utilization in each mean processing duration.

[0144] When the second expected resource utilization curve is obtained by collecting historical data, the second expected resource utilization curve is actually composed of a large number of sampling points. After the first expected resource utilization curve is generated on the basis of the second expected resource utilization curve, the mean processing time used when the first expected resource utilization curve is subsequently divided into sub-curves can also be obtained by setting a mean processing number for averaging the sampling points. If the set mean processing number is recorded as n, the process of correcting the first expected resource utilization curve can be specifically understood as: every consecutive n sampling points in the first expected resource utilization curve are divided into a group, and the sampling points of each group constitute a sub-curve. For the n sampling points in each group, determine the mean of the expected resource utilization represented by these n sampling points, adjust the expected resource utilization represented by the first sampling point to the mean, and filter out the data of the last n-1 sampling points.

[0145] Fig.12 An example is given, Fig.12 The horizontal axis of the middle curve is time, and the vertical axis is resource utilization. It shows the expected resource utilization curve before correction, and the schematic diagram of the corrected expected resource utilization curve obtained after the curve is corrected by the above-mentioned mean processing method. It can be seen that the expected resource utilization curve before correction contains more details, and the corrected expected resource utilization curve is significantly simplified.

[0146] Based on the same inventive concept, the present application embodiment also provides a method for establishing a resource utilization rate verification model, see Fig.13 , the method comprising:

[0147] Step 301: Obtain each first execution phase included in a first training task assigned to the hardware under test.

[0148] Step 302: For each first execution stage, determine an expected resource usage curve of the hardware under test during the execution of the first execution stage.

[0149] Step 303: Based on the expected resource usage curve corresponding to each first execution stage, generate the expected resource usage range of the tested hardware during the execution of each first execution stage, and obtain the resource usage verification model corresponding to each first execution stage.

[0150] The expected resource usage rate curve is used to monitor whether there is any abnormality in the actual resource usage rate of the tested hardware during the execution of the corresponding first execution phase.

[0151] For the specific contents and beneficial effects of the method for establishing the resource utilization rate verification model, please refer to the description of the embodiment of the hardware status monitoring method during the model training process in the previous text, which will not be repeated here.

[0152] In some embodiments of the present application, the aforementioned step 301 includes the following sub-steps:

[0153] Step 3011a: Collect a historical resource usage curve of the hardware under test during at least one historical training cycle of executing the first training task.

[0154] Step 3012a: Based on the execution stage information of each sampling point in the historical resource utilization rate curve, the historical resource utilization rate curve is cut to obtain the expected resource utilization rate curve corresponding to each first execution stage.

[0155] In some other embodiments of the present application, the aforementioned step 301 includes the following sub-steps:

[0156] Step 3011b: For each first execution stage included in the first training task, determine the calculation processes that the hardware under test needs to execute in the first execution stage, and the relative time period of each calculation process within the execution duration of the first execution stage.

[0157] Step 3012b: Based on the relative time periods corresponding to the calculation processes, the theoretical resource occupancy of the calculation processes and the total resources of the hardware under test, an initial resource usage rate curve of the hardware under test during the execution of the first execution phase is calculated.

[0158] Step 3013b: Based on the non-computing processes that the hardware under test needs to execute in the first execution phase, and the relative time periods of the execution duration of each non-computing process in the first execution phase, the initial resource utilization curve is corrected to obtain the expected resource utilization curve of the hardware under test during the execution of the first execution phase.

[0159] Based on the same inventive concept, an embodiment of the present application also provides a hardware status monitoring system during a model training process, see Figure 14 (a) and Figure 14 (b), the system includes: a device under test, a training task scheduling component, a model building component, a monitoring management component and a monitoring execution component.

[0160] The training task scheduling component is used to send training tasks to the device under test, and when monitoring that the training task enters each first execution stage, it sends a stage notification message to the monitoring management component.

[0161] The device under test includes a usage rate collection component and at least one hardware under test; the hardware under test is used to execute training tasks, and the usage rate collection component is used to collect resource usage rates of each hardware under test in the device under test.

[0162] The device under test is the hardware device related to model training, such as GPU servers, network switches, storage devices, etc. The hardware under test is the key hardware related to model training in the device under test, such as GPU cards, memory, network cards, etc. In actual applications, for a hardware under test, it may be necessary to monitor the utilization of multiple resources of the hardware under test. For example, for a GPU card, it may be necessary to monitor its computing resource utilization, PCIe (Peripheral Component Interconnect Express, the third-generation peripheral device bus) bandwidth utilization, video memory utilization, etc.

[0163] The usage collection component can be understood as software deployed on the device under test, which has the function of collecting resource usage for the hardware under test. Specifically, the usage collection component can read the relevant resource usage from the hardware under test through the software and hardware interface, obtain the resource usage of the hardware under test, and report the resource usage to the monitoring execution component.

[0164] The model building component is used to determine the resource utilization verification model corresponding to each hardware under test of the device under test, and report the determined resource utilization verification model to the monitoring management component for storage; the resource utilization verification model corresponding to each hardware under test includes: the resource utilization verification model corresponding to each first execution stage included in the training task; the resource utilization verification model is used to give: the expected resource utilization range of the hardware under test during the execution of the corresponding execution stage.

[0165] The monitoring management component is used to respond to the stage notification message sent by the training task scheduling component and send the resource usage verification model corresponding to the current first execution stage of each tested hardware to the monitoring execution component.

[0166] The monitoring execution component is used to obtain the resource usage rate collected by the usage collection component for each hardware under test, determine the expected resource usage rate range of each hardware under test through the resource usage rate verification model issued by the monitoring management component, and monitor the resource usage rate of each hardware under test through the determined expected resource usage rate range; when it is monitored that the resource usage rate of any hardware under test is not within the determined expected resource rate usage range, it is determined that the hardware under test has an abnormality.

[0167] Specifically, the model building component can determine the expected resource utilization curve of the tested hardware during each execution stage of the training task based on the aforementioned method of collecting historical data or theoretical calculations, and on this basis determine the resource utilization verification model corresponding to the tested hardware at each execution stage. In one example, the model building component can directly use the expected resource utilization curve corresponding to the tested hardware in an execution stage as the resource utilization verification model corresponding to the tested hardware in the execution stage. After the monitoring execution component obtains the issued resource utilization verification model, it offsets the expected resource utilization curve as the resource utilization verification model as the center to obtain the expected resource utilization range, and monitors whether the operating status of the tested hardware is abnormal based on the expected resource utilization range. In another example, after obtaining the expected resource utilization curve corresponding to the hardware under test in an execution stage, the model building component can directly offset the curve as the center to obtain the expected resource utilization range, and use the expected resource utilization range as the resource utilization verification model corresponding to the hardware under test in the execution stage. After the subsequent monitoring execution component obtains the issued resource utilization verification model, it can directly monitor whether the operating status of the hardware under test is abnormal based on the expected resource utilization range as the resource utilization verification model.

[0168] The monitoring execution component may be deployed in the device under test as shown in FIG. 14 (a), or may be independently deployed outside the device under test as shown in FIG. 14 (b), depending on actual needs.

[0169] In the scenario of distributed training, a set of model building components, monitoring management components and task scheduling components can be configured for each device under test, or a set of model building components, monitoring management components and task scheduling components can be uniformly configured for multiple devices under test, depending on actual needs.

[0170] It can be seen that for each hardware under test, the monitoring and management component will store the resource utilization verification model corresponding to each execution stage that the hardware under test will go through during the execution of the training task. Through cooperation with the task scheduling component, the monitoring and management component can send the resource utilization verification model corresponding to the execution stage of the hardware under test to the monitoring execution component each time the training task being executed by the hardware under test enters an execution stage. Thereby, the monitoring execution component can use the resource utilization verification model corresponding to the execution stage of the training task being executed by the hardware under test as the target monitoring model, and use the target monitoring model to conduct targeted monitoring of the actual resource usage of the hardware under test during the execution stage (since the amount of data of the resource utilization verification model is not large, the delay in sending the resource utilization verification model can be basically ignored).

[0171] In actual applications, if the hardware under test completes an execution phase of a training task and enters the next execution phase, and the next execution phase is the same as the previous execution phase, then since the resource usage verification models corresponding to the two execution phases are consistent, the monitoring management component does not need to repeatedly send the resource usage verification model to the monitoring execution component. However, the monitoring management component can still notify the monitoring execution component that the training task has entered a new execution phase, so that the monitoring execution component re-monitors the resource usage of the hardware under test based on the corresponding resource usage verification model.

[0172] In one example, when the monitoring execution component determines that the tested hardware is abnormal, it can report the abnormal state of the tested hardware to the monitoring management component. The monitoring management component can then notify the training task scheduling component of the collected abnormal state of the tested hardware, and the training task scheduling component can reschedule training resources and redeploy training tasks for the training tasks being executed by the tested hardware with abnormalities.

[0173] For more specific contents and beneficial effects of the above system, please refer to the description of the embodiment of the hardware status monitoring method during the model training process in the previous article, which will not be repeated here.

[0174] In one embodiment of the present application, when the model building component establishes a resource usage verification model corresponding to the hardware under test by collecting historical data, the usage collection component is also used to report the resource usage collected for each hardware under test of the device under test to the model building component.

[0175] The model building component is specifically used to determine, for each hardware under test, the historical resource utilization curve of the hardware under test in at least one training cycle based on the resource utilization reported by the utilization collection component for the hardware under test, and generate an expected resource utilization verification model corresponding to each execution stage of the training task for the hardware under test based on the historical resource utilization curve and the execution stage information of each sampling point in the historical resource utilization curve.

[0176] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0177] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0178] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for monitoring hardware status during model training, characterized in that: include: For each first execution stage of the first training task, during the process of the hardware under test executing the first execution stage, determining an expected resource usage range of the hardware under test during the process of executing the first execution stage by using a resource usage verification model corresponding to the first execution stage; When it is monitored that the actual resource usage rate of the tested hardware is not within the expected resource usage rate range, it is determined that the tested hardware is abnormal.

2. The method according to claim 1, characterized in that The first training task includes multiple training cycles, and the first execution phase of the first training task includes an iterative calculation phase and a model and training state preservation phase in each of the training cycles.

3. The method according to claim 1 or 2, characterized in that: The resource usage rate checking model corresponding to each of the first execution stages is obtained based on the following method: Determine an expected resource usage curve of the hardware under test during each first execution phase of executing the first training task; For each of the first execution stages, based on the expected resource utilization curve of the hardware under test during the execution of the first execution stage, a preset offset range is added to each data point on the expected resource utilization curve to obtain a resource utilization verification model corresponding to the first execution stage.

4. The method according to claim 3, characterized in that: The determining of the expected resource usage curve of the hardware under test during each first execution phase of executing the first training task includes: Collecting a historical resource usage curve of the hardware under test in at least one historical training cycle of executing the first training task; The historical resource usage curve is cut in combination with the execution stage information of each sampling point in the historical resource usage curve to obtain the expected resource usage curve corresponding to each first execution stage.

5. The method according to claim 4, characterized in that After collecting the historical resource usage curve of the hardware under test in at least one historical training cycle of executing the first training task, the method further includes: The historical resource utilization rate curve is regularized to obtain a regularized historical resource utilization rate curve; the maximum and minimum values ​​in the regularized historical resource utilization rate curve are converted into horizontal line segments.

6. The method according to claim 3, characterized in that The determining of the expected resource usage curve of the hardware under test during each first execution phase of executing the first training task includes: For each first execution stage included in the first training task, determine each computing process that the hardware under test needs to execute in the first execution stage, and a relative time period of each computing process within the execution duration of the first execution stage; Combined with the relative time periods corresponding to the calculation processes, the theoretical resource usage of the calculation processes and the total resource amount of the hardware under test, an initial resource usage rate curve of the hardware under test during the execution of the first execution phase is calculated; Based on the non-computing processes that the hardware under test needs to execute in the first execution phase, and the relative time periods of the execution duration of each non-computing process in the first execution phase, the initial resource utilization curve is corrected to obtain the expected resource utilization curve of the hardware under test during the execution of the first execution phase.

7. The method according to claim 3, characterized in that The first expected resource usage curve of the hardware under test during the first execution phase of the first training task is determined based on the following method: Obtaining a second expected resource usage curve of the reference hardware during a second execution phase of executing a second training task; the second training task and the first training task are associated tasks for the same model to be trained, and the expected resource usage of the reference hardware during the second execution phase is consistent with the expected resource usage of the tested hardware during the first execution phase; The first expected resource usage curve is generated based on the second expected resource usage curve in combination with the total resource amount of the tested hardware and the total resource amount of the reference hardware.

8. The method according to claim 7, characterized in that After generating the first expected resource usage curve based on the second expected resource usage curve, the method further includes: When the first monitoring sampling frequency of the hardware under test is less than the second monitoring sampling frequency of the reference hardware, the mean processing duration is determined in combination with the second monitoring sampling frequency, and based on the mean processing duration, the generated first expected resource utilization curve is divided into a plurality of sub-curves with a time length equal to the mean processing duration, and mean processing is performed on the data points in each sub-curve to obtain a corrected first expected resource utilization curve.

9. A method for establishing a resource utilization rate verification model, characterized in that: include: Obtaining each first execution phase included in a first training task assigned to the hardware under test; For each of the first execution stages, determining an expected resource usage curve of the hardware under test during execution of the first execution stage; Based on the expected resource usage rate curve corresponding to each first execution stage, generating the expected resource usage rate range of the hardware under test during the execution of each first execution stage, and obtaining the resource usage rate verification model corresponding to each first execution stage; The resource usage rate checking model is used to monitor whether there is an abnormality in the actual resource usage rate of the hardware during the execution of the corresponding first execution phase.

10. The method according to claim 9, characterized in that For each of the first execution stages, determining an expected resource usage curve of the hardware under test during execution of the first execution stage includes: Collecting a historical resource usage curve of the hardware under test in at least one historical training cycle of executing the first training task; The historical resource usage curve is cut in combination with the execution stage information of each sampling point in the historical resource usage curve to obtain the expected resource usage curve corresponding to each first execution stage.

11. The method according to claim 9, characterized in that For each of the first execution stages, determining an expected resource usage curve of the hardware under test during execution of the first execution stage includes: For each first execution stage included in the first training task, determine each computing process that the hardware under test needs to execute in the first execution stage, and a relative time period of each computing process within the execution duration of the first execution stage; Combined with the relative time periods corresponding to the calculation processes, the theoretical resource usage of the calculation processes and the total resource amount of the hardware under test, an initial resource usage rate curve of the hardware under test during the execution of the first execution phase is calculated; Based on the non-computing processes that the hardware under test needs to execute in the first execution phase, and the relative time periods of the execution duration of each non-computing process in the first execution phase, the initial resource utilization curve is corrected to obtain the expected resource utilization curve of the hardware under test during the execution of the first execution phase.

12. A hardware status monitoring system during model training, characterized in that: Includes the device under test, training task scheduling component, model building component, monitoring management component and monitoring execution component; The training task scheduling component is used to send the training task to the device under test, and send a stage notification message to the monitoring management component when monitoring that the training task enters each first execution stage; The device under test includes a usage rate collection component and at least one hardware under test; the hardware under test is used to perform training tasks, and the usage rate collection component is used to collect resource usage rates of each hardware under test in the device under test; The model building component is used to determine the resource usage rate verification model corresponding to each tested hardware of the tested device, and report the determined resource usage rate verification model to the monitoring management component for storage; The resource usage rate verification model corresponding to each of the tested hardware includes: a resource usage rate verification model corresponding to each first execution stage included in the training task; The monitoring management component is used to send a resource usage verification model corresponding to the current first execution stage of each of the tested hardware to the monitoring execution component in response to the stage notification message issued by the training task scheduling component; The monitoring execution component is used to obtain the resource usage rate collected by the usage rate collection component for each hardware under test, determine the expected resource usage rate range of each hardware under test through the resource usage rate verification model issued by the monitoring management component, and monitor the resource usage rate of each hardware under test through the determined expected resource usage rate range; when it is monitored that the resource usage rate of any hardware under test is not within the expected resource rate usage range, it is determined that the hardware under test has an abnormality.

13. The system according to claim 12, characterized in that The usage rate collection component is further used to report the resource usage rate collected for each tested hardware of the tested device to the model building component; The model building component is specifically used to determine, for each of the hardware under test, a historical resource utilization curve of the hardware under test in at least one training cycle based on the resource utilization reported by the utilization collection component for the hardware under test, and generate an expected resource utilization verification model corresponding to the hardware under test in each first execution stage of the training task based on the historical resource utilization curve and the execution stage information of each sampling point in the historical resource utilization curve.

Images