GPU resource monitoring method, device and program product

Abstract

The invention discloses a GPU resource monitoring method and device and a program product, and relates to the technical field of computers.The method comprises the steps that task parameters and GPU resource parameters are obtained, and a dynamic threshold value corresponding to a GPU resource monitoring index is determined according to the task parameters and the GPU resource parameters so as to adapt to the dynamic requirement of a running task on a GPU for GPU resources; in the embodiment of the invention, the alarm information is generated under the condition that the GPU resource monitoring index exceeds the dynamic threshold value, so that false alarm or missing alarm caused by configuring a fixed alarm threshold value for the GPU resource monitoring index is avoided, and the alarm accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a GPU resource monitoring method, device, and program product. Background Art

[0002] With the popularization of scenarios such as AI (Artificial Intelligence) training and high-performance computing, the scale of GPU (Graphics Processing Unit) clusters is constantly expanding, and traditional monitoring solutions face many challenges.

[0003] The traditional monitoring solution configures fixed alarm thresholds for the monitoring metrics of GPU resources, and if the monitoring metrics exceed the alarm thresholds, an alarm is triggered. However, different tasks executed by the GPU cluster have different requirements for GPU resources, and with the dynamic execution of different tasks, the requirements for GPU resources by the tasks fluctuate dynamically. For example, an AI training task may briefly occupy 99% of the video memory, while a graphics rendering task will continuously occupy 80% of the video memory. Using fixed alarm thresholds for different tasks is likely to result in false alarms or missed alarms. Summary of the Invention

[0004] In view of the above problems, this application provides a GPU resource monitoring method, device, and program product to achieve the purpose of improving the accuracy of alarms. The specific solutions are as follows:

[0005] The first aspect of this application provides a GPU resource monitoring method, including:

[0006] Obtain task parameters and GPU resource parameters;

[0007] Determine dynamic thresholds corresponding to GPU resource monitoring metrics according to the task parameters and the GPU resource parameters;

[0008] Generate an alarm message when the GPU resource monitoring metric exceeds the dynamic threshold.

[0009] In a possible implementation, the determining dynamic thresholds corresponding to GPU resource monitoring metrics according to the task parameters and the GPU resource parameters includes:

[0010] Calculate at least one task resource metric according to at least one of the task parameters and the GPU resource parameters;

[0011] Search for a target rule in the rule template library that matches the task parameters, the GPU resource parameters, and the task resource metric;

[0012] Determine the dynamic thresholds corresponding to the GPU resource monitoring metrics according to the target rule;

[0013] or

[0014] Input the task parameters and the GPU resource parameters into a dynamic threshold model to obtain the dynamic threshold corresponding to the GPU resource monitoring metric output by the dynamic threshold model, where the dynamic threshold model is obtained after training using training samples with labeled dynamic thresholds, and the training samples include historical task parameters and historical GPU resource parameters.

[0015] In a possible implementation, when the GPU resource monitoring metric exceeds the dynamic threshold, an alarm message is generated, including:

[0016] When the duration for which the GPU resource monitoring metric exceeds the dynamic threshold is greater than a preset duration, an alarm message is generated.

[0017] In a possible implementation, the obtaining of the task parameters and the GPU resource parameters includes:

[0018] When it is detected that a video memory operation function is called, sample the video memory occupancy at a first frequency;

[0019] If the difference between the currently sampled video memory occupancy and the previously sampled video memory occupancy is greater than a first value or the change rate of the video memory occupancy is greater than a second value, sample the task parameters and the GPU resource parameters at a second frequency, and the second frequency is higher than the first frequency.

[0020] In a possible implementation, after sampling the task parameters and the GPU resource parameters at the second frequency, the GPU resource monitoring method further includes:

[0021] Associate the GPU with the task process using a resource control group or a container;

[0022] Obtain the GPU resource usage of each task process.

[0023] In a possible implementation, after associating the GPU with the task process using a resource control group or a container, the GPU resource monitoring method further includes:

[0024] Calculate at least one task resource metric according to at least one of the task parameters and the GPU resource parameters;

[0025] The calculating of at least one task resource metric according to at least one of the task parameters and the GPU resource parameters includes at least one of the following:

[0026] Calculate the video memory fragmentation rate according to the total size of the GPU video memory and the size of the largest continuous free video memory block;

[0027] Calculate the task urgency according to the task priority, time urgency, and weight coefficient of the task priority and the time urgency, where the time urgency is calculated based on the running time of the task and the timeout threshold;

[0028] Calculate the resource task binding strength according to the utilization rate of each GPU thread, the dependence coefficient of the task on each GPU thread, and the total number of GPU threads occupied by the task;

[0029] Calculate the computing power demand volatility of the task according to the average computing power usage rate of the task within a preset time window and the standard deviation of the computing power usage rate.

[0030] In a possible implementation, after generating the alarm information, the GPU resource monitoring method further includes:

[0031] Determine the target task corresponding to the GPU resource monitoring metric that exceeds the dynamic threshold according to the association relationship between the GPU and the task process;

[0032] Associate the alarm information with the target task.

[0033] A second aspect of the present application provides an electronic device, including at least one processor and a memory connected to the processor, where:

[0034] The memory is used to store a computer program;

[0035] The processor is used to execute the computer program so that the electronic device can implement the GPU resource monitoring method of the first aspect or any implementation manner of the first aspect.

[0036] A third aspect of the present application provides a computer program product, including computer-readable instructions, which when running on an electronic device, enable the electronic device to implement the GPU resource monitoring method of the first aspect or any implementation manner of the first aspect.

[0037] A fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, can enable the electronic device to implement the GPU resource monitoring method of the first aspect or any implementation manner of the first aspect.

[0038] With the above technical solution, a GPU resource monitoring method, device, and program product provided by the present application determine dynamic thresholds corresponding to GPU resource monitoring metrics according to task parameters and GPU resource parameters to adapt to the dynamic requirements of tasks running on the GPU for GPU resources. When the GPU resource monitoring metrics exceed the dynamic thresholds, an alarm message is generated, avoiding false alarms or missed alarms caused by configuring fixed alarm thresholds for the GPU resource monitoring metrics and improving the accuracy of alarms. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the original elements and elements are not necessarily drawn to scale.

[0040] Figure 1 It is a flowchart of a GPU resource monitoring method provided by an embodiment of the present application;

[0041] Figure 2 It is a structural schematic diagram of a GPU resource monitoring device provided by an embodiment of the present application;

[0042] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The following describes the embodiments of the present application in combination with the accompanying drawings in the embodiments of the present application. The terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0044] The following describes the embodiments of the present application in combination with the accompanying drawings. Those of ordinary skill in the art will know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0045] The terms "first", "second", etc. in the specification, claims, and above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product, or device including a series of units does not necessarily need to be limited to those units, but may include other units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0046] An embodiment of the present application provides a GPU resource monitoring method. The GPU resource monitoring method according to the embodiment of the present application will be introduced in detail below with reference to the accompanying drawings.

[0047] Refer to Figure 1 , Figure 1 which is a schematic flowchart of a GPU resource monitoring method provided by an embodiment of the present application. As Figure 1 shown, a GPU resource monitoring method provided by an embodiment of the present application may include steps 101 to 103, and these steps will be described in detail below.

[0048] 101: Obtain task parameters and GPU resource parameters;

[0049] The task parameters include: task type, process priority, running time, computing power utilization rate, etc.

[0050] The GPU resource parameters include: total video memory size, maximum continuous free video memory block size, video memory occupancy, etc.

[0051] Among them, the task parameters and GPU resource parameters can be obtained periodically, and can also be obtained when the trigger condition is met. This embodiment does not make specific limitations.

[0052] 102: Determine the dynamic threshold corresponding to the GPU resource monitoring index according to the task parameters and GPU resource parameters;

[0053] There are various implementation methods for determining the dynamic threshold, such as dynamic threshold determination based on rule matching, dynamic threshold determination based on a model, etc. The following will be specifically introduced through different embodiments.

[0054] 103: Generate an alarm message when the GPU resource monitoring index exceeds the dynamic threshold.

[0055] A GPU resource monitoring method disclosed in this embodiment determines the dynamic threshold corresponding to the GPU resource monitoring index according to the task parameters and GPU resource parameters to adapt to the dynamic demand of the tasks running on the GPU for GPU resources. When the GPU resource monitoring index exceeds the dynamic threshold, an alarm message is generated, avoiding false alarms or missed alarms caused by configuring a fixed alarm threshold for the monitoring index of GPU resources, and improving the accuracy of the alarm.

[0056] Currently, generally based on the polling mechanism, the GPU status is frequently queried, such as performing a query operation once per second. High-frequency queries will occupy a large amount of CPU resources. Especially for the case where there are a large number of GPUs in the cluster, it will generate an exponentially growing CPU overhead, resulting in a decline in system performance. In addition, for situations such as the moment when the model is loaded, an instantaneous GPU video memory exhaustion event may be missed.

[0057] To solve this technical problem, one implementation of step 101 in the above embodiment includes the following steps 1011-1012:

[0058] 1011: When it is detected that the video memory operation function is called, sample the video memory occupancy at the first frequency;

[0059] Adopt the hooking method to intercept the video memory operation function. Taking the CUDA architecture (Compute Unified Device Architecture) as an example, using the Hook technology, hook the dynamic libraries related to CUDA (such as NVML, Cuda Runtime, Cudnn, Cublas, etc.) in the user mode to intercept the video memory operation functions at the Driver API layer such as cuMemAlloc and cuMemFree.

[0060] When it is detected that the video memory operation function is called, the video memory data collection thread can be triggered to sample and record the video memory occupancy at the first frequency.

[0061] Furthermore, while recording the video memory occupancy, the call stack can also be recorded, and the functions that are frequently called and occupy more video memory can be determined through the call stack.

[0062] 1012: If the difference between the currently sampled video memory occupancy and the previously sampled video memory occupancy is greater than the first value or the change rate of the video memory occupancy is greater than the second value, then sample the task parameters and GPU resource parameters at the second frequency.

[0063] The first value and the second value are preset. For example: the first value can be 5%, and the second value can be 10% per second.

[0064] The second frequency is higher than the first frequency. For example, the first frequency is at the second level, and the second frequency is at the millisecond level.

[0065] In this embodiment, by collecting the task parameters and GPU resource parameters only when an event occurs, that is, when the video memory operation function is called and the difference between the currently sampled video memory occupancy and the previously sampled video memory occupancy is greater than the first value or the change rate of the video memory occupancy is greater than the second value, that is, on-demand collection, a large amount of invalid data collection is reduced compared to the traditional polling mechanism, avoiding the waste of CPU resources. At the same time, since the collection is only performed when an event occurs, it can detect task events that occupy video memory such as the moment when the model is loaded in a timely manner, avoiding missing the instantaneous GPU video memory exhaustion event.

[0066] In addition, currently for GPU resource monitoring, only relevant metrics of the GPU are monitored, and it is impossible to associate GPU resources with tasks, making it difficult for operation and maintenance personnel to locate the root cause of resource contention. For example, operation and maintenance personnel only know that the video memory of a certain GPU is exhausted, but do not know which process caused the exhaustion of the video memory of this GPU.

[0067] To solve this technical problem, in this embodiment, after obtaining task parameters and GPU resource parameters, the GPU is associated with the task process by using a resource control group or a container, so as to obtain the GPU resource usage of each task process.

[0068] In a possible implementation, in the case where the NVIDIA driver and the CUDA toolkit are installed, a cgroup (resource control group) is created and resource limits are set. Specifically, the GPU device is added to the resource control group, and the task process is added to the resource control group, thereby associating the GPU with the task process.

[0069] In another possible implementation, in the case where the NVIDIA driver, the CUDA toolkit, and the NVIDIA Container Toolkit are installed, a Docker container is run and the GPU device to be used by the Docker container is specified, thereby associating the task process running in the Docker container with the GPU.

[0070] The association relationship between the GPU and the task process can have various applications.

[0071] Among them, one application is: according to the association relationship between the GPU and the task process, determine the target task corresponding to the GPU resource monitoring metric that exceeds the dynamic threshold, and associate the alarm information with the target task, which is convenient for operation and maintenance personnel to quickly locate the target task. For example, the alarm information indicates that the video memory occupied by the model training task exceeds the dynamic threshold.

[0072] Another application is: calculate at least one task resource metric according to at least one of the task parameters and the GPU resource parameters.

[0073] In a possible implementation, calculating at least one task resource metric according to at least one of the task parameters and the GPU resource parameters includes at least one of the following:

[0074] A1: Calculate the video memory fragmentation rate according to the total size of the GPU video memory and the size of the largest continuous free video memory block;

[0075] Exemplarily, the calculation formula of the video memory fragmentation rate (Memory Fragmentation Rate, MFR) is as follows:

[0076] ;

[0077] Among them, the total video memory size is the total video memory size of the GPU, such as 24GB, and the maximum continuous free video memory block size is the maximum value of the continuous video memory space that is not currently allocated.

[0078] The video memory fragmentation rate is used to quantify the degree of fragmentation of the video memory. The higher the video memory fragmentation rate, the more the video memory is divided into small pieces, which may cause the large model training task to fail because continuous video memory cannot be applied for. When the video memory fragmentation rate exceeds the threshold (such as 40%), video memory sorting or task migration can be triggered as needed.

[0079] A2: Calculate the task urgency according to the task priority, time urgency, and the weight coefficient of the task priority and time urgency;

[0080] Exemplarily, the calculation formula of the task urgency (Task Urgency Score, TUS) is as follows:

[0081] ;

[0082] Among them, represents the task priority, which can be normalized to 0-1. For example, the priority of a real-time task is 1, and the priority of a background task is 0.3;

[0083] respectively represent the weight coefficients of the task priority and time urgency;

[0084] The task timeout threshold represents the maximum running time allowed for the task. The task timeout thresholds corresponding to different tasks are different. For example, the task timeout threshold for a model training task is 2 hours.

[0085] The task urgency is used to quantify the urgency of the task. The higher the TUS (the closer to 1), the more the task needs to be guaranteed resources preferentially, and reserved video memory can be allocated for this task.

[0086] Furthermore, the weight coefficients of the task priority and time urgency can be dynamically adjusted according to the cluster load. For example: when the GPU resources are tight, increase the weight coefficient of time urgency.

[0087] A3: Calculate the resource-task binding strength according to the utilization rate of each GPU thread, the dependence coefficient of the task on each GPU thread, and the total number of GPU threads occupied by the task;

[0088] Exemplarily, the calculation formula of the resource-task binding strength (Resource-Task Binding Strength, RTBS) is as follows:

[0089] ;

[0090] Among them, Represents the utilization rate of the i-th GPU thread;

[0091] Represents the degree of dependence of a task on the i-th GPU thread. For example, the dependence coefficient of a task on the main computing thread is 1, and the dependence coefficient of a task on the auxiliary thread is 0.5;

[0092] The total number of task threads is the total number of GPU threads occupied by the task, which has now become an execution unit within the process.

[0093] The resource-task binding strength is used to locate key resource bottlenecks. The higher the resource-task binding strength, the stronger the dependence of the task on a specific GPU thread. If the utilization rate of this thread reaches saturation, the overall performance of the task will be limited.

[0094] Furthermore, when the resource-task binding strength is greater than the threshold, the task can be migrated to a GPU with more idle threads.

[0095] A4: Calculate the computing power demand volatility of a task based on the average computing power utilization rate and the standard deviation of the computing power utilization rate of the task within a preset time window.

[0096] Exemplarily, the calculation formula for the computing power demand volatility (CDV) of a task is as follows:

[0097] ;

[0098] where μ represents the average computing power utilization rate of the task within a preset time window, represents the standard deviation of the computing power utilization rate, reflecting the fluctuation range of the computing power demand of the task.

[0099] The computing power demand volatility of a task is used to identify tasks with unstable computing power demands. The higher the computing power demand volatility of a task, the greater the fluctuation of the computing power demand of the task. For example, interactive rendering tasks generally have a relatively large fluctuation in computing power demand and require dynamic adjustment of resource reservation strategies.

[0100] The computing power demand volatility of a task can also be used to adjust the alarm threshold, that is, the above-mentioned dynamic threshold. For example, for tasks with a high computing power demand volatility, the dynamic threshold can be increased to reduce false alarms.

[0101] The above task resource metrics can be used to generate a visual portrait. For example, display the computing power demand heat map, video memory fragmentation distribution, and task urgency ranking of each GPU in the display panel, providing data support for subsequent decisions on whether to migrate tasks or allocate reserved resources.

[0102] The above task resource metrics can also be used to determine the dynamic thresholds corresponding to GPU resource monitoring metrics. In one possible implementation, one implementation manner of step 102 in the above embodiment includes the following steps 1021-1023:

[0103] 1021: Calculate at least one task resource metric according to at least one of the task parameters and GPU resource parameters;

[0104] The above embodiment has already introduced in detail the specific calculation methods of each task resource metric, which will not be elaborated here.

[0105] 1022: Search for the target rule that matches the task parameters, GPU resource parameters, and task resource metrics in the rule template library;

[0106] 1023: Determine the dynamic threshold corresponding to the GPU resource monitoring metric according to the target rule.

[0107] Exemplarily, the rule template library includes the following 3 rules:

[0108] Rule 1: If the task type is a model training task and the video memory fragmentation rate is greater than 40%, then increase the video memory threshold to 1.3 times the baseline value.

[0109] Rule 2: If the task priority is greater than 0.8, then reduce the video memory threshold to 70% of the baseline value.

[0110] Rule 3: If the computing power demand volatility of the task is greater than 30%, then increase the video memory threshold to 1.2 times the baseline value.

[0111] In an actual application scenario, there may be a situation where a task satisfies more than one rule. For example, for a task whose task type is a model training task, the video memory fragmentation rate is greater than 40%, and the task priority is greater than 0.8. In this case, the target rule corresponding to the task can be determined according to the pre-set rule priority.

[0112] In another possible implementation, one implementation manner of step 102 in the above embodiment includes: inputting the task parameters and GPU resource parameters into a dynamic threshold model to obtain the dynamic threshold corresponding to the GPU resource monitoring metric output by the dynamic threshold model. Among them, the dynamic threshold model is obtained by training a model using training samples with labeled dynamic thresholds. The training samples include historical task parameters and historical GPU resource parameters. The dynamic threshold model can be a dynamic threshold model based on a decision tree, and the dynamic threshold model is used to learn the complex correlation relationship between task parameters, GPU resource parameters, and dynamic thresholds.

[0113] It should also be noted that since the threshold is dynamic, there may be a phenomenon of threshold jitter. For example, at a certain moment, the video memory threshold is 1.3 times the reference value, and at the next moment, the video memory threshold drops to 70% of the reference value, which may lead to false alarms.

[0114] To solve this technical problem, in this embodiment, a sliding window is used to smooth the instantaneous fluctuation of the dynamic threshold and avoid threshold jitter. Specifically, when the duration for which the GPU resource monitoring metric exceeds the dynamic threshold is greater than a preset duration (i.e., the duration of the above-mentioned sliding window, which can be 10 seconds for example), an alarm message is generated. That is, when the GPU resource monitoring metric instantaneously exceeds the dynamic threshold, no alarm message is generated, but when the GPU resource monitoring metric continuously exceeds the dynamic threshold, an alarm message is generated to avoid false alarms.

[0115] The above introduced a GPU resource monitoring method provided by an embodiment of the present application. Next, an apparatus for executing the above-mentioned GPU resource monitoring method will be introduced.

[0116] Please refer to Figure 2 , Figure 2 which is a schematic structural diagram of a GPU resource monitoring apparatus provided by an embodiment of the present application. As Figure 2 shown, the GPU resource monitoring apparatus includes:

[0117] A parameter acquisition unit 201, configured to acquire task parameters and GPU resource parameters;

[0118] A threshold determination unit 202, configured to determine a dynamic threshold corresponding to a GPU resource monitoring metric according to the task parameters and the GPU resource parameters;

[0119] An alarm generation unit 203, configured to generate an alarm message when the GPU resource monitoring metric exceeds the dynamic threshold.

[0120] In a possible implementation, the threshold determination unit 202 is specifically configured to calculate at least one task resource metric according to at least one of the task parameters and the GPU resource parameters; search for a target rule in a rule template library that matches the task parameters, the GPU resource parameters, and the task resource metric; and determine a dynamic threshold corresponding to the GPU resource monitoring metric according to the target rule.

[0121] In a possible implementation, the threshold determination unit 202 is specifically configured to input the task parameters and the GPU resource parameters into a dynamic threshold model to obtain the dynamic threshold corresponding to the GPU resource monitoring metric output by the dynamic threshold model, where the dynamic threshold model is obtained by training a model using training samples with labeled dynamic thresholds, and the training samples include historical task parameters and historical GPU resource parameters.

[0122] In a possible implementation, the alarm generation unit 203 is specifically configured to generate an alarm message when the duration for which the GPU resource monitoring metric exceeds the dynamic threshold is greater than the preset duration.

[0123] In a possible implementation, the parameter acquisition unit 201 is specifically configured to sample the video memory occupancy at a first frequency when it detects that a video memory operation function is called; if the difference between the currently sampled video memory occupancy and the previously sampled video memory occupancy is greater than a first value or the change rate of the video memory occupancy is greater than a second value, then sample the task parameters and the GPU resource parameters at a second frequency, where the second frequency is higher than the first frequency.

[0124] In a possible implementation, the GPU resource monitoring device further includes:

[0125] A first association unit, configured to associate a GPU with a task process by using a resource control group or a container; and obtain the GPU resource usage of each task process.

[0126] In a possible implementation, the GPU resource monitoring device further includes:

[0127] A metric calculation unit, configured to calculate at least one task resource metric according to at least one of the task parameters and the GPU resource parameters.

[0128] In a possible implementation, the metric calculation unit is specifically configured to perform at least one of the following:

[0129] Calculate a video memory fragmentation rate according to the total GPU video memory size and the size of the largest continuously free video memory block;

[0130] Calculate a task urgency according to the task priority, time urgency, and weight coefficient of the task priority and the time urgency, where the time urgency is calculated according to the elapsed time of the task and the timeout threshold;

[0131] Calculate a resource task binding strength according to the utilization rate of each GPU thread, the dependency coefficient of the task on each GPU thread, and the total number of GPU threads occupied by the task;

[0132] Calculate a computing power demand volatility of the task according to the average computing power usage rate of the task within a preset time window and the standard deviation of the computing power usage rate.

[0133] In a possible implementation, the GPU resource monitoring device further includes:

[0134] A second association unit, configured to determine a target task corresponding to the GPU resource monitoring metric that exceeds the dynamic threshold according to the association relationship between the GPU and the task process; and associate the alarm information with the target task.

[0135] A GPU resource monitoring device disclosed in this embodiment determines a dynamic threshold corresponding to a GPU resource monitoring metric according to task parameters and GPU resource parameters, so as to adapt to the dynamic demand of tasks running on the GPU for GPU resources. When the GPU resource monitoring metric exceeds the dynamic threshold, an alarm information is generated, avoiding false alarms or missed alarms caused by configuring a fixed alarm threshold for the monitoring metric of GPU resources, and improving the accuracy of alarms.

[0136] An electronic device is further provided in an embodiment of this application. As Figure 3 shown, the electronic device includes a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other through the bus 301.

[0137] The bus 301 may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0138] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0139] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may further include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0140] Among them, the memory 304 can be used to store software codes related to the GPU resource monitoring method. The processor 302 can execute the steps of the GPU resource monitoring method and can also schedule other units to implement corresponding functions.

[0141] It should be understood that the above electronic device can be a centralized or distributed device. The processor in the electronic device can be a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.) or a combination of these hardware circuits. For example, the processor can be a hardware system with the function of executing instructions, such as a CPU, a DSP, etc., or a hardware system without the function of executing instructions, such as an ASIC, an FPGA, etc., or a combination of the above hardware systems without the function of executing instructions and the hardware systems with the function of executing instructions.

[0142] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions run on an electronic device, the electronic device is enabled to implement any one of the GPU resource monitoring methods provided by the embodiments of the present application.

[0143] An embodiment of the present application also provides a computer-readable storage medium. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can be enabled to implement any one of the GPU resource monitoring methods provided by the embodiments of the present application.

[0144] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in the present application, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.

[0145] Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for this application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that makes a contribution to the prior art can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disc of a computer, etc., and includes several instructions to enable a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in various embodiments of this application.

[0146] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0147] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are generated in whole or in part. The computer can be a general computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, training device, or data center to another website, computer, training device, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

Claims

1. A GPU resource monitoring method, characterized in that, Including: Obtain task parameters and GPU resource parameters; Determine a dynamic threshold corresponding to a GPU resource monitoring metric according to the task parameters and the GPU resource parameters; Generate an alarm message when the GPU resource monitoring metric exceeds the dynamic threshold.

2. The GPU resource monitoring method according to claim 1, wherein The determining a dynamic threshold corresponding to a GPU resource monitoring metric according to the task parameters and the GPU resource parameters includes: Calculate at least one task resource metric according to at least one of the task parameters and the GPU resource parameters; Search for a target rule in a rule template library that matches the task parameters, the GPU resource parameters, and the task resource metric; Determine a dynamic threshold corresponding to the GPU resource monitoring metric according to the target rule; Or Input the task parameters and the GPU resource parameters into a dynamic threshold model to obtain the dynamic threshold corresponding to the GPU resource monitoring metric output by the dynamic threshold model, where the dynamic threshold model is obtained by training a model using training samples with labeled dynamic thresholds, and the training samples include historical task parameters and historical GPU resource parameters.

3. The GPU resource monitoring method according to claim 1 or 2, characterized in that, The generating an alarm message when the GPU resource monitoring metric exceeds the dynamic threshold includes: Generate an alarm message when the duration for which the GPU resource monitoring metric exceeds the dynamic threshold is greater than a preset duration.

4. The GPU resource monitoring method according to claim 1, wherein The obtaining task parameters and GPU resource parameters includes: When it is detected that a video memory operation function is called, sample the video memory occupancy at a first frequency; If the difference between the currently sampled video memory occupancy and the previously sampled video memory occupancy is greater than a first value or the change rate of the video memory occupancy is greater than a second value, sample the task parameters and the GPU resource parameters at a second frequency, where the second frequency is higher than the first frequency.

5. The GPU resource monitoring method according to claim 4, wherein After sampling the task parameters and the GPU resource parameters at the second frequency, the GPU resource monitoring method further includes: Associate the GPU with a task process using a resource control group or a container; Obtain the GPU resource usage of each task process.

6. The GPU resource monitoring method according to claim 5, wherein After associating the GPU with a task process using a resource control group or a container, the GPU resource monitoring method further includes: Calculate at least one task resource metric according to at least one of the task parameters and the GPU resource parameters; The calculating at least one task resource metric according to at least one of the task parameters and the GPU resource parameters includes at least one of the following: Calculate a video memory fragmentation rate according to the total GPU video memory size and the size of the largest continuous free video memory block; Calculate a task urgency according to a task priority, a time urgency, and a weight coefficient of the task priority and the time urgency, where the time urgency is calculated according to the elapsed time of the task and a timeout threshold; Calculate a resource task binding strength according to the utilization rate of each GPU thread, a task's dependence coefficient on each GPU thread, and the total number of GPU threads occupied by the task; Calculate a computing power demand volatility of the task according to the average computing power usage rate of the task within a preset time window and the standard deviation of the computing power usage rate.

7. The GPU resource monitoring method according to claim 5, wherein After generating the warning information, the GPU resource monitoring method further includes: Determining a target task corresponding to the GPU resource monitoring metric that exceeds the dynamic threshold according to the association relationship between the GPU and the task process; Associating the warning information with the target task.

8. An electronic device, characterized in that, Comprising at least one processor and a memory connected to the processor, wherein: The memory is used for storing a computer program; The processor is configured to execute the computer program so that the electronic device can implement the GPU resource monitoring method according to any one of claims 1 to 7.

9. A computer program product, characterized in that, Including computer-readable instructions, when the computer-readable instructions run on an electronic device, the electronic device is enabled to implement the GPU resource monitoring method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can be enabled to implement the GPU resource monitoring method according to any one of claims 1 to 7.

Images