Computing resource management method, apparatus, device, and system

Abstract

The application discloses a computing resource management method, device, system and equipment. The system determines the priority of a plurality of machine learning tasks running through a graphic processing unit (GPU); determines the computing resource usage information of the tasks and the running state information of high-priority tasks; adjusts the upper limit value of the computing resource usage of low-priority tasks according to the running state information and the usage information; sends a computing resource adjustment instruction to the machine learning framework of low-priority tasks; and the machine learning framework controls the computing resource usage of low-priority tasks according to the upper limit value included in the adjustment instruction. This processing manner dynamically adjusts and optimizes the GPU computing resources used by low-priority tasks running in parallel on one GPU according to the running performance of high-priority tasks, thereby effectively improving the utilization rate of GPU computing resources and guaranteeing the performance of high-priority tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a computing resource management method and device, a machine learning system, and an electronic device. BACKGROUND

[0002] With the continuous development of deep learning algorithms and the increase of GPU computing power, deep learning has become a crucial part of enterprise product data flow. In order to support large-scale deep learning applications, enterprises usually build large-scale multi-tenant shared GPU clusters to support the development of products across multiple fields, such as computer vision, natural language processing, speech recognition, recommendation and advertising services, etc.

[0003] In order to improve the utilization of GPU resources and the throughput of the entire GPU cluster, the deep learning system allows multiple deep learning tasks to run simultaneously on one GPU, which can complete more deep learning training tasks with the same amount of resources. At present, a typical GPU computing resource multiplexing mode is that multiple deep learning tasks running on the same GPU share GPU computing resources to fully utilize GPU resources and improve the throughput of cluster jobs.

[0004] However, in the process of implementing the present application, the inventors found that the above technical solutions all have at least the following problems: Since the use of GPU computing resources by deep learning tasks in various stages of running presents volatility, simply running multiple tasks simultaneously on the same GPU will cause tasks to compete for GPU computing resources, affecting the performance of high-priority tasks, thereby violating the service level (SLA) guarantee of the tasks, and even causing some tasks to fail. In summary, how to manage the shared GPU computing resources of a machine learning system to improve the utilization of a GPU cluster while ensuring the running performance of high-priority tasks has become a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] The present application provides a computing resource management method to solve the problem that the performance of high-priority tasks cannot be guaranteed in the prior art. The present application further provides a computing resource management device and system, a machine learning system, and an electronic device.

[0006] The present application provides a machine learning system, comprising:

[0007] a computing resource coordinator and a machine learning framework;

[0008] The computing resource coordinator is configured to determine priorities of a plurality of machine learning tasks running through a graphics processing unit (GPU), determine computing resource usage information of the tasks and running status information of a high-priority task, adjust an upper limit of computing resource usage of a low-priority task according to the running status information and the usage information, and send a computing resource adjustment instruction to a machine learning framework of the low-priority task.

[0009] The machine learning framework is configured to control the computing resource usage of the low-priority task according to the upper limit included in the adjustment instruction.

[0010] Optionally, the computing resource coordinator is specifically configured to, if the running status information of the high-priority task reaches a performance threshold, increase the upper limit of the computing resource usage of the low-priority task according to the usage information.

[0011] Optionally, the computing resource coordinator is specifically configured to, if the running status information of the high-priority task does not reach the performance threshold, decrease the upper limit of the computing resource usage of the low-priority task according to the usage information.

[0012] Optionally, the computing resource coordinator is further configured to, if a low-priority task to be run appears while the high-priority task is running, set a first upper limit of the computing resource usage of the low-priority task.

[0013] Optionally, the computing resource coordinator is further configured to, if a high-priority task to be run appears while the low-priority task is running, adjust the computing resource usage of the low-priority task to the first upper limit.

[0014] Optionally, the machine learning framework is specifically configured to determine a GPU operator emission frequency according to the upper limit, and emit a GPU operator to the GPU according to the emission frequency, so that the GPU operator is executed through the GPU.

[0015] Optionally, the machine learning framework is specifically configured to determine a time slot length and an insertion position before the GPU operator according to the emission frequency, and insert a time slot according to the time slot length and the insertion position when emitting the GPU operator to the GPU.

[0016] Optionally, the machine learning framework is further configured to send the running status information to the computing resource coordinator if the running status information of the high-priority task changes.

[0017] The application further provides a machine learning system, comprising:

[0018] a client configured to send priority information of a machine learning task to a server;

[0019] The server is used to determine the computing resource usage status information of the machine learning task running through the graphics processing unit (GPU) and the operating status information of the high-priority task; adjust the upper limit of the computing resource usage of the low-priority task based on the operating status information and the usage status information; and send a computing resource adjustment instruction to the machine learning framework of the low-priority task.

[0020] This application also provides a computing resource management method, including:

[0021] Prioritize multiple machine learning tasks running on graphics processing units (GPUs);

[0022] Determining computing resource usage information of the task and running status information of high-priority tasks;

[0023] The computing resource usage of the low-priority task is controlled according to the running status information and the usage status information.

[0024] Optionally, controlling the computing resource usage of the low-priority task according to the operating status information and the usage status information includes:

[0025] Adjusting an upper limit of computing resource usage of low-priority tasks according to the operating status information and the usage status information;

[0026] The speed at which low-priority tasks transmit GPU operators to the GPU is controlled according to the upper limit.

[0027] Optionally, adjusting the upper limit of computing resource usage of low-priority tasks according to the operating status information and the usage status information includes:

[0028] If the running status information of the high-priority task reaches the performance threshold, the upper limit of the computing resource usage of the low-priority task is increased according to the usage status information.

[0029] Optionally, adjusting the upper limit of computing resource usage of low-priority tasks according to the operating status information and the usage status information includes:

[0030] If the running status information of the high-priority task does not reach the performance threshold, the upper limit of the computing resource usage of the low-priority task is reduced according to the usage status information.

[0031] Optionally, also include:

[0032] If a low-priority task to be run appears when a high-priority task is being run, a first upper limit value of the computing resource usage of the low-priority task is set.

[0033] Optionally, further comprising:

[0034] If a high-priority task to be run occurs while a low-priority task is running, the amount of computing resource used by the low-priority task is adjusted to the first upper limit value.

[0035] Optionally, the speed at which the low-priority task sends GPU operators to the GPU is controlled according to the upper limit value, comprising:

[0036] The GPU operator sending frequency is determined according to the upper limit value.

[0037] The GPU operator is sent to the GPU according to the sending frequency, so that the GPU executes the GPU operator.

[0038] Optionally, the running status information comprises running performance information, which is determined in the following manner:

[0039] The running performance information is determined according to the number of iterations of the task in a unit of time.

[0040] The application further provides a computing resource management device, comprising:

[0041] A priority determination unit is configured to determine the priorities of a plurality of machine learning tasks running through a GPU.

[0042] An information collection unit is configured to determine the computing resource usage status information of the tasks and the running status information of high-priority tasks.

[0043] A computing resource adjustment unit is configured to control the amount of computing resource used by low-priority tasks according to the running status information and the usage status information.

[0044] The application further provides an electronic device, comprising:

[0045] A processor and a memory.

[0046] The memory is configured to store a program for implementing the above computing resource management method, and the device is powered on and runs the program of the method through the processor.

[0047] The application further provides a computer readable storage medium, which stores instructions, and when the instructions are run on a computer, the computer executes the above various methods.

[0048] The application further provides a computer program product comprising instructions, and when the instructions are run on a computer, the computer executes the above various methods.

[0049] Compared with the prior art, the application has the following advantages:

[0050] The machine learning system provided in the embodiments of the present application determines the priorities of a plurality of machine learning tasks running through a graphics processing unit (GPU), determines the computing resource usage status information of the tasks and the running status information of high-priority tasks, adjusts the upper limit of the computing resource usage of low-priority tasks according to the running status information and the usage status information, sends a computing resource adjustment instruction to the machine learning framework of the low-priority tasks, and controls the computing resource usage of the low-priority tasks according to the upper limit included in the adjustment instruction by the machine learning framework. This processing manner dynamically and flexibly optimizes the GPU computing resources used by the low-priority tasks in parallel on one GPU according to the running performance of the high-priority tasks, so that the idle computing resources can be fully utilized and allocated to the low-priority tasks under the premise of guaranteeing the performance of the high-priority tasks. Therefore, the GPU computing resource utilization can be effectively improved while guaranteeing the performance of the high-priority tasks.

[0051] The machine learning system provided in the embodiments of the present application sends priority information of machine learning tasks to a server by a client, determines computing resource usage status information of machine learning tasks running through a graphics processing unit (GPU) and running status information of high-priority tasks by the server, adjusts the upper limit of the computing resource usage of low-priority tasks according to the running status information and the usage status information, and sends a computing resource adjustment instruction to the machine learning framework of the low-priority tasks. This processing manner dynamically and flexibly optimizes the GPU computing resources used by the low-priority tasks in parallel on one GPU according to the running performance of the high-priority tasks, so that the idle computing resources can be fully utilized and allocated to the low-priority tasks under the premise of guaranteeing the performance of the high-priority tasks. Therefore, the GPU computing resource utilization can be effectively improved while guaranteeing the performance of the high-priority tasks.

[0052] The computing resource management method provided in the embodiments of the present application determines the priorities of a plurality of machine learning tasks running through a graphics processing unit (GPU), determines the computing resource usage status information of the tasks and the running status information of high-priority tasks, and controls the computing resource usage of low-priority tasks according to the running status information and the usage status information. This processing manner dynamically and flexibly optimizes the GPU computing resources used by the low-priority tasks in parallel on one GPU according to the running performance of the high-priority tasks, so that the idle computing resources can be fully utilized and allocated to the low-priority tasks under the premise of guaranteeing the performance of the high-priority tasks. Therefore, the GPU computing resource utilization can be effectively improved while guaranteeing the performance of the high-priority tasks. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1A schematic diagram of the structure of an embodiment of a machine learning system provided in this application;

[0054] Figure 2 A schematic diagram of a scenario of an embodiment of a machine learning system provided in this application;

[0055] Figure 3 A schematic diagram of the system architecture of an embodiment of a machine learning system provided in this application;

[0056] Figure 4 A schematic diagram of dynamic adjustment of computing resources of an embodiment of a machine learning system provided in this application;

[0057] Figure 5 A schematic diagram of the structure of an embodiment of a machine learning system provided in this application;

[0058] Figure 6 A flowchart of an embodiment of a computing resource management method provided by the present application;

[0059] Figure 7 This application provides a schematic structural diagram of an embodiment of a computing resource management device. DETAILED DESCRIPTION

[0060] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present application. Therefore, the present application is not limited to the specific implementations disclosed below.

[0061] In this application, a machine learning system, a computing resource management method, device and system, and an electronic device are provided. Each solution is described in detail in the following embodiments.

[0062] First embodiment

[0063] Please refer to Figure 1 , which is a structural diagram of an embodiment of the machine learning system of the present application. The system provided in this embodiment includes: a computing resource coordinator 1 and a machine learning framework 2.

[0064] The system can execute multiple machine learning tasks on a single GPU, sharing the GPU's computing resources. A GPU, also known as a graphics processing unit or graphics card, is a microprocessor specialized for image and graphics-related computations. Thanks to its efficient matrix computing capabilities, GPUs are crucial hardware devices for running deep learning tasks.

[0065] like Figure 2As shown, the system can be a distributed machine learning system running on one or more GPU-based computing nodes. The GPU-based computing node, also referred to as a GPU device, can include one or more GPU cards. Multiple machine learning tasks can run on one GPU card simultaneously, each machine learning task can run in parallel on multiple GPU cards, and these machine learning tasks share the computing resources of the GPU. In Figure 2 In the example shown in FIG. 1, node 1 includes GPU 1 and GPU 2, task D and task E run on GPU 1 simultaneously, task A, task B and task C run on GPU 2 simultaneously, and task A performs iterative training on node 1 and node 2 simultaneously. In addition, the GPU device also includes a central processing unit (CPU) and a memory, where the CPU can also be referred to as the host of the GPU.

[0066] The machine learning task, also referred to as a machine learning model training task, can learn a machine learning model from training data. The machine learning task can be a deep learning task, and the model can be a model based on a deep neural network. For example, the deep learning task can be a named entity recognition model, a speech recognition model, a product recommendation model, etc. learned from training data. The machine learning task can also be a non-deep learning task, and the model can be a model based on a general neural network or a non-neural network machine learning model such as a decision tree. The following will take the deep learning task as an example to illustrate the way the system dynamically adjusts the computing resources provided in this embodiment.

[0067] The system provided in this embodiment, through the cooperation of the computing resource coordinator 1 and the machine learning computing framework 2, realizes adaptive GPU computing resource dynamic scaling optimization for machine learning tasks. As shown in Figure 3 The computing resource coordinator 1 supports adaptive GPU computing resource adjustment and can be deployed in a GPU computing node to schedule and manage the computing resources of one or more GPUs in the node and dynamically adjust the usage of GPU computing resources for multiple tasks running on one GPU. The machine learning computing framework 2 corresponds to a machine learning task, and each task runs through the machine learning computing framework. As shown in Figure 3 As shown in FIG. 2, a computing node includes four GPUs (GPU 0 to GPU 3), where two tasks (task 0 and task 1) run on GPU 0 simultaneously, task 0 runs in the TensorFlow framework, and task 1 runs in the PyTorch framework.

[0068] The machine learning computing framework 2 can be an end-to-end machine learning platform, and various machine learning frameworks can have their own ecosystems, which include various tools, libraries, and other resources that can help developers easily build and deploy applications supported by machine learning. In this embodiment, the machine learning computing framework is a deep learning computing framework, including but not limited to TensorFlow, PyTorch, MXNet, Caffe, etc.

[0069] Compared with the machine learning computing framework of the prior art, the machine learning computing framework 2 provided in this embodiment adds a component for dynamically controlling the usage of GPU computing resources, which supports machine learning computing services with dynamic adjustment of computing resources and realizes dynamic control of the usage of GPU computing resources for machine learning tasks. Since the purpose of controlling the amount of computing resources used by the task is achieved by controlling the GPU operator emission mode of the task in this embodiment, the component is also referred to as a GPU operator control component.

[0070] In this embodiment, the computing resource coordinator 1 is configured to determine the priorities of a plurality of machine learning tasks running through a graphics processing unit (GPU), determine the computing resource usage information of the tasks and the running status information of high-priority tasks, adjust the upper limit of the computing resource usage of low-priority tasks according to the running status information and the usage information, and send a computing resource adjustment instruction to the machine learning framework of the low-priority tasks. Correspondingly, the machine learning framework is configured to control the computing resource usage of the low-priority tasks according to the upper limit included in the adjustment instruction.

[0071] The inventors of the present application have found through experiments that the usage of GPU computing resources by deep learning tasks presents volatility at various stages of running, which often results in idle computing resources. In order to make full use of the idle computing resources, the prior art simply runs multiple tasks on the same GPU at the same time. However, the volatility of the usage of computing resources also causes tasks to compete with each other for GPU computing resources, which affects the performance of “performance guarantee tasks”, thereby damaging the service level (SLA guarantee) of such tasks and even causing the failure of such tasks. In order to ensure the performance of such “performance guarantee tasks” and make full use of idle computing resources, the inventors propose a new data, i.e., the performance guarantee priority (referred to as priority for short) of a machine learning task. The higher the priority of a task, the more its performance needs to be guaranteed. This data is used as the data basis for the system to dynamically adjust the computing resource usage of deep learning tasks.

[0072] The computing resource coordinator 1 is configured to determine the priority of the plurality of machine learning tasks running through the GPU. The priority of the tasks can be determined according to the application requirements, such as setting only two priorities: high priority and low priority. For example, the learning task 1 of the named entity recognition model is a "performance guarantee task" with service level agreement, and the learning task 2 of the speech recognition model is a "speculative execution task" without service level agreement. The task 1 can be set as high priority, and the task 2 can be set as low priority.

[0073] In a specific implementation, a plurality of priorities can also be set. Table 1 shows the priority setting information of the machine learning tasks in an example.

[0074]

[0075]

[0076] Table 1, machine learning task table

[0077] As shown in Table 1, three task priorities are set in this embodiment, wherein the first priority is a high priority task, the second priority is a second high priority, and the third priority is a lowest priority. In this case, if there is a first priority task in the parallel task, the performance of the first priority task is guaranteed first; if there is no first priority task, the performance of the second priority task is guaranteed.

[0078] In an example, the system can be deployed on a server, and a task service device can be provided to a user through a client. The user can determine the machine learning task to be run through the task service device of the client, and set the task priority, such as selecting the "second priority". Different priorities can require different service fees. After determining the machine learning task to be run and setting the task priority, the user can submit a task running request to the server through the client. The server can store the priority information of the task in response to the request, and store the task in the task table. When the system runs the machine learning task through the machine learning framework 2, the computing resource coordinator 1 deployed in the computing node can obtain the priority information of the plurality of tasks to be run through the computing node from the server. Table 2 shows the machine learning task information in this embodiment.

[0079] Task identification User identification Task priority Task 1 (Named Entity Recognition Model) User A Primary (Performance Assurance Task) Task 2 (Named Entity Recognition Model) User B Secondary (Speculative Execution Task) Task 3 (Product Recommendation Model) User C Secondary (Speculative Execution Task) Task 4 (Language Model) User C Primary (Performance Assurance Task) …

[0080] Table 2, machine learning task table

[0081] As shown in Table 2, the learning task 1 of the named entity recognition model of the user A is a "performance guarantee task", and the learning task 2 of the named entity recognition model of the user B is a "speculative execution task". The priority of the "performance guarantee task" is higher than that of the "speculative execution task".

[0082] The computing resource coordinator 1 not only determines the priority of multiple machine learning tasks running through a graphics processing unit (GPU), but also determines the computing resource usage information of the tasks and the running status information of the high-priority task.

[0083] The computing resource usage information includes the computing resource utilization of the task process on the GPU. For example, Figure 3 Task 1 running on GPU 0 actually uses 30% of the computing resources, and task 2 actually uses 46% of the computing resources. The running status information of the high-priority task includes, but is not limited to, running performance information, the number of GPU cards used by the task, and the like. For the running characteristics of a deep learning task, the running performance information can be the number of iterations of the task running in a unit of time.

[0084] In addition, the computing resource coordinator 1 can also monitor the device status information of the GPU. The device status information includes, but is not limited to, GPU running power information, and the like. In this case, the computing resource coordinator 1 is specifically configured to adjust the upper limit value according to the running status information of the high-priority task, the usage information, and the device status information of the GPU.

[0085] As Figure 3 shown, in the present embodiment, the computing resource coordinator 1 deployed locally on the computing node can include a running information collection component and a resource control decision component. The computing resource coordinator 1 can dynamically adjust the computing resource usage of each GPU in the node. The computing resource coordinator 1 can collect, through the running information collection component therein, the computing resource utilization of each task process on the GPU and the GPU running power information from all GPU hardware on the node. In specific implementation, the running information collection component can collect the computing resource utilization and the GPU running power information periodically, such as collecting the information every 10 milliseconds. From Figure 3 It can be seen that the running information collection component in the present embodiment can collect task running status information from the deep learning computing framework of each task, which can include the number of GPU cards used by the task (such as task 1 using GPU 0 and GPU 1), the current running performance information of the task (such as task 1 running 10 times in 1 second), the computing resource utilization of the task on each GPU card used by the task, and the like.

[0086] In a specific implementation, the deep learning computing framework can collect statistics on task operation status information and GPU computing resource utilization, and submit the latest information to the operation information collection component in the computing resource coordinator 1. The operation information collection component can submit the collected operation status information and usage status information to the resource control decision component. The resource control decision component can dynamically adjust the upper limit of computing resource usage of low-priority tasks based on the real-time monitoring of the operation status information and usage status information of high-priority tasks.

[0087] In one example, if the operating status information of the high-priority task reaches the performance threshold, the upper limit of the computing resource usage of the low-priority task is increased based on the usage status information; if the operating status information of the high-priority task does not reach the performance threshold, the upper limit of the computing resource usage of the low-priority task is reduced based on the usage status information.

[0088] The performance threshold may be a threshold of the number of iterations that a task runs in a unit of time. For example, setting the performance threshold of task 1 to 20 times / second indicates that the task runs at least 20 times in 1 second. If the task actually runs 10 times in 1 second, it indicates that the running status information of the task does not meet the performance threshold.

[0089] like Figure 3 As shown, in this embodiment, the operation information collection component in the computing resource coordinator 1 can continuously monitor the task operation status information and computing resource usage information reported by each deep learning computing framework, and regularly monitor the GPU device status information. The resource control decision component can adjust the computing resource usage limit of low-priority tasks (such as all speculative execution tasks) based on the current performance data and computing resource usage of all high-priority tasks (such as performance-guaranteed tasks) on the current node, that is, the upper limit of computing resource usage.

[0090] For example, monitoring Figure 3 The high-priority task 1 running on GPU0 actually uses 40% of the computing resources and can reach the service level. At the same time, the low-priority task 2 running on GPU0 only uses 10% of the computing resources and has low running performance. The upper limit of the computing resource usage of the low-priority task can be appropriately increased to 20% to utilize idle computing resources to improve the running performance of the low-priority task. In this way, both high-priority tasks and low-priority tasks can have better running performance.

[0091] For another example, if it is detected that the performance of task 1 drops below a performance threshold, the upper limit of the computing resource usage of the low-priority task can be reduced to 15%, thereby ensuring the running performance of the high-priority task.

[0092] In one example, the computing resource coordinator is further configured to set a first upper limit of the computing resource usage of the low-priority task if a high-priority task to be executed occurs while the high-priority task is being executed. In this case, the high-priority task is scheduled to be executed first, and the low-priority task is scheduled to be executed later. The computing resource coordinator 1 can limit the GPU computing resource usage of the low-priority task to the first upper limit first. The first upper limit is also referred to as an initial upper limit. Thereafter, the upper limit of the computing resource usage of the low-priority task is gradually increased during the parallel execution of the two tasks, and the performance of the high-priority task is monitored in the process. If the performance of the high-priority task does not decrease, the upper limit of the GPU computing resource usage of the low-priority task can continue to be increased. In this way, the upper limit of the computing resource usage of the low-priority task is gradually increased from the first upper limit, which can effectively ensure the performance of the high-priority task.

[0093] In another example, the computing resource usage of the low-priority task is adjusted to the first upper limit if a high-priority task to be executed occurs while the low-priority task is being executed. In this case, the low-priority task is scheduled to be executed first, and the high-priority task is scheduled to be executed later. At this time, before the high-priority task is scheduled, the computing resource coordinator 1 can limit the GPU computing resource usage of the low-priority task to the first upper limit first, and then schedule the high-priority task. The performance of the high-priority task at this time is used as a benchmark performance (i.e., a performance threshold). Then, during the parallel execution of the two tasks, the GPU computing resource usage of the low-priority task is gradually increased, and the performance of the high-priority task is monitored in the process. If the performance of the high-priority task is not lower than the benchmark performance, the GPU computing resource usage of the low-priority task continues to be increased. In this way, the upper limit of the computing resource usage of the low-priority task is first reduced to the first upper limit, and then gradually increased from the first upper limit, which can effectively ensure the performance of the high-priority task.

[0094] It should be noted that in the case where the low-priority task is scheduled to be executed first and the high-priority task is scheduled to be executed later, the performance threshold of the high-priority task can be the actual performance of the low-priority task at the first upper limit. In actual implementation, the performance threshold can also be set artificially or calculated by an algorithm.

[0095] In this embodiment, the computing resource coordinator 1 issues the adjusted upper limit of the GPU computing resource usage of the low-priority task to the deep learning computing framework 2 that manages the execution of the task, and the framework 2 limits the GPU computing resource usage of the low-priority task.

[0096] The computing resource coordinator 1 can send a computing resource adjustment instruction to the machine learning framework of the low-priority task after re-determining the upper limit value for the low-priority task. The machine learning framework 2 controls the amount of computing resources used by the low-priority task according to the upper limit value included in the adjustment instruction.

[0097] The computing resource coordinator 1 can cyclically perform the entire execution process of data monitoring, upper limit value adjustment, and control of the amount of computing resources used by the low-priority task, so as to achieve adaptive GPU computing resource dynamic scaling for deep learning applications, and ultimately achieve the purpose of maximizing resource utilization while ensuring SLA, thereby significantly improving the throughput of cluster jobs.

[0098] The task can include multiple small-batch model iteration training during execution, and there can be several GPU operators in one small batch. The deep learning computing framework 2 needs to transmit these GPU operators to the GPU for execution. In this embodiment, the deep learning computing framework 2 supporting GPU computing resource dynamic scaling controls the number of GPU operators executed by the low-priority task per unit time through the GPU operator control component, so as to control the amount of GPU computing resources used by the low-priority task. The unit time can be determined according to application requirements, such as one unit time per second or one unit time per ten milliseconds.

[0099] Figure 4 The execution process of the deep learning computing framework 2 is shown. As shown in FIG. a, when the high-priority task Job-A is executed alone on a GPU card, the deep learning computing framework 2 can count the GPU computing resource usage and running status information of the task. In specific implementation, the GPU computing resource usage information can be obtained by calling a framework management library (such as NVIDIA management library NVML) interface. As shown in FIG. b, the deep learning computing framework 2 can count the GPU computing resource usage and running status information of the low-priority task Job-B. Figure 4 As shown in FIG. a, when the high-priority task Job-A is executed alone on a GPU card, the deep learning computing framework 2 can count the GPU computing resource usage and running status information of the task. In specific implementation, the GPU computing resource usage information can be obtained by calling a framework management library (such as NVIDIA management library NVML) interface. As shown in FIG. b, the deep learning computing framework 2 can count the GPU computing resource usage and running status information of the low-priority task Job-B. Figure 4 As shown in FIG. a, when the high-priority task Job-A is executed alone on a GPU card, the deep learning computing framework 2 can count the GPU computing resource usage and running status information of the task. In specific implementation, the GPU computing resource usage information can be obtained by calling a framework management library (such as NVIDIA management library NVML) interface. As shown in FIG. b, the deep learning computing framework 2 can count the GPU computing resource usage and running status information of the low-priority task Job-B.

[0100] However, the speed of transmitting GPU operators by different deep learning tasks is usually different. When the low-priority task and the high-priority task are executed together on the same GPU, the two tasks compete for the computing resources of the GPU. If the low-priority task transmits GPU operators faster than the high-priority task, the running performance of the high-priority task will be reduced.

[0101] As shown in FIG. a, when the high-priority task Job-A is executed alone on a GPU card, the deep learning computing framework 2 can count the GPU computing resource usage and running status information of the task. In specific implementation, the GPU computing resource usage information can be obtained by calling a framework management library (such as NVIDIA management library NVML) interface. As shown in FIG. b, the deep learning computing framework 2 can count the GPU computing resource usage and running status information of the low-priority task Job-B. Figure 4b, when another low-priority task Job-B is executed together with the high-priority task Job-A, Job-B and Job-A compete for the computing resources of the same GPU, and finally cause the performance of Job-A to decrease because Job-B emits GPU operators at a faster speed. The number of GPU operators that can be executed by Job-A in a unit of time decreases from 6 in Figure a to 3. At this time, the deep learning computing framework 2 monitors the performance decrease of Job-A and can immediately report the performance decrease value of Job-A to the computing resource coordinator 1 deployed locally on the GPU device. The computing resource coordinator 1 can determine the upper limit value of the computing resource usage of the low-priority task (Job-B) again according to the performance decrease of the high-priority task (Job-A) and the computing resource usage of the low-priority task (Job-B), and deliver the upper limit value to the deep learning computing framework 2 in which the low-priority task (Job-B) is executed.

[0102] In one example, the machine learning framework is specifically configured to determine a GPU operator emission frequency according to the upper limit value, and emit a GPU operator to a GPU according to the emission frequency. For example, as shown in Figure c, the deep learning computing framework in which Job-B is executed receives the new upper limit value of the computing resource usage, and determines an adjusted GPU operator emission frequency in a unit of time according to the upper limit value. For example, the adjustment instruction is to lower the upper limit value of the computing resource usage of the low-priority task, and the emission frequency of Job-B before the adjustment is 5 GPU operators per 10 milliseconds, and the emission frequency after the adjustment is 2 GPU operators per 10 milliseconds. After the GPU operator emission frequency is determined, a GPU operator is emitted to a GPU according to the emission frequency, so as to execute the GPU operator by the GPU. Figure 4 c, the deep learning computing framework in which Job-B is executed receives the new upper limit value of the computing resource usage, and determines an adjusted GPU operator emission frequency in a unit of time according to the upper limit value. For example, the adjustment instruction is to lower the upper limit value of the computing resource usage of the low-priority task, and the emission frequency of Job-B before the adjustment is 5 GPU operators per 10 milliseconds, and the emission frequency after the adjustment is 2 GPU operators per 10 milliseconds. After the GPU operator emission frequency is determined, a GPU operator is emitted to a GPU according to the emission frequency, so as to execute the GPU operator by the GPU.

[0103] In specific implementation, the machine learning framework 2 can be specifically configured to determine a time slot length and an insertion position before a GPU operator according to the emission frequency, and insert a time slot when emitting a GPU operator to a GPU according to the time slot length and the insertion position. The time slot refers to a waiting time before emitting a GPU operator. Inserting a time slot can cause the task to pause sending a GPU operator, and continue sending the GPU operator after the time length of the time slot.

[0104] After the machine learning framework 2 determines the emission frequency, a fixed number of GPU operators can be emitted to a GPU in a unit of time according to the frequency information. The machine learning framework 2 controls the number of GPU operators emitted to a GPU in a unit of time by inserting a time slot before a GPU operator, so as to meet the requirement of the frequency, thereby achieving the purpose of controlling the GPU computing resource usage of a low-priority task.

[0105] The time slot length can be determined based on the unit time length, the number of GPU operators launched per unit time, and the runtime of a GPU operator. The insertion position can be determined based on the launch time of the GPU operator of the high-priority task. In practice, the length of the time slot inserted before each GPU operator can be the same or different. For example, if the unit time length is 10 milliseconds, the number of GPU operators launched per unit time is 2, and the runtime of a GPU operator is 2 milliseconds, the time slot length can be 3 milliseconds.

[0106] Depend on Figure 4 As can be seen from c, the machine learning framework 2 calculates the size and position of the time slot to be inserted according to the new operator emission frequency through the GPU operator control component. Figure 4 The GPU operator control component in Figure c inserts time slots during the GPU operator launch process to control the number of GPU operators launched per unit time for Job-B, thereby controlling the execution time of Job-B's GPU operators and ultimately controlling Job-B's computing resource usage. By controlling the number of GPU operators launched per unit time for Job-B, the number of operators launched and executed per unit time for Job-A (Figure 4c) is restored to the number it would have if it were executed independently. As a result, Job-A's performance is restored, ensuring the SLA.

[0107] It should be noted that during this process, the launch and execution of the CPU operators in the low-priority task Job-B are not affected. This can improve the utilization of GPU computing resources and CPU computing resources while ensuring the performance of the high-priority task Job-A, thereby significantly improving the cluster job throughput.

[0108] When implementing, Figure 5 After the dynamic adjustment of the computing resource usage of the low-priority task in c, the deep learning computing framework 2 of the high-priority task will continue to monitor the operating status information of the high-priority task running on it, and report the information to the computing resource coordinator 1, so as to continuously and adaptively adjust the computing resource usage of the low-priority task according to the operating status of the high-priority task during the operation of the high-priority task. While ensuring the service level of the high-priority task, the low-priority task will obtain more available computing resources as much as possible, improve the utilization of computing resources, and thus significantly improve the cluster job throughput.

[0109] As can be seen from the above embodiments, the machine learning system provided in the embodiments of the present application determines the priorities of a plurality of machine learning tasks running through a graphics processing unit (GPU), determines the computing resource usage status information of the tasks and the running status information of high-priority tasks, adjusts the upper limit of the computing resource usage of low-priority tasks according to the running status information and the usage status information, sends a computing resource adjustment instruction to the machine learning framework of the low-priority tasks, and controls the computing resource usage of the low-priority tasks according to the upper limit included in the adjustment instruction by the machine learning framework. This processing manner enables dynamic scaling optimization of the GPU computing resources used by the low-priority tasks running in parallel on one GPU according to the running performance of the high-priority tasks, so that the idle computing resources can be allocated to the low-priority tasks for use on the premise of guaranteeing the performance of the high-priority tasks. Therefore, the GPU computing resource utilization can be effectively improved while guaranteeing the performance of the high-priority tasks.

[0110] Second embodiment

[0111] Corresponding to the machine learning system described above, the present application further provides a machine learning system. The same parts of the present embodiment as those of the first embodiment will not be described herein again, and please refer to the corresponding parts in the first embodiment.

[0112] Please refer to Figure 6 which is a structural schematic diagram of an embodiment of the machine learning system of the present application. The system provided in the present embodiment includes a client 51 and a server 52.

[0113] The client is configured to send priority information of machine learning tasks to the server, and the server is configured to determine computing resource usage status information of machine learning tasks running through a graphics processing unit (GPU) and running status information of high-priority tasks, adjust the upper limit of the computing resource usage of low-priority tasks according to the running status information and the usage status information, and send a computing resource adjustment instruction to the machine learning framework of the low-priority tasks.

[0114] The client includes but is not limited to mobile communication devices, i.e., commonly known as mobile phones or smart phones, and also includes personal computers, PADs, iPads and other terminal devices. The server can run machine learning tasks on a GPU cluster.

[0115] In one example, the server can be further configured to determine performance information of machine learning tasks, and adjust the priorities of the tasks according to the performance information.

[0116] For example, the priority of task A was originally level 2, but the system did not enable the performance of the task to meet the "level 2" service level requirements required by the user. In this case, the priority of the task can be adjusted to level 1 so that its actual performance can meet the "level 2" service level requirements required by the user.

[0117] In practice, changes in computing resources during machine learning tasks can be recorded and, based on this information, the task's priority can be adjusted. For example, if a high-priority task only uses 60% of computing resources 30% of the time, indicating that the task's performance does not meet service level requirements, the task's priority can be raised.

[0118] In another example, the server can also be used to determine performance information of a machine learning task; and determine serviceable machine learning tasks based on the performance information.

[0119] For example, if the service level requirements of Task A and Task B can be met through the system, but the service level requirement of Task C cannot be met, resource management services can be provided for Task A and Task B.

[0120] As can be seen from the above embodiments, the machine learning system provided by the embodiments of the present application sends the priority information of the machine learning task to the server through the client; the server determines the computing resource usage status information of the machine learning task run by the graphics processing unit GPU, and the running status information of the high-priority task; according to the running status information and the usage status information, adjusts the upper limit of the computing resource usage of the low-priority task; sends a computing resource adjustment instruction to the machine learning framework of the low-priority task; this processing method enables the dynamic scaling optimization of the GPU computing resources used by the low-priority tasks running in parallel on a GPU according to the running performance of the high-priority task, so that the idle computing resources can be fully utilized and allocated to the low-priority tasks under the premise of ensuring the performance of the high-priority task; therefore, the GPU computing resource utilization rate can be effectively improved while ensuring the performance of the high-priority task.

[0121] Third embodiment

[0122] Corresponding to the above-mentioned machine learning system, the present application also provides a computing resource management method. The parts of this embodiment that are the same as those in the first embodiment will not be repeated here, and please refer to the corresponding parts in the first embodiment.

[0123] Please refer to Figure 7 , which is a flow chart of an embodiment of the computing resource management method of the present application. The method provided in this embodiment may include the following steps:

[0124] Step S601: determining priorities of a plurality of machine learning tasks running through a graphics processing unit (GPU).

[0125] Step S603: determining the computing resource usage information of the tasks and the running status information of the high-priority tasks.

[0126] The running status information includes, but is not limited to, running performance information. The running performance information can be determined according to the number of iterations of the tasks in a unit of time.

[0127] Step S605: controlling the amount of computing resource used by the low-priority tasks according to the running status information and the usage information.

[0128] In one example, step S605 can include the following steps:

[0129] Step S6051: adjusting the upper limit of the amount of computing resource used by the low-priority tasks according to the running status information and the usage information.

[0130] In one example, step S6051 can be implemented in the following manner: if the running status information of the high-priority tasks reaches a performance threshold, the upper limit of the amount of computing resource used by the low-priority tasks is increased according to the usage information. Correspondingly, if the running status information of the high-priority tasks does not reach the performance threshold, the upper limit of the amount of computing resource used by the low-priority tasks is decreased according to the usage information.

[0131] In one example, the method can further include the following step: if there is a low-priority task to be run while a high-priority task is running, setting a first upper limit of the amount of computing resource used by the low-priority task.

[0132] In one example, the method can further include the following step: if there is a high-priority task to be run while a low-priority task is running, adjusting the amount of computing resource used by the low-priority task to the first upper limit.

[0133] Step S6053: controlling the speed of sending GPU operators to the GPU by the low-priority tasks according to the upper limit.

[0134] In one example, step S6053 can include the following sub-steps: 1) determining a GPU operator sending frequency according to the upper limit; and 2) sending GPU operators to the GPU according to the sending frequency to execute the GPU operators through the GPU.

[0135] As can be seen from the above embodiments, the computing resource management method provided in the embodiments of the present application determines the priorities of a plurality of machine learning tasks running through a graphics processing unit (GPU), determines the computing resource usage information of the tasks and the running state information of high-priority tasks, and controls the amount of computing resource used by low-priority tasks according to the running state information and the usage information. This processing manner enables dynamic scaling optimization of the GPU computing resources used by low-priority tasks running in parallel on one GPU according to the running performance of high-priority tasks, so that the idle computing resources can be allocated to low-priority tasks for use on the premise of guaranteeing the performance of high-priority tasks, thereby effectively improving the utilization rate of GPU computing resources while guaranteeing the performance of high-priority tasks.

[0136] The fourth embodiment

[0137] In the above embodiments, a computing resource management method is provided, and the present application further provides a computing resource management apparatus corresponding thereto. The apparatus corresponds to the embodiments of the above method. Since the apparatus embodiments are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments. The apparatus embodiments described below are merely illustrative.

[0138] For reference ​ which is a structural schematic diagram of an embodiment of the computing resource management apparatus of the present application. The present application further provides a computing resource management apparatus, comprising:

[0139] a priority determination unit configured to determine the priorities of a plurality of machine learning tasks running through a graphics processing unit (GPU);

[0140] an information collection unit configured to determine the computing resource usage information of the tasks and the running state information of high-priority tasks;

[0141] a computing resource adjustment unit configured to control the amount of computing resource used by low-priority tasks according to the running state information and the usage information.

[0142] The fifth embodiment

[0143] In the above embodiments, a computing resource management method is provided, and the present application further provides an electronic device corresponding thereto. The apparatus corresponds to the embodiments of the above method. Since the device embodiments are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments. The device embodiments described below are merely illustrative.

[0144] The electronic device of the embodiment comprises a processor and a memory; the memory is used to store a program for implementing a computing resource management method; after the device is powered on and the program of the method is run by the processor, the following steps are executed: determining priorities of a plurality of machine learning tasks run by a graphics processing unit (GPU); determining computing resource usage information of the tasks and running status information of high-priority tasks; and controlling an amount of computing resource used by low-priority tasks according to the running status information and the usage information.

[0145] Although the present application is disclosed with reference to the preferred embodiments above, it is not intended to limit the present application, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application, and therefore the protection scope of the present application should be subject to the scope defined by the claims of the present application.

[0146] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0147] The memory can include non-persistent memory in computer readable media, random access memory (RAM), and / or non-volatile memory such as read only memory (ROM) or flash memory (flash RAM). The memory is an example of computer readable media.

[0148] 1. Computer readable media includes permanent and non-permanent, removable and non-removable media can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this paper, computer readable media does not include non-transitory computer readable media (transitory media) such as modulated data signals and carriers.

[0149] 2. Those skilled in the art will appreciate that the embodiments of the present application can be devised for as methods, systems or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon for use by or in connection with an instruction execution system.

Claims

1. A machine learning system, characterized in that include: Computational resource orchestrator and machine learning framework; The computing resource coordinator is configured to prioritize multiple machine learning tasks executed by a graphics processing unit (GPU); Determining computing resource usage information of the task and operating status information of the high-priority task, the operating status information including operating performance information and computing resource utilization of the task on the GPU card used by the task; if the operating status information of the high-priority task reaches a performance threshold, increasing an upper limit of computing resource usage of the low-priority task based on the usage information; if the operating status information of the high-priority task does not reach the performance threshold, reducing an upper limit of computing resource usage of the low-priority task based on the usage information; and sending a computing resource adjustment instruction to the machine learning framework of the low-priority task; The machine learning framework is used to send the operating status information of the high-priority task to the computing resource coordinator if the operating status information of the high-priority task changes; and control the computing resource usage of the low-priority task according to the upper limit value included in the adjustment instruction.

2. The system according to claim 1, wherein: The computing resource coordinator is further configured to set a first upper limit value for the computing resource usage of the low-priority task if a low-priority task to be run appears when the high-priority task is being run.

3. The system according to claim 1, wherein: The computing resource coordinator is further configured to adjust the computing resource usage of the low-priority task to a first upper limit value if a high-priority task to be run appears while the low-priority task is being run.

4. The system according to claim 1, wherein: The machine learning framework is specifically used to determine the GPU operator emission frequency according to the upper limit value; and emit the GPU operator to the GPU according to the emission frequency so as to execute the GPU operator through the GPU.

5. The system according to claim 4, characterized in that The machine learning framework is specifically used to determine the length and insertion position of the time slot before inserting into the GPU operator according to the transmission frequency; and insert the time slot when transmitting the GPU operator to the GPU according to the time slot length and insertion position.

6. A machine learning system, characterized in that include: The client sends the priority information of the machine learning task to the server; The server is used to determine the computing resource usage status information of the machine learning task running through the graphics processing unit (GPU) and the running status information of the high-priority task, wherein the running status information includes the running performance information and the computing resource utilization rate of the task on the GPU card used by the task; if the running status information of the high-priority task reaches the performance threshold, then the upper limit of the computing resource usage of the low-priority task is increased according to the usage information; if the running status information of the high-priority task does not reach the performance threshold, then the upper limit of the computing resource usage of the low-priority task is reduced according to the usage information; and a computing resource adjustment instruction is sent to the machine learning framework of the low-priority task.

7. A computing resource management method, characterized in that: include: Prioritize multiple machine learning tasks running on graphics processing units (GPUs); Determine computing resource usage status information of the task and running status information of the high-priority task, wherein the running status information includes running performance information and computing resource utilization rate of the task on the GPU card used by the task; If the operating status information of the high-priority task reaches the performance threshold, the upper limit of the computing resource usage of the low-priority task is increased based on the usage status information; if the operating status information of the high-priority task does not reach the performance threshold, the upper limit of the computing resource usage of the low-priority task is reduced based on the usage status information.

8. The method according to claim 7, characterized in that The controlling the computing resource usage of the low-priority task according to the operating status information and the usage status information includes: Adjusting an upper limit of computing resource usage of low-priority tasks according to the operating status information and the usage status information; The speed at which low-priority tasks transmit GPU operators to the GPU is controlled according to the upper limit.

9. The method according to claim 7, characterized in that Also includes: If a low-priority task to be run appears when a high-priority task is being run, a first upper limit value of the computing resource usage of the low-priority task is set.

10. The method according to claim 7, characterized in that Also includes: If a high-priority task to be run appears while a low-priority task is being run, the computing resource usage of the low-priority task is adjusted to a first upper limit value.

11. The method according to claim 7, characterized in that The controlling the speed at which the low-priority task transmits the GPU operator to the GPU according to the upper limit value includes: Determine the GPU operator emission frequency according to the upper limit value; The GPU operator is transmitted to the GPU according to the transmission frequency, so that the GPU operator is executed by the GPU.

12. The method according to claim 7, characterized in that The operating status information includes operating performance information, which is determined in the following manner: The running performance information is determined according to the number of iterative trainings of the task within a unit time.

13. A computing resource management device, characterized in that: include: a prioritization unit for prioritizing multiple machine learning tasks running through a graphics processing unit (GPU); An information collection unit, configured to determine computing resource usage information of the task and operating status information of high-priority tasks, wherein the operating status information includes operating performance information and computing resource utilization of the task on the GPU card used by the task; A computing resource adjustment unit is used to increase the upper limit of computing resource usage of low-priority tasks based on the usage information if the running status information of the high-priority task reaches the performance threshold; if the running status information of the high-priority task does not reach the performance threshold, then reduce the upper limit of computing resource usage of low-priority tasks based on the usage information.

14. An electronic device, characterized in that: include: processor and memory; A memory is used to store a program for implementing the computing resource management method according to claims 7 to 12, and the device is powered on and runs the program of the method through the processor.

Images