GPU resource management device and management method therefor

Abstract

The present invention relates to a GPU resource management device and a management method therefor, wherein the GPU resource management device performs GPU resource control through a sidecar pattern of a container cluster. The GPU management device according to the present invention comprises: a scheduler for allocating GPU resources to a container in a node selected in response to a container GPU computing resource request made by a client and performing a task requested at a cluster level; and a pod included in the node and managing, as a group, a plurality of containers configured for a single task. In addition, the pod includes: a main container to which GPU computing resources generated via the scheduler are allocated; and a sidecar container which monitors a memory utilization rate of GPUs and a utilization rate of the GPU computing resources of a process, and controls a computing resource utilization rate of a GPU container according to the monitoring result.

Description

GPU resource management device and management method thereof

[0001] The disclosed invention relates to a GPU resource management device and a management method thereof that performs GPU resource control through a sidecar pattern of a container cluster.

[0002] GPU sharing scheduling is a technique that can increase GPU availability because it allows multiple containers to share and utilize a GPU. Without GPU sharing scheduling, a GPU device is always allocated exclusively to a single container.

[0003] However, these techniques for improving GPU availability are not easily utilized in container-based cluster environments. For example, prior literature 2 (GPU-Operator About the NVIDIA GPU Operator, NVIDIA GPU Operator 23.9.0 documentation) allows multiple pods to utilize the same GPU, but does not provide GPU resource isolation. Therefore, it is unsuitable for cluster operation targeting commercial services in various fields.

[0004] And although prior art document 3 (MPS: Multi-Process Service: GPU Deployment and Management Documentation (nvidia.com)) discloses resource isolation technology based on MPS, prior art document 3 supports isolation of computational resources and memory resources at the process level. This is not a container-level isolation technology, and therefore, there is a problem in that resource isolation at the container level is not possible when multiple GPU-utilizing processes are executed within a container.

[0005] Meanwhile, in a container-based cluster environment, GPU sharing scheduling provides a controller that controls resources on each node. Therefore, the controller comprehensively manages GPU resources within each node, allowing it to allocate precise amounts of resources to each container based on its resource requirements. However, when the controller manages multiple processes, it cannot immediately allocate computational resources, resulting in delayed computational processing times.

[0006] For the above reasons, the present invention provides a device and method capable of efficiently managing GPU resources without using a centralized GPU resource controller for each node.

[0007] The present invention provides a device and method for managing GPU resources through a sidecar pattern of a container cluster.

[0008] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0009] In order to achieve the above object, according to an embodiment of the present invention, a GPU resource management device includes a scheduler that allocates GPU resources to a container in a selected node according to a container GPU computational resource request requested by a client and performs a task requested at a cluster level; and a pod included in the node and managing a plurality of containers configured for a single task as a group, wherein the pod is configured to include a main container to which GPU computational resources generated through the scheduler are allocated, and a sidecar container that monitors the memory utilization rate and GPU computational resource utilization rate of the GPU of a process, and controls the computational resource utilization rate of the GPU container according to the monitoring result.

[0010] The present invention can be utilized in a container-based cluster.

[0011] The above main container is linked with the GPU computational resource management tool provided by the sidecar container through API Hooking when executing GPU tasks.

[0012] The above sidecar container terminates the process running in the current container if it consumes too much memory through API Hooking.

[0013] The above sidecar container kills the process for memory consumption that cannot be confirmed through API Hooking.

[0014] The above API Hooking performs pre-checking of GPU memory resource allocation API.

[0015] In the present invention, the sidecar container includes a rate limiter that schedules computational resources according to the amount of GPU computational resources.

[0016] The above rate limiter controls the number of API requests based on the GPU computational resources allocated to the pod.

[0017] The above scheduler induces GPU computational resource utilization by distinguishing between pod affinity and pod anti-affinity depending on the GPU task type of batch and iterative tasks.

[0018] Another feature of the present invention relates to a method for managing GPU resources, comprising: a scheduler; a pod having a main container and a sidecar container; and a method for scheduling sidecar-based GPU sharing in a container-based cluster environment, wherein the sidecar container is characterized in that, when a process currently running in the container exceeds memory consumption through API Hooking, the process is stopped or a GPU computational resource utilization API is scheduled according to the requested computational resource amount.

[0019] The above sidecar container kills the process for memory consumption that cannot be confirmed through API Hooking.

[0020] The above GPU computational resources control the computational resource utilization rate by using a rate limiter that controls the request frequency of the GPU computational resource utilization API.

[0021] The utilization rate of the above GPU computational resource API is to share the request frequency control of the GPU computational resource utilization API among multiple processes at the container level, and to provide opportunities for scheduling the computations of the above processes in a fair manner.

[0022] The above fair method may be a round robin method.

[0023] Scheduling of the above GPU computational resource utilization API includes a step of determining a waiting time required for a GPU computational resource utilization API request before the main container performs a GPU computation; and a step of waiting until the GPU computational resource utilization API can be requested based on the determination.

[0024] The waiting time required for the above GPU computational resource utilization API request is determined by the difference between the currently utilized computational resources and the computational resources allocated to the container.

[0025] If the above computational resources are insufficient compared to the resources allocated to the Pod, the request latency of the GPU computational resource utilization API decreases, and if the above computational resources are more allocated than the resources allocated to the Pod, the request latency of the GPU computational resource utilization API increases.

[0026] Scheduling of the above GPU computational resource utilization API is performed within the size of the basic allocated computational resources depending on whether there are idle computational resources, and if there are idle computational resources, the limit size of the computational resources that can be allocated is increased in proportion to the size of the idle computational resources, and if there are no idle computational resources, the limit size is decreased until there are a certain size of idle computational resources.

[0027] The size of the above-mentioned basic allocated computational resources is limited to the value obtained by subtracting the minimum idle computational resources from the total computational resources of the GPU.

[0028] According to one aspect of the disclosed invention, since resource management is performed through a sidecar pattern of a container cluster, resource management of a container can be performed immediately regardless of resource utilization of other containers within a server, thereby speeding up computational processing time.

[0029] According to another aspect of the present invention, memory resource control is possible even when multiple processes are executed in a single container.

[0030] FIG. 1 is a configuration diagram of a GPU resource management device according to an embodiment of the present invention.

[0031] FIG. 2 is a configuration diagram illustrating a work process according to GPU memory capacity using a GPU resource management device of the present invention.

[0032] FIG. 3 is a schematic diagram illustrating a work process for GPU computational resource utilization using a GPU resource management device of the present invention.

[0033] Throughout the specification, the same reference numerals denote the same components. This specification does not describe all elements of the embodiments, and any content that is general in the technical field to which the disclosed invention belongs or that overlaps between the embodiments is omitted. The terms 'part, module, element, block' used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple 'parts, modules, elements, blocks' may be implemented as a single component, or a single 'part, module, element, block' may include multiple components.

[0034] Throughout the specification, when a part is said to be 'connected' to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0035] Additionally, when a part is said to 'include' a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0036] Throughout the specification, when we say that an element is located 'on' another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0037] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0038] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0039] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0040]

[0041] The operating principle and embodiments of the disclosed invention are described below with reference to the attached drawings.

[0042] FIG. 1 is a configuration diagram of a GPU resource management device according to an embodiment of the present invention.

[0043] Referring to FIG. 1, a GPU resource management device (100) includes a scheduler (110) and a node (120).

[0044] The scheduler (110) can perform scheduling on a selected node at the request of a client. Specifically, it allocates GPU resources to a container based on the container GPU resource request made by the client and performs the task of requesting this at the cluster level.

[0045] A node (120) is a physical machine on which a pod (130) runs. The node (120) may include a pod (130). A pod (130) may be a unit for managing multiple containers configured for a single task as a group. As mentioned above, the embodiment will be described as a unit that manages containers required for a specific task as a group.

[0046] According to an embodiment, the pod (130) may include a sidecar container (150 sidecar container) and a main container (140, main container).

[0047] The sidecar container (150) runs together with the created container, and through API Hooking, it can stop the process running in the current container if it exceeds the memory consumption (GPU memory operation) or schedule the computational resources according to the requested computational resources (GPU computational resource utilization related operation). That is, the sidecar container (150) monitors the GPU memory utilization and GPU computational resource utilization of the running process, and controls the resource utilization of the GPU container according to the monitoring results. Here, the API Hooking performs a preliminary check of the GPU memory resource allocation API. In addition, the sidecar pattern of the sidecar container (150) can be defined in a way that a container that assists the main controller in the pod (130) is run together.

[0048] Additionally, the sidecar container (150) may include a rate limiter (see FIG. 3, 152) for controlling tasks related to GPU computational resource utilization. The rate limiter (152) serves to control the request frequency of the GPU computational resource utilization API.

[0049] The main container (140) refers to a container to which GPU resources generated through the scheduler (110) are allocated. When executing a GPU task, the main container (140) can be linked with a GPU resource management tool provided by the sidecar container (150) through the API Hooking.

[0050] In this way, the present embodiment has the advantage of performing resource management based on a sidecar, thereby enabling resource management of a container to be performed immediately regardless of resource utilization of other containers within the server.

[0051] FIG. 2 is a configuration diagram explaining a work process according to GPU memory capacity using a GPU resource management device of the present invention.

[0052] Referring to FIG. 2, according to one embodiment, the sidecar container (150) monitors the GPU memory utilization of processes. That is, the sidecar container (150) compares the container's memory consumption with the container memory limit. Here, the container memory consumption may be measured and recorded using NVML.

[0053] And when the container memory consumption exceeds the container memory limit, the sidecar container (150) causes the main container (140) to terminate the process.

[0054] Meanwhile, before allocating GPU memory, the main container (140) may further perform a process of checking whether the sum of the memory size collected through the sidecar container (160) and the currently allocated memory size exceeds the memory size allocated to the container. If the memory size is exceeded according to this process, an error will be generated.

[0055] In another embodiment, the main container (140) performs the task of managing processes even when there is memory consumption that cannot be confirmed through API hooking during GPU task execution. Specifically, when memory consumption occurs, such as memory allocation during a Cuda driver call, an OOM (Out of Memory) Killer task is performed to kill the process to prevent excessive memory consumption.

[0056] In this way, the sidecar container (150) can determine memory overflow and terminate or stop processes. Memory resource control is particularly possible even when multiple processes are running in a single container. Furthermore, because resource control is not performed through a node-specific controller, container resource control is possible regardless of the allocated containers and resource utilization status within the node. Furthermore, context memory can be controlled through container management.

[0057]

[0058] FIG. 3 is a schematic diagram illustrating a work process for GPU computational resource utilization using a GPU resource management device of the present invention.

[0059] For GPU compute resources, resource utilization is controlled through a rate limiter (152) that controls the request latency of the GPU compute resource utilization API. This method of resource utilization enables scheduling of the GPU compute resource utilization API regardless of the resource utilization of multiple containers within a node, and has the advantage of enabling resource control without a centralized node controller. In other words, although precise numerical resource control is difficult, it is possible because GPU compute resource utilization between each container is not considered. Previously, when multiple processes existed within a container, GPU compute resources could not be utilized among them. Therefore, processes were put in a state of competition when executing compute tasks, which resulted in an imbalanced resource allocation problem between processes.

[0060] Scheduling of the GPU computation resource utilization API first checks the request latency of the GPU computation resource utilization API before the sidecar container (150) performs the GPU computation. As a result of the check, the GPU computation resource utilization API will be performed after the request latency of the GPU computation resource utilization API.

[0061] At this time, the request latency for the GPU compute resource utilization API is determined by the difference between the currently utilized compute resources and the compute resources allocated to the container. If compute resources are underutilized, the request latency for the GPU compute resource utilization API decreases. Conversely, if compute resources are overutilized, the request latency for the GPU compute resource utilization API increases.

[0062] In this way, the present embodiment can manage the utilization of computational resources among multiple processes executed within a container through a sidecar container (150). At this time, managing the utilization of computational resources means configuring a token that is checked before performing computations to be shared among multiple processes at the container level, not the process level. In addition, even if the token is shared, in order to configure the computations of the processes to be executed in a balanced manner, an opportunity for the computations of each process to be scheduled is provided in a fair manner. An example of fair scheduling at this time is round-robin scheduling, and round-robin scheduling means giving an opportunity to perform a computation by circulating the currently executing GPU-utilizing process, and if a process does not request computational performance, the opportunity of the process is skipped and the opportunity is passed to the next process.

[0063] Furthermore, the sidecar-based GPU resource management method can provide a way to flexibly utilize idle compute resources depending on the presence of idle compute resources. Specifically, if there are idle compute resources, the limit size of the compute resources that can be allocated can be increased in proportion to the size of the idle compute resources. Conversely, if there are no idle compute resources, the limit size is decreased until a certain amount of idle compute resources exist. At this time, the range of increase or decrease in the compute resource limit will be limited to the size of the default allocated compute resources (total compute resources of the GPU - minimum idle compute resources).

[0064]

[0065] The present invention also provides a method for efficiently utilizing computational resources depending on the type of GPU work.

[0066] Specifically, the scheduler (110) can induce effective GPU resource utilization by defining pod affinity and pod anti-affinity depending on the type of GPU task. Here, GPU tasks can be divided into interaction tasks and batch task types. Interaction tasks mean utilizing GPU resources only when there is a user request, such as an IDE (Integrated Development Environment)-based development process or an inference task, and batch tasks mean tasks that continuously utilize computational resources while a process is executed through a data set stored in storage.

[0067] This embodiment can induce that if the types of GPU tasks are the same, the tasks can be placed on different nodes, and if the types of GPU tasks are different, the tasks can be placed on the same node. For example, since pod affinity / disaffinity refers to the affinity / disaffinity of the pod being scheduled to existing deployed pods from the perspective of the pod being scheduled, if the affinity of the pod being scheduled to an existing deployed pod is specified, the pod can be scheduled on the same node as the pod, and on the other hand, if the disaffinity is specified, the pod can be prevented from being scheduled on the node where the disagree pod is running.

[0068] Batch jobs that continuously utilize computational resources in this way can flexibly utilize additional computational resources when interactive tasks are idle, in addition to the basic computational resources allocated to the container. This allows batch jobs to terminate more quickly, and because interactive tasks do not occupy resources when idle, these resources can be provided more cheaply.

[0069]

[0070] As such, it can be seen that the present invention provides a method for performing sidecar-based GPU sharing scheduling in a container-based cluster environment.

[0071]

[0072] The above description is merely an illustrative illustration of the technical idea of ​​the present invention, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate, rather than limit, the technical idea of ​​the present invention, and the scope of the technical idea of ​​the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

[0073] The present invention can be usefully applied to GPU computing resource technology, etc.

Claims

1. A scheduler that allocates GPU resources to containers on selected nodes based on container GPU compute resource requests requested by clients and performs tasks requested at the cluster level; and It is equipped on the above node and includes a pod that manages multiple containers configured for a single task as a group, A GPU resource management device characterized in that the above pod is configured with a sidecar container that monitors the GPU memory utilization rate and GPU computational resource utilization rate of the main container and process to which the GPU computational resources generated through the above scheduler are allocated, and controls the computational resource utilization rate of the GPU container based on the monitoring results.

2. In paragraph 1, The above main container is, A GPU resource management device that links with the GPU computational resource management tool provided by the sidecar container through API Hooking when executing GPU tasks.

3. In paragraph 1, The above sidecar container, A GPU resource management device that terminates a process running in the current container if it consumes excessive memory through API Hooking.

4. In paragraph 1, The above sidecar container, A GPU resource management device that kills processes for memory consumption that cannot be confirmed through API Hooking.

5. In paragraph 3 or 4, The above API Hooking is a GPU resource management device that performs pre-checking of GPU memory resource allocation API.

6. In paragraph 1, The above sidecar container, A GPU resource management device including a rate limiter that schedules computational resources according to the amount of GPU computational resources.

7. In paragraph 6, The above rate limiter is, A GPU resource management device that controls the number of API requests based on the GPU computational resources allocated to the above pod.

8. In paragraph 6, The above rate limiter is, A GPU resource management device that controls the request frequency of the GPU computational resource utilization API.

9. In paragraph 1, The above scheduler, A GPU resource management device that induces GPU computational resource utilization by distinguishing between pod affinity and pod anti-affinity depending on the GPU work type of batch and iterative work.

10. Scheduler; A method for scheduling sidecar-based GPU sharing in a container-based cluster environment, including a pod with a main container and a sidecar container. The above sidecar container, A method for managing GPU resources, characterized by terminating a process running in the current container when memory consumption exceeds the limit through API Hooking, or scheduling a GPU computational resource utilization API according to the requested computational resource amount.

11. In paragraph 10, The above sidecar container, A method of managing GPU resources that kills processes for memory consumption that cannot be confirmed through API Hooking.

12. In paragraph 10, The above GPU computational resources are: A method for managing GPU resources that controls computational resource utilization by using a rate limiter that controls the request frequency of the GPU computational resource utilization API.

13. In paragraph 12, The above computational resource utilization rate is, The request frequency control of the above GPU computational resource utilization API is shared among multiple processes at the container level. A method of managing GPU resources that provides fair opportunities for scheduling operations of the above processes.

14. In paragraph 13, The above fair method is a round-robin method, a method of managing GPU resources.

15. In paragraph 10, Scheduling of the above GPU computational resource utilization API is A step for determining the waiting time required for a GPU operation resource utilization API request before the above main container performs GPU operation; and A method for managing GPU resources, including a step of waiting until the GPU computational resource utilization API can be requested based on the above judgment.

16. In paragraph 15, A method of managing GPU resources, wherein the waiting time required for the above GPU computational resource utilization API request is determined by the difference between the computational resources currently being utilized and the computational resources allocated to the container.

17. In paragraph 16, If the above computational resources are insufficient compared to the resources allocated to the Pod, the request latency of the GPU computational resource utilization API is reduced. A method for managing GPU resources, wherein the request waiting time of the GPU computational resource utilization API increases when the above computational resources are allocated more than the resources allocated to the above pod.

18. In paragraph 10, Scheduling of the above GPU computational resource utilization API is It is executed within the size of the default allocated computational resources, depending on the presence of idle computational resources. If there are idle operator resources, the limit size of the operator resources that can be allocated increases in proportion to the size of the idle operator resources. A method for managing GPU resources, wherein, if there are no idle computation resources, the limit size is reduced until a certain size of idle computation resources exists.

19. In paragraph 18, A method for managing GPU resources, wherein the size of the above-mentioned basic allocated computational resources is limited to a value obtained by subtracting the minimum idle computational resources from the total computational resources of the GPU.

Images