GPU resource management device and management method thereof

Abstract

The present invention relates to a GPU resource management device and a method for managing GPU resources that perform GPU resource control through a sidecar pattern of a container cluster. The GPU resource management device of the present invention comprises: a scheduler that allocates GPU resources to a container at a selected node in accordance with a request for container GPU computing resources requested by a client and performs a task requested at the cluster level; and a Pod that is included in the node and manages a group of multiple containers configured for a single task. The Pod includes a main container to which GPU computing resources created through the scheduler are allocated, and a sidecar container that monitors the memory utilization rate and GPU computing resource utilization rate of the process's GPU and controls the computing resource utilization rate of the GPU container according to the monitoring results.

Description

Technology Field

[0001] The disclosed invention relates to a GPU resource management device and a management method thereof that performs GPU resource control through a sidecar pattern of a container cluster. Background Technology

[0002] GPU sharing scheduling is a technology that can increase GPU availability because it allows multiple containers to share and utilize the GPU. This is because, when GPU sharing scheduling is not available, the GPU unit is always exclusively allocated to a single container.

[0003] However, it is not easy to utilize such technologies for improving GPU availability in container-based cluster environments. For example, in the case of prior art 2 (GPU-Operator About the NVIDIA GPU Operator, NVIDIA GPU Operator 23.9.0 documentation), multiple Pods can utilize the same GPU, but it does not provide GPU resource isolation capabilities. Therefore, it is unsuitable for cluster operations targeting commercial services in various fields.

[0004] In addition, prior art 3 (MPS: Multi-Process Service: GPU Deployment and Management Documentation (nvidia.com)) discloses MPS-based resource isolation technology, but prior art 3 supports process-level computing resource and memory resource isolation. This is not a container-level isolation technology, and therefore, when multiple GPU-utilizing processes are running within a container, there is a problem in that container-level resource isolation cannot be performed.

[0005] Meanwhile, in a container-based cluster environment, GPU sharing scheduling is provided with a controller that manages resources for each node. Therefore, the controller manages GPU resources within each node comprehensively to allocate the exact amount of resources to each container in proportion to the resource requirements. However, since the controller cannot immediately allocate computational resources while managing multiple processes, this results in a delay in computation processing time. Prior art literature

[0006] (Prior Art 1) Korean Published Patent KR20230069483A (2023-05-19)

[0007] (Prior document 2) GPU-Operator about the NVIDIA GPU Operator ― NVIDIA GPU Operator 23.9.0 documentation (Prior document 3) MPS: Multi-Process Service: GPU Deployment and Management Documentation (nvidia.com) The problem to be solved

[0008] For the reasons above, the present invention provides an apparatus and method capable of efficiently managing GPU resources without using a centralized GPU resource controller for each node.

[0009] The present invention provides an apparatus and method for managing GPU resources through a sidecar pattern of a container cluster.

[0010] The technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0011] A GPU resource management device according to an embodiment of the present invention for achieving such an objective comprises: a scheduler that allocates GPU resources to a container at a selected node in response to a request for container GPU computing resources requested by a client and performs a task requested at the cluster level; and a Pod that is included in the node and manages a group of containers configured for a single task, wherein the Pod is configured to include a sidecar container that monitors the memory utilization rate and GPU computing resource utilization rate of the GPU of a main container and a process to which GPU computing resources generated through the scheduler have been allocated, and controls the computing resource utilization rate of the GPU container according to the monitoring results.

[0012] The present invention can be utilized in a container-based cluster.

[0013] The main container above interacts with the GPU computation resource management tool provided by the sidecar container through API hooking when executing GPU tasks.

[0014] The above sidecar container terminates the process if the process running in the current container consumes excessive memory through API hooking.

[0015] The above sidecar container kills processes for memory consumption that cannot be verified through API Hooking.

[0016] The above API Hooking performs a preliminary check of the GPU memory resource allocation API.

[0017] In the present invention, the sidecar container includes a rate limiter that schedules computational resources according to the amount of GPU computational resources.

[0018] The above rate limiter controls the number of API requests based on the GPU computing resources allocated to the pod.

[0019] The above scheduler induces the utilization of GPU computing resources by distinguishing Pod Affinity and Anti-Pod Anffinity based on the GPU task types of batch and iteration tasks.

[0020] A method for managing GPU resources according to another feature of the present invention comprises a scheduler; a Pod having a main container and a sidecar container, and is a method for scheduling sidecar-based GPU sharing in a container-based cluster environment, wherein the sidecar container is characterized by suspending a process when a process running in the current container consumes memory in excess through API Hooking, or scheduling a GPU computing resource utilization API in accordance with the requested amount of computing resources.

[0021] The above sidecar container kills processes for memory consumption that cannot be verified through API Hooking.

[0022] The above GPU computational resources control the computational resource utilization rate using a rate limiter that controls the request frequency of the GPU computational resource utilization API.

[0023] The utilization rate of the aforementioned GPU computing resource API involves the control of the request frequency of the GPU computing resource utilization API being shared among multiple processes at the container level, and the opportunity for the computations of the said processes to be scheduled is provided in an equitable manner.

[0024] The above fair method may be a round-robin method.

[0025] The scheduling of the above GPU computation resource utilization API is performed by including: a step of determining the waiting time required for a GPU computation resource utilization API request before the main container performs GPU computation; and a step of waiting until the GPU computation resource utilization API can be requested according to the determination.

[0026] The waiting time required for the above GPU computational resource utilization API request is determined by the difference between the computational resources currently in use and the computational resources allocated to the corresponding container.

[0027] If the above computational resources are insufficient compared to the resources allocated to the Pod, the request waiting time for the GPU computational resource utilization API decreases, and if the above computational resources are allocated in excess compared to the resources allocated to the Pod, the request waiting time for the GPU computational resource utilization API increases.

[0028] The scheduling of the above GPU computational resource utilization API is executed within the size of the default allocated computational resource depending on the existence of idle computational resources; if idle computational resources exist, the limit size of the computational resources that can be allocated is increased in proportion to the size of the idle computational resources, and if idle computational resources do not exist, the limit size is decreased until a certain size of idle computational resources exists.

[0029] The size of the above-mentioned default allocated computational resources is limited to the value obtained by subtracting the minimum idle computational resources from the total computational resources of the corresponding GPU. Effects of the invention

[0030] According to one aspect of the disclosed invention, since resource management is performed through a sidecar pattern of a container cluster, resource management of a container can be performed immediately regardless of resource utilization by other containers within the server, thereby enabling faster computation processing time.

[0031] According to another aspect of the present invention, memory resource control is possible even in cases where multiple processes are executed in a single container. Brief explanation of the drawing

[0032] FIG. 1 is a configuration diagram of a GPU resource management device according to an embodiment of the present invention. FIG. 2 is a configuration diagram explaining the work process according to GPU memory capacity using the GPU resource management device of the present invention. FIG. 3 is a configuration diagram illustrating a process for managing GPU computing resources using the GPU resource management device of the present invention. Specific details for implementing the invention

[0033] Throughout the specification, the same reference numerals refer to the same components. This specification does not describe all elements of the embodiments, and general content in the art to which the disclosed invention pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.

[0034] Throughout the specification, when a part is described as being 'connected' to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0035] Furthermore, when it is stated that a part 'includes' a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0036] Throughout the specification, when it is stated that a component is located 'on' another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.

[0037] Terms such as "first," "second," etc., are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0038] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0039] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0041] The operating principle and embodiments of the disclosed invention will be described below with reference to the attached drawings.

[0042] FIG. 1 is a configuration diagram of a GPU resource management device according to an embodiment of the present invention.

[0043] Referring to FIG. 1, a GPU resource management device (100) includes a scheduler (110, scheduler) and a node (120, Node).

[0044] The scheduler (110) can perform scheduling on one selected node based on a request from a client. Specifically, it allocates GPU resources to a container in accordance with the container GPU resource request made by the client and performs the task of requesting this at the cluster level.

[0045] A node (120) is a physical machine on which a pod (130) runs. A node (120) may contain a pod (130). A pod (130) may be a unit for managing a group of containers configured for a single task. As previously mentioned, the embodiment will be described as a unit for managing a group of containers required for a specific task.

[0046] According to an embodiment, the pod (130) may include a sidecar container (150) and a main container (140).

[0047] The sidecar container (150) runs together with the created container and, through API Hooking, can stop the process if the process running in the current container consumes too much memory (GPU memory operation) or schedule computation resources according to the requested amount of computation resources (GPU computation resource utilization operation). That is, the sidecar container (150) monitors the GPU memory utilization and GPU computation resource utilization of the running process and controls the resource utilization of the GPU container according to the monitoring results. Here, the API Hooking performs a preliminary check of the GPU memory resource allocation API. And the sidecar pattern of this sidecar container (150) can be defined as running together with a container that plays a role in assisting the main controller in the pod (130).

[0048] Additionally, the sidecar container (150) may include a rate limiter (see FIG. 3, 152) for controlling operations related to GPU computation resource utilization. The rate limiter (152) performs the role of controlling the frequency of requests for the GPU computation resource utilization API.

[0049] The main container (140) refers to a container to which GPU resources created through the scheduler (110) are allocated. When executing a GPU task, the main container (140) can be linked with the GPU resource management tool provided by the sidecar container (150) through the API Hooking mentioned above.

[0050] As such, this embodiment has the advantage of enabling immediate resource management of a container regardless of resource utilization by other containers within the server by performing sidecar-based resource management.

[0052] FIG. 2 is a configuration diagram explaining the work process according to GPU memory capacity using the GPU resource management device of the present invention.

[0053] Referring to FIG. 2, according to one embodiment, a sidecar container (150) monitors the GPU memory utilization of processes. That is, the sidecar container (150) compares the memory consumption of the container with the container memory limit. Here, the container memory consumption may be information measured and recorded in NVML.

[0054] And if the above container memory consumption exceeds the container memory limit, the sidecar container (150) instructs the main container (140) to terminate the process.

[0055] Meanwhile, before allocating GPU memory, the main container (140) may further perform a process of checking whether the sum of the memory size collected through the sidecar container (160) and the currently allocated memory size exceeds the memory size allocated to the container. If the memory size exceeds the limit according to this process, an error is generated.

[0056] According to another embodiment, the main container (140) performs a process management task even when there is memory consumption that cannot be verified through API Hooking during GPU task execution. Specifically, if memory consumption occurs, such as memory allocation during the Cuda driver call process, it performs an OOM (Out of Memory) Killer task to kill the process to prevent excessive memory consumption.

[0057] In this way, the sidecar container (150) can determine the memory overflow situation and terminate or stop the process. In particular, memory resource control is possible even under conditions where multiple processes are executed in a single container. Also, since resources are not controlled through a node-specific controller, container resource control is possible regardless of the container allocated within the node and the status of resource utilization. Additionally, context memory can be controlled through container management.

[0059] FIG. 3 is a configuration diagram illustrating a process for managing GPU computing resources using the GPU resource management device of the present invention.

[0060] In the case of GPU computing resources, resource utilization is controlled through a rate limiter (152) that controls the request waiting time for the GPU computing resource utilization API. This method of resource utilization has the advantage of enabling scheduling of the GPU computing resource utilization API regardless of resource utilization by multiple containers within the node, and allowing resource control without a centralized node controller. In other words, while precise numerical resource control is difficult, it is possible because the GPU computing resource utilization between each container is not taken into account. Conventionally, when multiple processes exist within a container, GPU computing resources could not be utilized among them. Consequently, when a process executes a computation task, it is placed in a race condition, leading to a problem of unbalanced resource allocation between processes.

[0061] The scheduling of the GPU computation resource utilization API first checks the request waiting time of the GPU computation resource utilization API before the sidecar container (150) performs the GPU computation. As a result of the check, the GPU computation resource utilization API will be executed after the request waiting time of the GPU computation resource utilization API.

[0062] In this case, the request wait time for GPU computation resource utilization APIs is determined by the difference between the computation resources currently in use and those allocated to the container. If computation resources are underutilized, the request wait time for GPU computation resource utilization APIs decreases. Conversely, if computation resources are overutilized, the request wait time for GPU computation resource utilization APIs increases.

[0063] As such, this embodiment can manage the utilization rate of computational resources among multiple processes executed within a container through a sidecar container (150). In this case, managing the utilization rate of computational resources means configuring the tokens checked before performing computations to be shared among multiple processes at the container level rather than at the process level. Furthermore, even if the tokens are shared, in order to ensure that the computations of the processes are executed in a balanced manner, the opportunity for each process's computation to be scheduled is provided in a fair manner. An example of fair scheduling is Round Robin scheduling, which means giving the opportunity to perform computations by circulating through the currently executing GPU-utilizing processes. If a process does not request to perform computations, the opportunity for that process is skipped and passed to the next process.

[0064] Furthermore, the sidecar-based GPU computational resource management method can provide a way to flexibly utilize idle computational resources depending on their availability. Specifically, if idle computational resources are present, the limit on the amount of resources that can be allocated can be increased in proportion to their size. Conversely, if there are no idle resources, the limit is reduced until a certain amount of idle resources remains. In this case, the range of increase or decrease in the computational resource limit will be restricted to the size of the default allocated computational resources (total computational resources of the GPU minus the minimum idle computational resources).

[0066] The present invention also provides a method for efficiently utilizing computational resources according to the type of GPU task.

[0067] Specifically, the scheduler (110) can induce effective utilization of GPU resources by defining Pod Affinity and Pod Anti-Affinity according to the type of GPU task. Here, GPU tasks can be classified into interaction tasks and batch tasks. Interaction tasks refer to utilizing GPU resources only when there is a user request, such as in an IDE (Integrated Development Environment)-based development process or inference task, while batch tasks refer to tasks that continuously utilize computational resources during the time the process is executed through a data set stored in storage.

[0068] This embodiment can induce the placement of GPU tasks on different nodes if the types of GPU tasks are the same, and the placement of GPU tasks on the same node if the types of GPU tasks are different. For example, since Pod affinity / non-affinity refers to the preference / non-preference for existing deployed Pods from the perspective of the scheduled Pod, if the affinity of the scheduled Pod and the existing deployed Pod is specified, the scheduled Pod is scheduled to the same node as that Pod; conversely, if non-affinity is specified, the Pod is prevented from being scheduled to a node where a non-preferred Pod is running.

[0069] In this way, batch jobs that continuously utilize computational resources can be encouraged to flexibly utilize additional computational resources when interaction tasks are idle, in addition to the basic computational resources allocated to the container. Consequently, batch jobs can be terminated more quickly, and since interaction tasks do not occupy resources when idle, it is possible to provide these resources at a lower cost.

[0071] As such, it can be seen that the present invention provides a method for performing sidecar-based GPU sharing scheduling in a container-based cluster environment.

[0073] The foregoing description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0074] 100: GPU resource management device of the present invention 110: Scheduler 120: Node 130: Pod 140: Main container 150: Sidecar container 152: Rate limiter

Claims

Claim 1 A GPU resource management device comprising: a scheduler that allocates GPU resources to a container on a selected node in accordance with a request for container GPU computing resources requested by a client and performs a task requested at the cluster level; and a Pod provided on the node and managing a group of multiple containers configured for a single task, wherein the Pod is configured to include a sidecar container that monitors the memory utilization and GPU computing resource utilization of the GPU of a main container and a process to which GPU computing resources generated through the scheduler are allocated, and controls the computing resource utilization of the GPU container according to the monitoring result, and wherein the main container generates an error if the sum of the memory size allocated to the main container and the memory size collected through the sidecar container exceeds the memory size allocated to the container. Claim 2 In claim 1, the main container is a GPU resource management device that links with a GPU computation resource management tool provided by the sidecar container through API Hooking when executing a GPU task. Claim 3 In claim 1, the sidecar container is a GPU resource management device that stops a process when a process executed in the current container consumes excessive memory through API Hooking. Claim 4 In claim 1, the sidecar container is a GPU resource management device that kills processes for memory consumption that cannot be verified through API Hooking. Claim 5 In paragraph 3 or 4, the above API Hooking is a GPU resource management device that performs a prior check of a GPU memory resource allocation API. Claim 6 In claim 1, the sidecar container is a GPU resource management device comprising a rate limiter that schedules computation resources according to the amount of GPU computation resources. Claim 7 In claim 6, the rate limiter is a GPU resource management device that controls the number of API requests based on the GPU computing resources allocated to the Pod. Claim 8 In paragraph 6, the rate limiter is a GPU resource management device that controls the request frequency of a GPU computation resource utilization API. Claim 9 In claim 1, the scheduler is a GPU resource management device that induces the utilization of GPU computing resources by distinguishing Pod Affinity and Anti-Pod Anffinity according to the GPU task types of batch tasks and iteraction tasks. Claim 10 A method for managing GPU resources, comprising a scheduler; a Pod having a main container and a sidecar container, and scheduling sidecar-based GPU sharing in a container-based cluster environment, wherein the main container generates an error if the sum of the memory size allocated to the main container and the memory size collected through the sidecar container exceeds the memory size allocated to the container, and the sidecar container terminates the process if the process executed in the current container consumes memory in excess through API Hooking, or schedules a GPU computing resource utilization API according to the requested amount of computing resources. Claim 11 In claim 10, the above-mentioned sidecar container is a method for managing GPU resources that kills processes for memory consumption that cannot be verified through API Hooking. Claim 12 In claim 10, the above-mentioned GPU computing resource is a method for managing GPU resources, wherein the computing resource utilization rate is controlled using a rate limiter that controls the request frequency of a GPU computing resource utilization API. Claim 13 A method for managing GPU resources according to claim 12, wherein the computational resource utilization rate is such that the control of the request frequency of the GPU computational resource utilization API is shared among a plurality of the processes at the container level, and the opportunity for the computation of the processes to be scheduled is provided in a fair manner. Claim 14 In Clause 13, the above-mentioned fair method is a round-robin method, a method for managing GPU resources. Claim 15 A method for managing GPU resources according to claim 10, wherein the scheduling of the above-mentioned GPU computation resource utilization API comprises: a step of determining a waiting time required for a request to the GPU computation resource utilization API before the main container performs GPU computation; and a step of waiting until the GPU computation resource utilization API can be requested according to the determination. Claim 16 A method for managing GPU resources according to claim 15, wherein the waiting time required for the above-mentioned GPU computational resource utilization API request is determined by the difference between the computational resources currently in use and the computational resources allocated to the container. Claim 17 A method for managing GPU resources according to claim 16, wherein if the computing resources are insufficient compared to the resources allocated to the Pod, the request waiting time of the GPU computing resource utilization API is reduced, and if the computing resources are allocated more than the resources allocated to the Pod, the request waiting time of the GPU computing resource utilization API is increased. Claim 18 A method for managing GPU resources according to claim 10, wherein the scheduling of the above-mentioned GPU computational resource utilization API is executed within the size of the computational resource allocated by default depending on whether there is an idle computational resource, and if there is an idle computational resource, the limit size of the computational resource that can be allocated is increased in proportion to the size of the idle computational resource, and if there is no idle computational resource, the limit size is decreased until there is an idle computational resource of a certain size. Claim 19 A method for managing GPU resources according to claim 18, wherein the size of the basic allocated computational resource is limited to the value obtained by subtracting the minimum idle computational resource from the total computational resource of the GPU.

Images