A method for sharing GPU space based on Kubernetes cluster

Abstract

The invention discloses a method for sharing GPU space based on Kubernetes cluster, relates to the field of computer technology, and aims at the problem of low GPU resource utilization in current data processing clusters. A scoring rule combining GPU video memory with active threads is proposed, and in order to effectively schedule GPU resources, a Scheduler Extender and a Device Plugin are added to the cluster, further changing the static scheduling scheme in the previous scheduling strategy. By using an improved ant colony algorithm, the cluster resource utilization can be effectively improved, the enterprise use cost can be reduced, and the benefit can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for sharing GPU resources based on a Kubernetes cluster. Background Art

[0002] With the development and progress of network technology, the amount of data in the network has experienced explosive growth. There is often huge value hidden behind the data. How to process the data quickly is a hot issue at present. The solutions that have emerged are roughly the following two aspects:

[0003] One is to use GPU. Previously, people constructed the CPU to look like the human brain, which is suitable for logical processing and serial computing, as well as multi-task parallel processing. Therefore, people are used to using CPU to calculate data. However, in the GPU architecture, the number of parallel computing units (SMs) is much larger than that of CPU. Later, in the process of processing data, engineers gradually noticed the powerful ability of GPU in processing parallel data. In 2006, NVIDIA released CUDA, which is a general parallel computing platform and programming model based on NVIDIA's CPUs. CUDA programming can use the parallel computing engine of GPUs to solve more complex computing problems more efficiently.

[0004] One is the cloud computing cluster. In recent years, the popularity and development of cloud computing systems and services have grown exponentially. As an emerging, hybrid computing paradigm, cloud computing services are becoming the main computing power for commercial, personal and mobile computing applications, usually deployed on distributed virtualized infrastructure. With the development of virtualization technology and container technology, the use of cloud services has become simpler and more convenient. Because of the lighter weight of containers, the original virtual machine technology has gradually been replaced. There are many types of container technologies, but currently Docker technology is almost the absolute mainstream of containers. Kubernetes is a container orchestration tool developed by Google. Its predecessor is the Borg system that Google has used for many years. Therefore, when Kubernetes was first made public, its architecture and technology were relatively mature, and it attracted widespread attention in the cloud computing industry, and quickly became the mainstream of the orchestration tool market. By using a distributed cloud computing cluster platform based on Kubernetes, the data processing speed is accelerated. Compared with the use of personal computers or traditional clusters, cloud clusters have greatly improved the deployment and execution efficiency of tasks.

[0005] Therefore, combining the two is a common choice in today's society, but there is a big problem facing users or enterprises. In most clusters now, a Docker or container completely occupies a GPU card, and with the iteration and update of GPU versions, GPU resources are becoming more and more expensive, which not only causes resource waste, but also sharply increases the cost of use. How to achieve GPU sharing has become a hot topic.

[0006] The patent publication number is CN111506419A, and the patent name is a GPU resource sharing method and device. It proposes a method, including comparing the GPU physical resource requirements of the virtual machine with the unallocated resources in the resource pool; if the unallocated resources in the resource pool do not meet the GPU physical resource requirements of the virtual machine, then adjust the GPU physical resources allocated to the virtual machine according to the utilization rate of the GPU physical resources allocated to the virtual machine, and allocate GPU physical resources to the virtual machine according to the unallocated resources in the adjusted resource pool; otherwise, allocate GPU physical resources to the virtual machine according to the GPU physical resource requirements of the virtual machine. The invention does not specify the types of GPU physical resources, and the method may not be applicable in practice. In addition, the method still uses virtual machine technology, which will occupy a lot of resources at the system level when used.

[0007] The patent publication number is CN111475303A, and the name of the patent is a GPU sharing scheduling, single machine multi-card method, system and device. A GPU sharing scheduling method is proposed, which is applied to the scheduler of the central control host, including: using the resource occupation mark and update mark of the pod to query the GPU information in the environment variables of the unupdated pod in each controlled host; the unupdated pod is a pod that has been running but has not updated the GPU information; updating the GPU information to the annotation of the unupdated pod, and adding the update mark to the unupdated pod; screening out the schedulable controlled host without unmarked pod from multiple controlled hosts; the unmarked pod is a pod without the update mark; using the status information of the GPU in the schedulable controlled host, screening out the target controlled host that meets the first preset condition and the target GPU in the target controlled host that meets the second preset condition; writing the GPU information of the target GPU in the annotation of the pod to be allocated; and allocating the pod to be allocated to the target controlled host. This sharing method does not actually solve the GPU sharing problem for each card, and the problem of single card resource waste still exists.

[0008] Most current GPU sharing methods have more or less problems, and some methods are not even suitable for Kubernetes cloud clusters. A container alone occupies a GPU card, resulting in a large waste of resources. Many users are deterred by the high usage fees. Summary of the invention

[0009] The purpose of the present invention is to provide a method based on Kubernetes cluster GPU space sharing to improve the GPU utilization rate in the cluster and reduce the use cost.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A method for sharing GPU space in a Kubernetes cluster includes the following steps:

[0012] 1) Use a preset number of nodes to build a Kubernetes cluster, where the nodes include a master node consisting of a server and a worker node for executing tasks assigned by the Master. The components of the master node include a data bus API Server, a controller, and a scheduler; set a scheduler plug-in Scheduler Extender on the scheduler, and set a device plug-in Device Plugin on the worker node. The Kubernetes cluster caches the GPU model, GPU memory resources, and multi-process service MPS information of the worker node reported by the Device Plugin into the Scheduler Extender;

[0013] 2) When the user submits Pod information through the client, Pod is the smallest scheduling unit of the cluster. The cluster receives the Pod information and determines whether the format of the Pod information content conforms to the cluster characteristics. If not, it will feedback to the user;

[0014] 3) If the format of the Pod information content meets the cluster characteristics, the API Server stores the Pod information in the distributed storage database Etcd and establishes a query relationship for the Pod information on the client;

[0015] 4) The scheduler filters out unqualified worker nodes based on the scheduling resources required by the Pod, and then sends the qualified worker node information, GPU memory allocated to the Pod, and active thread ratio information to the SchedulerExtender;

[0016] 5) SchedulerExtender scores, and then adds the score and the information of the Pod to be scheduled to the ant colony algorithm for calculation. If a solution is obtained, the solution is sent to the scheduler. Otherwise, it plans to open a closed working node and include the working node information in the working node set of the Pod to be scheduled, and iterates the calculation again through this step;

[0017] 6) If the solution obtained contains a work node that is planned to be opened, open the work node; otherwise, bind the Pod to be executed to the work node through Scheduler, and write the binding information to Etcd;

[0018] 7) Verify on the working node whether the bound Pod can run on the working node. If it runs successfully, re-score it, remove the failed working node from the set of deployable working nodes, and allocate it again;

[0019] 8) Repeat the above steps until all Pods to be scheduled are completed;

[0020] 9) The scheduler sends the Pod and worker node binding information, worker node status information, and Pod usage scheduling information to Etcd. Etcd obtains the worker node switch status information and Pod operation status information in the cluster according to a fixed time period and updates the corresponding stored information.

[0021] Furthermore, each working node runs a Kubelet service process, listens to the port, receives and executes instructions from the master node, and manages Pod information and containers in the Pod; each Kubelet service process registers the working node's own information on the API Server, regularly reports resource usage to the master node, and monitors the resources of the working node and the containers in the Pod.

[0022] Furthermore, Device Plugin connects to the Device Plugin Manager in Kubelet through the remote procedure call system GRPC as a client to obtain the GPU model and GPU memory resources of the working node and report and monitor them.

[0023] Furthermore, the user submits Pod information through the command line tool Kubectl installed on the client.

[0024] Furthermore, Scheduler Extender performs scoring according to a scoring rule, which is:

[0025]

[0026]

[0027] G=α*S gpum +β*S mps ;

[0028] Among them, T gpum Indicates the GPU memory of the current working node, T mps Indicates the active thread ratio of the current working node; Ugpum Indicates the GPU memory consumed by the tasks deployed on the current working node, U mps Indicates the ratio of active threads consumed by the tasks deployed on the current working node; R gpum Indicates the GPU memory currently requested by the task to be scheduled, R mps Indicates the ratio of active threads to the current task requests to be scheduled; S gpum Indicates the idle GPU memory of the working node, S mps It represents the ratio of active threads to idle threads of working nodes; α and β represent weights; G represents the resource priority of Node.

[0029] Furthermore, the ant colony algorithm is composed of the following three formulas:

[0030]

[0031]

[0032]

[0033] Where i represents the starting point of the current ant, j represents the destination that the current ant can reach, Λ represents the set of destinations that the ant can reach, and η ij represents the heuristic function, τ ij represents the pheromone concentration on the path from starting point i to end point j, α represents the pheromone weight factor, β represents the heuristic function weight factor, n represents the number of iterations, m represents the total number of ants, represents the size of pheromone secreted by ant k from starting point i to end point j, ρ represents the volatility factor of pheromone selected from starting point i to end point j, R represents the evaluation coefficient of pheromone risk, L k represents the distance of ant k from starting point i to end point j in the current iteration, It represents the probability that ant k chooses to crawl from starting point i to end point j.

[0034] Furthermore, the heuristic function formula is as follows:

[0035]

[0036] Furthermore, the objective function of the optimal ant path in the ant colony algorithm is used as the volatilization factor ρ of the pheromone to calculate the objective function value; the smaller the objective function value, the better the scheduling plan found by the ants.

[0037] Furthermore, the objective function is formulated as follows:

[0038]

[0039] Among them, f(q) represents the objective function, q represents the number of working nodes used in the current scheduling, Indicates the total GPU memory of the jth worker node, They represent the total active thread ratio of the jth worker node respectively; Indicates the GPU memory consumed by the tasks deployed on the jth worker node. Indicates the ratio of active threads consumed by the tasks deployed on the jth worker node, Indicates the GPU memory requested by the i-th task to be scheduled, It represents the ratio of active threads of the i-th task to be scheduled, ρ1 and ρ2 represent weights, m represents the number of working nodes used in the current scheduling, x represents x Pods to be scheduled, y represents y working nodes; c represents the number of tasks on the current working node.

[0040] Furthermore, the constraints of the optimal ant path in the ant colony algorithm are:

[0041]

[0042]

[0043] q≤y;

[0044] The above constraints are respectively allocated to the sum of the ratio of GPU memory to active threads used by all tasks on the jth worker node, which is less than or equal to the total ratio of GPU memory to active threads of the current worker node.

[0045] Furthermore, the initial pheromone concentration value τ of the ant colony algorithm is ij (0) is set to the percentage of idle resources of the cluster working nodes. The formula is as follows:

[0046]

[0047] Furthermore, during the solution process of the ant colony algorithm, a roulette mechanism is used to select a path; after a path selection is performed for a single ant, the local pheromone is updated through the global pheromone formula; and after all ants have completed the path selection, the global pheromone is updated through the global pheromone formula.

[0048] Furthermore, local pheromone: τ ij (t+1)=ρΔτ ij (t)+G ij τ ij (t);

[0049] Global pheromone: τ ij (t+1)=ρΔτ ij (t)+(1-f(q))τ ij (t);

[0050] Where ρ is the volatility factor, Δτ ij (t) represents the increment of pheromone left by the ant colony from the starting point i to the end point j in each iteration. The pheromone on each path in the initial stage is zero; G represents the total amount of pheromone secreted by k ants in the tth iteration task from assigning task i to worker node j; ij Indicates the resource priority of the working node.

[0051] The advantages of the present invention are as follows: In view of the problem of low GPU resource utilization in the current data processing cluster, the present invention proposes a scoring rule combining GPU video memory and active threads, and in order to effectively schedule GPU resources, adds Scheduler Extender and Device Plugin to the cluster, further changes the static scheduling scheme in the previous scheduling strategy, and uses an improved ant colony algorithm to effectively improve cluster resource utilization, reduce enterprise usage costs, and improve benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a simplified cluster architecture diagram.

[0053] Figure 2 This is an improved Kubernetes cluster architecture diagram proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the above features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.

[0055] The embodiments of the present invention are Figure 1 Based on the Kubernetes cloud cluster shown in the figure, the cluster architecture is improved. The improved architecture is as follows Figure 2 As shown, first briefly introduce the cluster architecture:

[0056] Master: It is the core node of the entire cluster. All execution commands for the Kubernetes cluster are executed by it. It is responsible for the scheduling and management of the entire cluster. It is generally an independent server in the cluster. Master includes many components, mainly including API Server, Controller Manager and Scheduler. API Server is the data bus and data center of the entire system. It provides the addition, deletion, modification and query of various resource objects in the cluster system, the REST API interface for cluster management (including authentication and authorization, data verification and cluster status change) and data interaction and communication between other modules. Other modules query or modify data through API Server, and only API Server directly operates Etcd. Controller is the internal management and control center of the cluster. It is responsible for obtaining the relevant parameters such as development dependency libraries and system configuration required by users and generating tasks for these parameters and sending them to the scheduler. The scheduler accepts the tasks sent by the controller, monitors the legitimacy of the Pod, finds a suitable working node to run the Pod according to the scheduling strategy, and sends the binding information to Etcd through API Server for storage.

[0057] Worker Node: In a Kubernetes cluster, except for the master node, all other nodes are called worker nodes. Each Node node is assigned a reasonable workload by the master node according to the scheduling rules and is the executor of the task. A Kubelet service process runs on each Node node, which listens to the port, receives and executes the instructions sent by the master, and manages the Pod and the containers in the Pod. Each Kubelet process registers the Node node's own information on the API Server, regularly reports resource usage to the master node, and monitors the resources of the Node node and container through the software or module cAdvisor that monitors resource utilization.

[0058] Command line tool Kubectl: Kubectl is usually installed on the client. It can manage the cluster itself and install and deploy containerized applications on the cluster. As a command line tool for Kubernetes, Kubectl's main responsibility is to operate on the objects of resources in the cluster, including creating, deleting, and viewing resource objects, so as to manage the Kubernetes cluster.

[0059] Distributed storage database (Etcd): Etcd is a very important component in the Kubernetes cluster. It is used to save all network configurations and status information of all objects in the cluster, that is, it saves the status of the entire cluster.

[0060] Controller Manager: responsible for maintaining the status of the cluster, such as fault detection, automatic expansion, rolling updates, etc.

[0061] Scheduler: Responsible for resource scheduling, scheduling Pods to corresponding machines according to predetermined scheduling strategies.

[0062] Service process Kubelet: runs on the Node node, is responsible for maintaining the life cycle of the container, is also responsible for the management of Volume (CVI) and network (CNI), and provides the current status information of the Node node to the Master.

[0063] The Pod in the present invention is the smallest scheduling unit of the Kubernetes cluster. Pod can be seen as the working form of one or a group of containers in the cluster, and the container is the running state of Docker.

[0064] In recent years, the adoption of a unified GPU architecture and the use of high-level language programming (such as C++ programming language, etc.) development platforms have pushed up GPU general computing and enabled it to enter a relatively rapid development. In current cloud clusters, most of the tasks submitted by users or enterprises are also data parallel computing tasks, so GPUs are important resources in clusters. Among GPU resources, GPU video memory and streaming multiprocessors (SM) are the two core resources that everyone is most concerned about now. Video memory is an important space for storing temporary data, whether in data calculations or deep learning training. However, the Kubernetes cluster does not support the scheduling of video memory resources. Similarly, for SM, it can be regarded as the heart of the GPU, an important part of the GPU architecture, and an important hardware for CUDA kernel function calculations, but it cannot be used as an important reference resource to assist in sharing. The present invention is aimed at both and proposes a method for sharing GPU resources.

[0065] The Multi-Process Service (MPS) is another binary-compatible implementation of the CUDA application programming interface (API). The MPS runtime architecture is designed to transparently enable multi-process CUDA applications (usually MPI jobs) to take advantage of the Hyper-Q feature on the latest NVIDIA (Kepler-based) GPUs. Hyper-Q allows CUDA kernels to be processed concurrently on the same GPU. MPS is a binary-compatible client-server runtime implementation of the CUDA API, providing a way for multi-process CUDA applications to share NVIDIA GPUs. And it is possible to set different active thread ratios within the MPS job, so that different jobs can be distinguished in GPU computing. In simple terms, MPS can schedule the CUDA context of non-operating as a whole, conceptually realizing the allocation of SMs. The reason for setting different active thread ratios is that the multi-thread service mechanism allows users to adjust the calculation speed by setting the available active thread ratio variable of the program. If all threads in a streaming multiprocessor are working at the same time, the efficiency or occupancy rate of the streaming multiprocessor is 100%. In this case, the processing speed of the program is the fastest and the efficiency is the highest. In practice, due to resource limitations, not all threads work at the same time, and the threads that are running are active threads. The multi-threaded service can set the active thread ratio and allocate it to different application processes. Although the multi-threaded service is an independent system that implements multi-process sharing of the GPU on the host, the container running on the Node node can be regarded as a process on the host. Therefore, the active thread ratio can be set when the container is started, and MPS can be regarded as a fine-grained resource that can be divided.

[0066] In order to improve the utilization of cluster resources, the present invention designs a node scoring rule that combines the above two factors to evaluate whether a Node node is suitable for scheduling Pods. GPU memory is based on 1G as a unit, and MPS as a resource refers to the active thread ratio as a divisible fine-grained resource, with a maximum of 100 and a minimum of 0, with 10 parts as a unit, and the active thread ratio of all tasks on a Node node is less than or equal to 100. The scoring rules are:

[0067]

[0068]

[0069] G=α*S gpum +β*S mps ;

[0070] T gpum 、T mpsRespectively represent the GPU memory and active thread ratio of the current Node; U gpum , U mps Respectively represent the GPU memory and active thread ratio consumed by the tasks deployed on the current Node; R gpum , R mps Respectively represent the GPU memory and active thread ratio of the current task to be scheduled; S gpum , S mps They represent the ratio of idle GPU memory and active threads of Node respectively; α and β represent weights, where α is 0.6 and β is 0.4; G represents the resource priority of Node. The larger the value, the higher the score of Node.

[0071] Ant colony algorithm is a swarm intelligence algorithm. It is a group of non-intelligent or slightly intelligent individuals who show intelligent behavior through mutual cooperation, thus providing a new possibility for solving complex problems. Ant colony algorithm was first proposed by Italian scholars Colorni A., Dorigo M., etc. in 1991. After more than 20 years of development, ant colony algorithm has made great progress in theory and application research. Ant colony algorithm is a bionic algorithm inspired by the foraging behavior of ants in nature. In nature, during the foraging process of ants, ant colonies can always find the optimal path from the ant nest and the food source. The application of artificial ants, through information exchange and cooperation between individuals, can not only solve the traveling salesman problem, but also has good results in solving combinatorial optimization problems, such as scheduling, network routing, etc.

[0072] The conventional ant colony algorithm model is constructed by the following three formulas:

[0073]

[0074]

[0075]

[0076] i represents the starting point of the current ant; j represents the destination that the current ant can reach; Λ represents the set of destinations that the ant can reach; η ij represents the heuristic function; τ ij represents the pheromone concentration on the road from starting point i to end point j; α represents the pheromone weight factor; β represents the heuristic function weight factor; n represents the number of iterations; m represents the total number of ants; represents the size of pheromone secreted by ant k from starting point i to end point j; ρ represents the volatility factor of pheromone in route selection from starting point to end point; R is an evaluation coefficient of pheromone risk; L k represents the distance from the starting point to the end point of ant k in this iteration; Indicates the probability that ant k chooses to crawl from starting point i to end point j. The β heuristic function weight factor is used to indicate the influence of pheromones and heuristic functions on the ant's foraging process, which directly affects whether the optimal solution can be found in the solution space. If the β value is larger, the current local optimal solution will be selected; the ρ volatility factor is mainly used to maintain a certain constraint between the heuristic function and the pheromone, so that this constraint reaches a dynamic constraint balance.

[0077] The objective function of the optimal ant path in the ant colony algorithm is:

[0078]

[0079] In the formula, Respectively represent the total GPU memory and total active thread ratio of the j-th Node; They represent the GPU memory and active thread ratio consumed by the tasks deployed on the j-th Node node; Respectively represent the ratio of GPU memory and active threads requested by the i-th task to be scheduled; ρ1 and ρ2 represent weights; m represents the number of Node nodes used in this scheduling; x represents x Pods to be scheduled, y represents y Node nodes; c represents the number of tasks on the current Node node. f(q) represents the objective function of the optimal ant path in the ant colony algorithm. It is processed accordingly and added to the pheromone concentration update formula (i.e., the second formula of the above three formulas) as the pheromone volatilization factor ρ. The smaller the objective function value f(q), the better the scheduling scheme sought by the ant, and the greater its pheromone concentration, indicating that the allocation scheme is better. q represents the number of Node nodes used in the current scheduling.

[0080] Constraints for the optimal ant path in the ant colony algorithm:

[0081]

[0082]

[0083] q≤y;

[0084] The above constraints indicate that the sum of GPU memory and active thread ratio used by all tasks allocated on the jth Node node should be less than or equal to the total GPU memory and active thread ratio of the current Node node.

[0085] The initial pheromone concentration of the ant colony algorithm is randomly set and assigned according to the cluster information environment. Therefore, the percentage of idle resources of the cluster Node is used as the pheromone concentration value in the initialization algorithm model:

[0086]

[0087] Heuristic function:

[0088]

[0089] When the scheduling strategy is applied, the heuristic function and pheromone are initialized first. At the same time, the relevant weight factors are set according to the results obtained in the previous experiments. In the solution process, the path selection method mainly relies on the roulette mechanism, and the pheromone generated by the current selected path affects the next round of iterations together. After selecting a Node node for a single ant, the global pheromone formula is used to update the local pheromone, and after all ants have completed the path selection this time, the global pheromone formula is used to update the global pheromone.

[0090] Topical pheromones:

[0091] τ ij (t+1)=ρΔτ ij (t)+G ij τ ij (t);

[0092] Global pheromones:

[0093] τ ij (t+1)=ρΔτ ij (t)+(1-f(q))τ ij (t);

[0094] ρ is the volatility factor, Δτ ij (t) represents the increment of pheromone left by the ant colony from the starting point i to the end point j in each iteration. The pheromone on each path in the initial stage is zero.

[0095] The following formula is used to calculate the increment of pheromone, It represents the total amount of pheromone secreted by k ants in t iterations of the task i assigned to Node j:

[0096]

[0097] The larger the total amount of pheromone, the better the scheduling plan. ij It represents the resource priority of the Node node, and is added as an influencing factor to the local pheromone concentration update formula. The larger the value, the greater the possibility of deployment to the Node node. f(q) represents the objective function value of the optimal ant in the ant colony solution. It is processed accordingly and added to the pheromone concentration update formula as an influencing factor. The smaller the value, the better the scheduling plan sought by the ant, and the greater the pheromone concentration, the easier it is to find the optimal allocation plan.

[0098] As an open source container management tool, Kubernetes can use its ExtendedResource rules for the two resources involved in the present invention, GPU video memory and MPS, which can conveniently declare a resource as a cluster-recognizable resource. In order to achieve better resource scheduling, the present invention designs Scheduler Extender (scheduler plug-in) and Device Plugin (device plug-in) between the client and the Kubernetes cluster according to the Scheduler Extender mechanism and the Device Plugin mechanism. SchedulerExtender uses the extensibility of the Scheduler component and is responsible for scheduling the Pod that uses the Extender Resource resource object in the container. Since the extension mechanism of Scheduler Extender is implemented through HTTP, in order not to affect the performance of the default scheduler of the cluster, scheduling is provided for Pods that only need to use extended resources through a multi-scheduler approach, and this approach is portable. Scheduler Extender is responsible for determining whether a single GPU device on a Node can provide sufficient GPU memory when the global scheduler performs filtering and binding, and records the GPU allocation results in the Pod Spec annotation for subsequent filtering during binding. By default, the Scheduler initiates an http request to the Scheduler Extender, and the Cheduler Extender makes decisions based on its own resource data and the intermediate scheduling results passed by the Scheduler.

[0099] The resources and status of each Node in the Kubernetes cluster are summarized by Kubelet, which sends this information to the Master. When creating a Pod, the request is sent to the Scheduler. The Scheduler selects an optimal Node based on the Node status, and finally the Kubelet of the optimal Node creates the Pod. In order to manage and allocate Node Extender Resource resources, it is necessary to create a specified program, namely the device plug-in, between the Node Kubelet and the GPU. These Device Plugins connect to the Device Plugin Manager in Kubelet as clients through the remote procedure call system GRPC. DevicePlugin can report and monitor the GPU memory resources of the Node, as well as assist in the scheduling of Pods. For the monitoring of MPS resources, it is necessary to set up data structures in Scheduler Extender for operation. When the cluster is initialized, the total MPS of all Node nodes in the cluster is set to 100. When the cluster is running, the total MPS of each Node node is updated based on the active thread ratio information in the Pod that is about to be deployed to the Node node and the Pod that is about to end running on the Node node. For the scheduling of MPS resources, it is necessary to judge whether the video memory is suitable for scheduling the corresponding Pod at the same time as the Scheduler Extender determines whether it is suitable for scheduling the corresponding Pod.

[0100] The present invention proposes a method for sharing GPU space based on Kubernetes cluster, comprising the following steps:

[0101] Step 1: Initialization. Use an appropriate number of nodes to build a cluster, and cache the graphics card information (including GPU model, GPU memory resources) and MPS information of the Node nodes into the Scheduler Extender.

[0102] Step 2: Submit Pod. Users use YAML files to submit Pod information to the cluster through Kubectl. When submitting, the cluster will determine whether the format of the submitted content conforms to the cluster characteristics, such as Pod version information. If not, feedback will be given to the user; if it is, proceed to the next step.

[0103] Step 3: After receiving the information, the API Server stores it in Etcd, and the client can query the Pod information using Kubectl. It is then ready to schedule the Pods on the Nodes.

[0104] Step 4: The default scheduler filters out the Node nodes that do not meet the conditions according to other scheduling resources required by the Pod, and sends the qualified Node node information together with the GPU video memory and active thread ratio information allocated to the Pod to the Scheduler Extender.

[0105] Step 5: Node scoring. Scheduler Extender scores according to the above custom scoring rules and stores the scoring results in the scheduler.

[0106] Step 6: Use the scheduler to perform pre-scheduling. The Scheduler Extender adds the scoring and the information of the Pod to be scheduled to the ant colony algorithm for calculation. If no solution is obtained, a Node node that is in a closed state is planned to be opened, and the information of this Node node is included in the Node node set of the Pod to be scheduled, and the fifth and sixth steps are re-entered for iterative calculation. If a solution is obtained, proceed to the next step.

[0107] Step 7: Scheduler Extender sends the calculated solution back to the default scheduler Scheduler.

[0108] Step 8: If the solution obtained, that is, the scheduling plan, contains a Node node that is planned to be turned on, then turn on the Node node; otherwise, proceed to the next step.

[0109] Step 9: Execute according to the pre-set allocation plan, bind the Pod to be executed and the Node node through the scheduler. At this time, the binding method in the cluster is to update the Pod and Node node information cached in the cluster, write the binding information into Etcd, and finally verify on the Node node whether the bound Pod can run on the Node node.

[0110] Step 10: If the Pod does not run successfully on the Node, it will re-enter step 5 and execute in order, remove the failed Node from the deployable Node set, and allocate it again.

[0111] Step 11: Repeat the above steps until the list of tasks to be scheduled is empty.

[0112] Step 12: Then the scheduler sends the Pod and Node binding information, Node status information (that is, the status information of whether the Node is turned on or off), and Pod usage scheduling information to Etcd.

[0113] Step 13: The information cache module obtains the Node switch status information and Pod operation status information (i.e., which Pods are running and have finished running) in the cluster at a fixed time period, and updates the corresponding stored information.

[0114] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.

Claims

1. A method for sharing GPU space based on Kubernetes cluster, characterized in that: The following steps are involved: 1) Use a preset number of nodes to build a Kubernetes cluster, where the nodes include a master node consisting of a server and a worker node for executing tasks assigned by the Master. The components of the master node include a data bus API Server, a controller, and a scheduler; set a scheduler plug-in Scheduler Extender on the scheduler, and set a device plug-in Device Plugin on the worker node. The Kubernetes cluster caches the GPU model, GPU memory resources, and multi-process service MPS information of the worker node reported by the Device Plugin into the Scheduler Extender; 2) When the user submits Pod information through the client, Pod is the smallest scheduling unit of the cluster. The cluster receives the Pod information and determines whether the format of the Pod information content conforms to the cluster characteristics. If not, it will feedback to the user; 3) If the format of the Pod information content meets the cluster characteristics, the API Server stores the Pod information in the distributed storage database Etcd and establishes a query relationship for the Pod information on the client; 4) The scheduler filters out unqualified worker nodes based on the scheduling resources required by the Pod, and then sends the qualified worker node information, GPU memory allocated to the Pod, and active thread ratio information to the Scheduler Extender; 5) Scheduler Extender scores, and then adds the score and the information of the Pod to be scheduled to the ant colony algorithm for calculation. If a solution is obtained, the solution is sent to the scheduler. Otherwise, it plans to open a closed working node and include the working node information in the working node set of the Pod to be scheduled, and iterates the calculation again through this step; 6) If the solution obtained contains a work node that is planned to be opened, open the work node; otherwise, bind the Pod to be executed to the work node through Scheduler, and write the binding information to Etcd; 7) Verify on the working node whether the bound Pod can run on the working node. If it runs successfully, re-score it, remove the failed working node from the set of deployable working nodes, and allocate it again; 8) Repeat the above steps until all Pods to be scheduled are completed; 9) The scheduler sends the Pod and worker node binding information, worker node status information, and Pod usage scheduling information to Etcd. Etcd obtains the worker node switch status information and Pod operation status information in the cluster according to a fixed time period and updates the corresponding stored information.

2. The method according to claim 1, characterized in that Each working node runs a Kubelet service process, listens to the port, receives and executes instructions from the master node, and manages Pod information and containers in the Pod; each Kubelet service process registers the working node's own information on the API Server, regularly reports resource usage to the master node, and monitors the resources of the working node and the containers in the Pod.

3. The method according to claim 2, characterized in that As a client, Device Plugin connects to the Device Plugin Manager in Kubelet through the remote procedure call system GRPC to obtain the GPU model and GPU memory resources of the working node and report and monitor them.

4. The method according to claim 1, characterized in that Scheduler Extender scores according to the scoring rules, which are: G=α*S gpum +β*S mps ; Among them, T gpum Indicates the GPU memory of the current working node, T mps Indicates the active thread ratio of the current working node; U gpum Indicates the GPU memory consumed by the tasks deployed on the current working node, U mps Indicates the ratio of active threads consumed by the tasks deployed on the current working node; R gpum Indicates the GPU memory currently requested by the task to be scheduled, R mps Indicates the ratio of active threads to the current task requests to be scheduled; S gpum Indicates the idle GPU memory of the working node, S mps represents the ratio of active threads to idle threads of a working node; α and β represent weights; G represents the resource priority of the working node.

5. The method according to claim 4, characterized in that The ant colony algorithm consists of the following three formulas: Where i represents the starting point of the current ant, j represents the destination that the current ant can reach, Λ represents the set of destinations that the ant can reach, and η ij represents the heuristic function, τ ij represents the pheromone concentration on the path from starting point i to end point j, α represents the pheromone weight factor, β represents the heuristic function weight factor, n represents the number of iterations, m represents the total number of ants, represents the size of pheromone secreted by ant k from starting point i to end point j, ρ represents the volatility factor of pheromone selected from starting point i to end point j, R represents the evaluation coefficient of pheromone risk, L k represents the distance of ant k from starting point i to end point j in the current iteration, It represents the probability that ant k chooses to crawl from starting point i to end point j.

6. The method according to claim 5, characterized in that The objective function of the optimal ant path in the ant colony algorithm is used as the pheromone volatilization factor ρ to calculate the objective function value; the smaller the objective function value, the better the scheduling scheme found by the ants; the formula of the objective function is as follows: Among them, f(q) represents the objective function, q represents the number of working nodes used in the current scheduling, Indicates the total GPU memory of the jth worker node, They represent the total active thread ratio of the j-th worker node; Indicates the GPU memory consumed by the tasks deployed on the jth worker node. Indicates the ratio of active threads consumed by the tasks deployed on the jth worker node, Indicates the GPU memory requested by the i-th task to be scheduled, It represents the ratio of active threads of the i-th task to be scheduled, ρ1 and ρ2 represent weights, m represents the number of working nodes used in the current scheduling, x represents x Pods to be scheduled, y represents y working nodes; c represents the number of tasks on the current working node.

7. The method according to claim 6, characterized in that The heuristic function formula is as follows:

8. The method according to claim 6, characterized in that The constraints of the optimal ant path in the ant colony algorithm are: q≤y; The above constraints are respectively allocated to the sum of the ratio of GPU memory to active threads used by all tasks on the jth worker node, which is less than or equal to the total ratio of GPU memory to active threads of the current worker node.

9. The method according to claim 6, characterized in that Initialization pheromone concentration value τ of ant colony algorithm ij (0) is set to the percentage of idle resources of the cluster working nodes. The formula is as follows:

10. The method according to claim 6, characterized in that During the ant colony algorithm solution process, the roulette mechanism is used to select the path. After a single ant selects a path, the local pheromone is updated through the global pheromone formula. After all ants have completed the path selection, the global pheromone is updated through the global pheromone formula. Among them, local pheromone: τ ij (t+1)=ρΔτ ij (t)+G ij τ ij (t); Global pheromone: τ ij (t+1)=ρΔτ ij (t)+(1-f(q))τ ij (t); Where ρ is the volatility factor, Δτ ij (t) represents the increment of pheromone left by the ant colony from the starting point i to the end point j in each iteration. The pheromone on each path in the initial stage is zero; G represents the total amount of pheromone secreted by k ants in the tth iteration task from assigning task i to worker node j; ij Indicates the resource priority of the working node.

Images