Cluster Node Resource Scheduling Method and Device

Abstract

The present invention provides a method and apparatus for scheduling cluster node resources, belonging to the field of computer technology. The method for scheduling cluster node resources is used to schedule physical nodes in a cluster to create a container cluster for running user applications, and includes: obtaining parameters of a user application, where the parameters of the user application include at least one of the following: training data set, expected training duration, application type, number of containers, graphics processing unit (GPU) model in the cluster, and machine learning model; obtaining resource usage information of each physical node in the cluster; calculating a score for each physical node according to the resource usage information of each physical node and the parameters, and allocating containers to each physical node according to the score of each physical node to create a container cluster. The technical solution of the present invention can improve the utilization rate of cluster resources.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method and device for scheduling cluster node resources. Background Art

[0002] In recent years, parallel computing clusters have been increasingly applied to artificial intelligence-related fields such as speech recognition, image recognition, and natural language understanding, significantly improving the task processing capabilities in these fields. In particular, the processing capabilities of deep learning tasks in artificial intelligence-related fields have been greatly enhanced. With the increase in data processing volume, a single resource often cannot handle complex computing tasks. Therefore, more and more resources in the parallel computing cluster are required to execute corresponding tasks to meet actual application requirements. For example, multiple Graphic Processing Units (GPUs) are used to form a GPU cluster to meet the computing requirements in deep learning tasks for massive data. This requires efficient allocation of cluster resources to fully utilize their parallel computing capabilities.

[0003] In the prior art, when allocating resources to cluster nodes, the usage status of resources of each node is not considered. Instead, containers in the nodes are simply allocated according to the required computing resources, resulting in problems such as long waiting times for computing tasks, decreased parallel processing capabilities of the cluster, and low resource utilization. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for scheduling cluster node resources, which can improve the utilization rate of cluster resources.

[0005] To solve the above technical problem, the embodiments of the present invention provide the following technical solutions:

[0006] On the one hand, a method for scheduling cluster node resources is provided, which is used to schedule physical nodes in a cluster to create a container cluster to run user applications, including:

[0007] Obtain parameters of the user application, where the parameters of the user application include at least one of the following: training data set, expected training duration, application type, number of containers, GPU model in the cluster, and machine learning model;

[0008] Obtain the resource usage information of each physical node in the cluster;

[0009] Calculate the score of each physical node according to the resource usage information of each physical node and the parameters, and allocate containers to each physical node according to the score of each physical node to create a container cluster.

[0010] Optionally, the resources of the physical node include GPU, CPU, memory, and disk;

[0011] The resource usage information of physical nodes includes: GPU load, CPU usage rate, memory usage rate, and disk usage rate.

[0012] Optionally, calculating the score of each physical node according to the resource usage information of each physical node and the parameter includes:

[0013] Determining the weight value of each resource according to the application type;

[0014] Calculating the score of each physical node according to the weight value of each resource and the number of free resources of each resource.

[0015] Optionally, the application types include GPU-intensive applications, CPU-intensive applications, and memory-intensive applications.

[0016] For GPU-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the CPU weight value is greater than the memory weight value, and the GPU weight value is greater than the sum of the CPU weight value, memory weight value, and disk weight value;

[0017] For CPU-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the memory weight value is greater than the disk weight value, the CPU weight value is greater than the memory weight value, and the GPU weight value is 0;

[0018] For memory-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the CPU weight value is greater than the disk weight value, the memory weight value is greater than the CPU weight value, and the GPU weight value is 0.

[0019] Optionally, the number of free GPUs Free gpu(i) is calculated using the following formula:

[0020] Free gpu(i) = (free GPU num(i) + ∑min(free GPU memory ratio, (1 - GPUusage))

[0021] where free GPU num(i) is the number of free GPUs of the physical node, free GPU memory ratio is the memory free rate of the non-free GPUs of the physical node, and GPU usage is the computing load of the non-free GPUs of the physical node. If the GPU memory usage rate exceeds the first threshold or the GPU computing load exceeds the second threshold, then this GPU is considered to have no free resources.

[0022] Optionally, allocating containers to each physical node according to the score of each physical node includes:

[0023] Schedule containers to different physical nodes according to the score ranking of each physical node and the cluster roles of the containers.

[0024] Optionally, the scheduling of containers to different physical nodes according to the score ranking of each physical node and the cluster roles of the containers includes:

[0025] Label physical nodes according to the physical addresses of each physical node, the timestamps at the time of scheduling, the cluster roles of the containers, and the score ranking of the physical nodes;

[0026] Determine the physical nodes corresponding to the containers according to the cluster roles of each container and the timestamps at the time of scheduling, and label the containers according to the physical addresses of the physical nodes corresponding to the containers, the cluster roles of the containers, the timestamps at the time of scheduling, and the score ranking of the corresponding physical nodes;

[0027] Match the container labels with the physical node labels, and schedule and bind the containers to the matched physical nodes.

[0028] Optionally, before calculating the scores of each physical node according to the resource usage information of each physical node and the parameters, the method further includes:

[0029] Determine whether the application type is a GPU-intensive application;

[0030] If so, calculate the required number of GPUs according to the training dataset, the expected training duration, and the GPU model;

[0031] Compare the number of idle GPUs with the required number of GPUs. If the number of idle GPUs is less than the required number of GPUs, adjust the expected training duration and the training dataset until the number of idle GPUs is not less than the required number of GPUs.

[0032] Optionally, after creating the container cluster, the method further includes:

[0033] Run the user application on the container cluster;

[0034] After the operation is completed, obtain the output result of the user application and release the resources occupied by the container cluster.

[0035] An embodiment of the present invention also provides a cluster node resource scheduling device for scheduling physical nodes in a cluster to create a container cluster to run a user application, including:

[0036] The first acquisition module is used to acquire the parameters of the user application, where the parameters of the user application include at least one of the following: training data set, expected training duration, application type, number of containers, graphics processing unit (GPU) model in the cluster, and machine learning model;

[0037] The second acquisition module is used to acquire the resource usage information of each physical node in the cluster;

[0038] The processing module is used to calculate the score of each physical node according to the resource usage information of each physical node and the parameters, and allocate containers to each physical node according to the score of each physical node to create a container cluster.

[0039] Optionally, the resources of the physical node include GPU, CPU, memory, and disk;

[0040] The resource usage information of the physical node includes: GPU load, CPU usage, memory usage, and disk usage.

[0041] Optionally, the processing module includes:

[0042] The determination sub-module is used to determine the weight value of each resource according to the application type;

[0043] The calculation sub-module is used to calculate the score of each physical node according to the weight value of each resource and the number of free resources of each resource.

[0044] Optionally, the application type includes GPU-intensive applications, CPU-intensive applications, and memory-intensive applications.

[0045] For GPU-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the CPU weight value is greater than the memory weight value, and the GPU weight value is greater than the sum of the CPU weight value, memory weight value, and disk weight value;

[0046] For CPU-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the memory weight value is greater than the disk weight value, the CPU weight value is greater than the memory weight value, and the GPU weight value is 0;

[0047] For memory-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the CPU weight value is greater than the disk weight value, the memory weight value is greater than the CPU weight value, and the GPU weight value is 0.

[0048] Optionally, the number of free GPUs Free gpu(i) is calculated using the following formula:

[0049] Free gpu(i) = (free GPU num(i) + ∑min(free GPU memory ratio, (1 - GPUusage))

[0050] Among them, free GPU num(i) is the number of idle GPUs of the physical node, free GPU memory ratio is the memory free ratio of the non-idle GPUs of the physical node, and GPU usage is the computing load of the non-idle GPUs of the physical node. Among them, if the GPU memory usage rate exceeds the first threshold or the GPU computing load exceeds the second threshold, it is considered that this GPU has no idle resources.

[0051] Optionally, the processing module is specifically configured to schedule containers to different physical nodes according to the score sorting of each physical node and the cluster role of the containers.

[0052] Optionally, the processing module includes:

[0053] The first tagging sub-module is used to tag physical nodes according to the physical address of each physical node, the timestamp at the time of scheduling, the cluster role of the container, and the score sorting of the physical node;

[0054] The second tagging sub-module is used to determine the physical node corresponding to the container according to the cluster role of each container and the timestamp at the time of scheduling, and tag the container according to the physical address of the physical node corresponding to the container, the cluster role of the container, the timestamp at the time of scheduling, and the score sorting of the corresponding physical node;

[0055] The matching module is used to match the container tag with the physical node tag, schedule the container to the matched physical node and bind them.

[0056] Optionally, the device further includes:

[0057] The judgment module is used to determine whether the application type is a GPU-intensive application;

[0058] The calculation module is used to calculate the required number of GPUs according to the training data set, the expected training duration, and the GPU model if the application type is a GPU-intensive application;

[0059] The adjustment module is used to compare the number of idle GPUs with the required number of GPUs. If the number of idle GPUs is less than the required number of GPUs, adjust the expected training duration and the training data set until the number of idle GPUs is not less than the required number of GPUs.

[0060] Optionally, the device further includes:

[0061] A running module, configured to run a user application on the container cluster;

[0062] A releasing module, configured to obtain the output result of the user application after the running is completed, and release the resources occupied by the container cluster.

[0063] An embodiment of the present invention further provides a cluster node resource scheduling device, including:

[0064] A processor; and

[0065] A memory, in which computer program instructions are stored,

[0066] wherein, when the computer program instructions are run by the processor, the processor is caused to execute the steps in the cluster node resource scheduling method as described above.

[0067] An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is run by a processor, the processor is caused to execute the steps in the cluster node resource scheduling method as described above.

[0068] The embodiments of the present invention have the following beneficial effects:

[0069] In the above solution, scores of each physical node are calculated according to the resource usage information of each physical node and the parameters of the user application, and containers are allocated to each physical node according to the scores of each physical node to create a container cluster. In this embodiment, according to the resource utilization situation of the physical nodes, the containers are deployed to the physical nodes in order according to the resource idle status of the physical nodes, so that the containers can be reasonably scheduled to different physical nodes, improving the resource utilization efficiency and the processing performance at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic flowchart of a cluster node resource scheduling method according to Embodiment 1 of the present invention;

[0071] Figure 2 It is a schematic diagram of cluster node resource scheduling according to an embodiment of the present invention;

[0072] Figure 3 It is a schematic flowchart of a cluster node resource scheduling method according to Embodiment 2 of the present invention;

[0073] Figure 4 It is a schematic flowchart of a cluster node resource scheduling method according to Embodiment 3 of the present invention;

[0074] Figure 5 It is a schematic flowchart of a cluster node resource scheduling method according to Embodiment 4 of the present invention;

[0075] Figure 6The block diagram of the cluster node resource scheduling device according to the fifth embodiment of the present invention;

[0076] Figure 7 The block diagram of the cluster node resource scheduling device according to the sixth embodiment of the present invention. Detailed implementation manners

[0077] To make the technical problems, technical solutions and advantages to be solved by the embodiments of the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0078] Kubernetes is a container orchestration engine open-sourced by Google and is an open-source system for automatically deploying, scaling, and managing "containerized applications". The goal of Kubernetes is to eliminate the burden of orchestrating physical and / or virtual computing, network, and storage infrastructure, and enable application operators and developers to fully focus on self-operation with container-centric primitives. Kubernetes also provides a stable and compatible foundation (platform) for building customized workflows and more advanced automated tasks. Kubernetes has perfect cluster management capabilities, including multi-level security protection and admission mechanisms, multi-tenant application support capabilities, transparent service registration and service discovery mechanisms, built-in load balancers, fault discovery and self-healing capabilities, service rolling upgrades and online scaling, scalable resource automatic scheduling mechanisms, multi-granularity resource quota management capabilities, and also provides perfect management tools covering all aspects such as development, deployment testing, and operation and maintenance monitoring.

[0079] In the related art, when allocating resources to cluster nodes, only the complexity of the neural network model is simply analyzed, and the usage status of the resources of each physical node is not considered. The containers are not allocated more refinedly according to the resource utilization rate of the physical nodes, which is relatively rough.

[0080] The technical problem to be solved by the present invention is to provide a cluster node resource scheduling method and device, which can improve the utilization rate of cluster resources.

[0081] Embodiment 1

[0082] This embodiment provides a cluster node resource scheduling method for scheduling physical nodes in a cluster to create a container cluster to run user applications, such as Figure 2 As shown, deploy a distributed container cluster on the Kubernetes platform for user applications. The user applications include GPU-intensive applications, compute-intensive applications, and memory-intensive applications. The Kubernetes platform schedules the containers to matching physical nodes for binding and starts the container cluster to run the user applications.

[0083] Such asFigure 1 As shown in the figure, this embodiment includes the following steps:

[0084] Step 101: Obtain the parameters of the user application, where the parameters of the user application include at least one of the following: training data set, expected training duration, application type, number of containers, graphics processing unit (GPU) model in the cluster, and machine learning model;

[0085] Among them, the application type includes GPU-intensive applications, compute-intensive applications, and memory-intensive applications.

[0086] Step 102: Obtain the resource usage information of each physical node in the cluster;

[0087] Among them, the resources of the physical node can include GPUs, CPUs, memory, and disks; the resource usage information of the physical node includes: GPU load, CPU usage, memory usage, and disk usage. In this embodiment, it is necessary to dynamically and continuously monitor the resource usage information of each physical node.

[0088] Step 103: Calculate the score of each physical node according to the resource usage information of each physical node and the parameters, and allocate containers to each physical node according to the score of each physical node to create a container cluster.

[0089] The specific calculation of the score of each physical node according to the resource usage information of each physical node and the parameters includes: determining the weight value of each resource according to the application type; calculating the score of each physical node according to the weight value of each resource and the free number of each resource.

[0090] For GPU-intensive applications, the sum of the GPU weight value W1, CPU weight value W2, memory weight value W3, and disk weight value W4 is equal to 1, the CPU weight value W2 is greater than the memory weight value W3, and the GPU weight value W1 is greater than the sum of the CPU weight value W2, memory weight value W3, and disk weight value W4; in a specific example, W1 + W2 + W3 + W4 = 1, W1 > W2 > W3 = W4, W1 = 0.55, W2 = 0.25, W3 = W4 = 0.1.

[0091] For compute-intensive applications, the sum of the GPU weight value W1, CPU weight value W2, memory weight value W3, and disk weight value W4 is equal to 1, the memory weight value W3 is greater than the disk weight value W4, the CPU weight value W2 is greater than the memory weight value W3, and the GPU weight value is 0; in a specific example, W1 + W2 + W3 + W4 = 1, W2 > W3 > W4, W1 = 0, W2 = 0.5, W3 = 0.3, W4 = 0.2.

[0092] For memory-intensive applications, the sum of the GPU weight value W1, the CPU weight value W2, the memory weight value W3, and the disk weight value W4 is equal to 1. The CPU weight value W2 is greater than the disk weight value W4, the memory weight value W3 is greater than the CPU weight value W2, and the GPU weight value is 0. In a specific example, W1 + W2 + W3 + W4 = 1, W3 > W2 > W4, W1 = 0, W2 = 0.3, W3 = 0.5, and W4 = 0.2.

[0093] Among them, the number of idle GPUs Free gpu(i) of the GPU can be calculated using the following formula:

[0094] Free gpu(i) = (free GPU num(i) + ∑min(free GPU memory ratio, (1 - GPUusage))

[0095] Among them, free GPU num(i) is the number of idle GPUs of the physical node, free GPU memory ratio is the memory idle rate of the non-idle GPUs of the physical node, and GPU usage is the computing load of the non-idle GPUs of the physical node. If the GPU memory usage rate exceeds the first threshold or the GPU computing load exceeds the second threshold, then this GPU is considered to have no idle resources. Specifically, the first threshold can be 0.5, and the second threshold can be 0.7. Of course, the first threshold and the second threshold can also take other values according to the actual application situation.

[0096] Among them, the memory idle rate of the GPU = the idle memory of the running GPU / the total memory.

[0097] Specifically, the score SCORE(i) of each physical node can be calculated using the following formula:

[0098] SCORE(i) = W1 * (Free gpu(i)) + W2 * (idle CPU(i)) + W3 * (free memory(i)) + W4(free disk(i))

[0099] Among them, idle CPU(i) is the number of idle CPUs, free memory(i) is the number of idle memories, and freedisk(i) is the number of idle disks.

[0100] The score of a physical node reflects the amount of idle resources of the physical node. The higher the score of the physical node, the more idle resources of the physical resources; the lower the score of the physical node, the fewer idle resources of the physical resources. After calculating the scores of the physical nodes, the containers can be scheduled to different physical nodes according to the score sorting of each physical node and the cluster roles of the containers. The roles of the distributed machine learning cluster include Master, PS, and Worker. When allocating physical nodes, the physical node with the least idle resources can be allocated to PS, the physical node with the second least idle resources can be allocated to Master, and the remaining physical nodes can be allocated to Worker.

[0101] Specifically, tags can be assigned to physical nodes according to the physical address of each physical node, the timestamp at the time of scheduling, the cluster role of the container, and the score sorting of the physical nodes; the physical node corresponding to the container can be determined according to the cluster role of each container and the timestamp at the time of scheduling, and tags can be assigned to the container according to the physical address of the physical node corresponding to the container, the cluster role of the container, the timestamp at the time of scheduling, and the score sorting of the corresponding physical node; the container tags are matched with the physical node tags, and the container is scheduled to the matched physical node and bound.

[0102] When assigning tags to physical nodes, first obtain the Mac address of the current physical node, convert the Mac address from hexadecimal to decimal to get the converted value MacInt; then obtain the timestamp T when the physical node is allocated; determine the cluster role Role of the container corresponding to the physical node according to the score of the physical node. For example, the value of Role can be 0, 1, 2, where 0 represents Master, 2 represents PS, and 3 represents Worker; finally, obtain the tag of the physical node Label(i) = MacInt_Role_T_No, where No is the index assigned to the physical node after sorting the physical nodes according to their scores.

[0103] When tagging containers, randomly number all the containers in the cluster according to the cluster roles as PodNo, PodNo = Num(shuf(Podlist)), and the containers can be numbered starting from 0; combine PodNo with the timestamp T as the key of the current container, key = T + PodNo; perform a modulo operation on the total number of physical nodes of each role in the cluster with the key to obtain the physical node index nodeIndex corresponding to this container, nodeIndex = key mod num(NodeList(i)). After that, the Mac address and role Role of the physical node corresponding to this nodeIndex can be obtained, and then Mac, Role, timestamp T, and Pod_index are combined together as the label of this container Pod_Node_label = MacInt_Role_T_nodeIndex.

[0104] Match the Pod_label with the physical node label. After successful matching, schedule this container to the matched physical node and bind it. Loop this operation until all containers are scheduled. Specifically, the container can be scheduled to the physical node by calling a custom Kubernetes scheduler.

[0105] The establishment of the container cluster can be completed through the above steps. After creating the container cluster, user applications can be run on the container cluster. Kubernetes allocates the containers of each cluster role to the specified physical nodes for training; after the operation is completed, the output results of the user application are obtained, and all resources are automatically recycled by Kubernetes, releasing the resources occupied by the container cluster for the next user application to use. At the same time, Kubernetes will delete the labels of each physical node.

[0106] In addition, before calculating the score of each physical node according to the resource usage information of each physical node and the parameters, it is also necessary to determine whether the application type is a GPU-intensive application; if the application type is a GPU-intensive application, it is necessary to calculate the required number of GPUs according to the training dataset, the expected training duration, and the GPU model; compare the number of idle GPUs with the required number of GPUs. If the number of idle GPUs is less than the required number of GPUs, adjust the expected training duration and the training dataset until the number of idle GPUs is not less than the required number of GPUs. Among them, the required number of GPUs = training dataset / GPU speed / expected training duration, and the GPU speed can be calculated from the GPU computing power and GPU memory determined according to the GPU model.

[0107] It should be noted that the above embodiments are applicable to the scenario where the number of physical nodes is greater than 1. If the number of physical nodes is equal to 1, all containers are directly allocated to the same physical node.

[0108] In this embodiment, the scores of each physical node are calculated according to the resource usage information of each physical node and the parameters of the user application, and containers are allocated to each physical node according to the scores of each physical node to create a container cluster. In this embodiment, according to the resource utilization of the physical nodes, the containers are deployed to the physical nodes in order according to the resource idle status of the physical nodes, which can reasonably schedule the containers to different physical nodes, improving the processing performance while more efficiently utilizing the resources.

[0109] Embodiment Two

[0110] In this embodiment, sentiment analysis is the application that the user needs to process. This is a GPU-intensive application involving deep learning, and a distributed TensorFlow container cluster needs to be deployed on the Kubernetes platform to process this application. In this embodiment, the Kubernetes platform is deployed on 5 physical nodes, and each physical node is configured with a GPU, CPU, memory, and disk. In this embodiment, the resource usage information of each physical node is continuously monitored dynamically, and the TensorFlow containers are reasonably scheduled to different physical nodes according to the resource utilization rate, improving the processing performance while more efficiently utilizing the resources.

[0111] As Figure 3 shown, this embodiment includes the following steps:

[0112] Step 201: Obtain the parameters of the input sentiment analysis application;

[0113] After the user determines the application, parameters such as the training data set, the expected training duration, the GPU model, and the number of Pods (containers) to be created will be input. In this embodiment, the application type is a GPU-intensive application, the size of the training data set is 1TB, the expected training duration is 4 hours, and the GPU model is GeForce RTX2070.

[0114] Step 202: Obtain the current resource usage of all physical nodes;

[0115] In this embodiment, there are 5 physical nodes, and the resources of each physical node include a GPU, CPU, memory, and disk.

[0116] Step 203: Set the weight ratio for the resources of each physical node;

[0117] Set weight ratios for GPU, CPU, memory, and disk. The principle for setting is that the sum of the weight values of the four (GPU, CPU, memory, disk) is 1; the weight value of memory is equal to the weight value of disk; the weight value of CPU is greater than the weight value of memory; and at the same time, the weight value of GPU is greater than the sum of the weight values of the other three. Set the weight values of these four resource metrics according to such a distribution principle.

[0118] Step 204: Obtain the default computing power and total memory value of the current GPU according to the GPU model;

[0119] Among them, the GPU memory usage rate of each physical node can be obtained by comparing the used memory value of the physical node GPU with the total memory value. If the GPU memory usage rate exceeds 0.5 or the GPU computing load exceeds 0.7, it is considered that this GPU has no idle resources.

[0120] Step 205: Calculate the number of idle GPUs on the physical node;

[0121] For the current physical node, according to the usage situation of the GPU, set the number of completely idle GPUs to 1. For non-idle GPUs, take the minimum value of the memory idle rate and the GPU computing idle rate (e.g., min(memory idle rate 0.6, computing idle rate 0.65) = 0.6). Then, perform an iterative addition process on the number of GPUs of the current physical node and take the integer (e.g., the total available GPUs of the current physical node is 1 + 0.6 + 0 + 0.7 = 2). In this way, the number of idle GPUs on the current physical node can be calculated. For the other physical nodes, obtain the number of idle GPUs Free_GPUs according to the same algorithm, and the number of idle GPUs Free_GPUs of all physical nodes can be obtained.

[0122] Step 206: Calculate the idle resource score of the physical node;

[0123] Obtain the weight values of the four resources. For each physical node, multiply the GPU weight value by the number of unused GPUs, multiply the CPU weight value by the ratio of unused CPU, multiply the memory weight value by the ratio of unused memory, and multiply the disk weight value by the ratio of unused disk. Then, add the products of these four (GPU, CPU, memory, and disk) and sum them up to obtain the idle resource score of each physical node.

[0124] Step 207: Sort the physical nodes according to the idle resource score of each physical node;

[0125] Specifically, the physical nodes can be sorted from high to low according to the idle resource score.

[0126] Step 208: Calculate the required number of GPUs Expected_GPUs;

[0127] Since sentiment analysis is a GPU-intensive application involving deep learning, it is also necessary to calculate the required number of GPUs, Expected_GPUs, to determine whether the number of idle GPUs in the cluster can meet the application requirements.

[0128] Specifically, divide the training dataset by the expected training duration to calculate the data scale that can be trained per hour, and then divide the data scale that can be trained per hour by the GPU computing power obtained from the GPU model to calculate the required number of GPUs, Expected_GPUs.

[0129] Step 209: Compare the required number of GPUs with the number of currently idle GPUs in the cluster. If the required number of GPUs is more than the number of currently idle GPUs in the cluster, adjust the training dataset scale or the expected training duration, and loop this process until the required number of GPUs is not more than the number of currently idle GPUs in the cluster.

[0130] Among them, the number of currently idle GPUs in the cluster is the sum of the idle GPUs, Free_GPUs, of all physical nodes.

[0131] Step 210: Label the physical nodes.

[0132] Obtain the Mac address of the current physical node, and then perform decimal conversion (e.g., 48-89-E7-2A-60-11–72137231429617) to get the converted value Num1.

[0133] The TensorFlow distributed container cluster contains three roles, Role, namely Master, PS, and Worker (e.g., the values are 0, 1, 2 respectively); first, label the physical node with the least idle resource score as the PS role, then label the physical node with the second least idle resource score as the Master role, and finally label all the remaining physical nodes as the Worker role.

[0134] In addition, automatically obtain the timestamp T (e.g., 20191211141125) when the physical node is allocated.

[0135] Then sort according to the idle resource scores of the physical nodes to obtain the index Index of the physical nodes.

[0136] Use Num1, Role, T, and the allocated physical node index as the label Num1_Role_T_Index of the physical node. In a specific example, the label of the physical node can be 72137231429617_1_20191211141125_0.

[0137] Repeat the above process to obtain the labels of all physical nodes.

[0138] Step 211: Label the containers;

[0139] Randomly number all containers in the TensorFlow cluster according to the cluster role as PodNum. For example, the PodNum of the third container (03) with the Worker role (2) is 203.

[0140] Combine PodNum and the timestamp T as the key of the current container, and then perform a modulo operation on the total number of physical nodes of each role in the cluster with the key to obtain the physical node index Pod_index corresponding to this container.

[0141] Obtain the Mac address and role Role of the physical node corresponding to this Pod_index, and then combine Mac, Role, timestamp T, and Pod_index together as the label Pod_label of this container. In a specific example, the label of the container is 72137231429617_2_20191211141125_1.

[0142] Repeat the above process to obtain the labels of all containers.

[0143] Step 212: Match the container label with the physical node label. After successful matching, schedule this container to the matched physical node and bind it. Loop this operation until all containers are scheduled.

[0144] After scheduling all the containers required by the user, start the distributed TensorFlow cluster to perform sentiment analysis model training, and then return the results after the training execution to the user. Kubernetes stops running all containers related to this application on each physical node. Then, destroy all containers allocated for the sentiment analysis application on all physical nodes, release the resources occupied by the containers, and provide them for the user to execute the next application.

[0145] In this embodiment, the user only needs to determine parameters such as the training dataset and GPU model required by the application, and can automatically select appropriate physical nodes to quickly create containers and build a cluster. This embodiment can intelligently deploy containers to physical nodes in order according to the resource idle status based on the resource utilization of physical nodes, which can ensure the balanced utilization of resources while greatly reducing the time cost of cluster construction. At the same time, after the application is executed, the containers are automatically destroyed to release physical resources, ensuring that the resources can be reused.

[0146] Embodiment III

[0147] In this embodiment, WordCount is the application that the user needs to process. This is a CPU-intensive application, and a distributed Spark container cluster needs to be deployed on the Kubernetes platform to process this application. The Kubernetes platform is deployed on 5 physical nodes, and each node is configured with GPU, CPU, memory, and disk. In this embodiment, the resource usage information of each physical node is continuously monitored dynamically, and the Spark containers are reasonably scheduled to different physical nodes according to the resource utilization rate to process the WordCount application, improving the processing performance while more efficiently utilizing resources.

[0148] As Figure 4 shown, this embodiment includes the following steps:

[0149] Step 301: Obtain the parameters of the input WordCount application;

[0150] After the user determines the application, the user will input the application type, the number of Pods (containers) to be created, etc. In this embodiment, the WordCount application type is a GPU-intensive application.

[0151] Step 302: Obtain the current resource usage of all physical nodes;

[0152] In this embodiment, there are 5 physical nodes, and the resources of each physical node include GPU, CPU, memory, and disk.

[0153] Step 303: Set the weight ratio for the resources of each physical node;

[0154] For each physical node, set the weight ratio for GPU, CPU, memory, and disk. The setting principle is that the sum of the weight values of the four (GPU, CPU, memory, and disk) is 1; the weight value of memory is greater than the weight value of disk; the weight value of CPU is greater than the weight value of memory; and at the same time, since this is a CPU-intensive application, the weight value of GPU is set to 0. Set the weight values of these several resource indicators according to such an allocation principle.

[0155] Step 304: Calculate the free resource score of the physical node;

[0156] Obtain the weight values of the four resources. For each physical node, multiply the GPU weight value 0 by the number of unused GPUs, multiply the CPU weight value by the ratio of unused CPU, multiply the memory weight value by the ratio of unused memory, and multiply the disk weight value by the ratio of unused disk. Then, add the products of these four (GPU, CPU, memory, and disk) together to obtain the weight value of each physical node as its free resource score.

[0157] Step 305: Sort the physical nodes according to the free resource scores of each physical node;

[0158] Step 306: Label the physical nodes;

[0159] Obtain the Mac address of the current node, and then perform decimal conversion (e.g., 48-89-E7-2A-60-11–72137231429617) to get the converted value Num1.

[0160] The Spark distributed container cluster contains two roles, namely Master and Worker (the values are 0 and 1 respectively). Label the physical node with the least free resource score as the Master role, and then label all the remaining physical nodes as the Worker role.

[0161] In addition, automatically obtain the timestamp T when the physical node is allocated (e.g., 20191211141125).

[0162] Then sort according to the free resource scores of the physical nodes to obtain the index Index of the physical nodes.

[0163] Use Num1, Role, T, and the allocated physical node index as the label Num1_Role_T_Index of the physical node. In a specific example, the label of the physical node is 72137231429617_1_20191211141125_0.

[0164] Repeat the above process to obtain the labels of all physical nodes.

[0165] Step 307: Label the containers;

[0166] Randomly number all the containers in the Spark cluster according to the cluster role as PodNum. For example, the PodNum of the third container in the Worker role is 103. Combine PodNum and the timestamp T as the current container key, and then perform a modulo operation on the total number of physical nodes of each role in the cluster with the key to obtain the physical node Pod_index corresponding to this container;

[0167] Obtain the Mac address and role Role of the physical node corresponding to this Pod_index, and then combine Mac, Role, the timestamp T, and Pod_index together as the label Pod_label of this container.

[0168] Repeat the above process to obtain the labels of all containers.

[0169] Step 308: Match the container labels with the physical node labels. After successful matching, schedule this container to the matching physical node and bind it. Loop this operation until all containers are scheduled.

[0170] After scheduling all the containers required by the user, start the distributed Spark cluster to execute the WordCount application, and then return the execution result to the user. Kubernetes stops running all the containers related to this application on each physical node. Then, destroy all the containers allocated for the WordCount application on all physical nodes, release the resources occupied by the containers, and provide them for the user to execute the next application.

[0171] In this embodiment, the user only needs to determine the application type, and the appropriate physical nodes can be automatically selected according to the number of containers to be created to quickly create containers and build a cluster. This embodiment can dynamically deploy containers to physical nodes in order according to the resource idle status based on the resource utilization of physical nodes, which can ensure the balanced utilization of resources and greatly reduce the time cost of cluster building. At the same time, after the application is executed, the containers are automatically destroyed to release physical resources, ensuring that the resources can be reused.

[0172] Embodiment 4

[0173] In this embodiment, the leaderboard is the application that the user needs to process. This is a memory-intensive application, and a distributed Redis container cluster needs to be deployed on the Kubernetes platform to process this application. The Kubernetes platform is deployed on 5 physical nodes, and each node is configured with a GPU, CPU, memory, and disk. In this embodiment, continuously monitor the resource usage information of each physical node dynamically, and reasonably schedule Redis containers to different nodes according to the resource utilization rate to process the leaderboard application, which improves the processing performance while using resources more efficiently.

[0174] As Figure 5 shown, this embodiment includes the following steps:

[0175] Step 401: Obtain the parameters of the input leaderboard application;

[0176] After the user determines the application, the user will input the application type, the number of Pods (containers) to be created, etc. In this embodiment, the leaderboard application resource-intensive type is GPU-intensive.

[0177] Step 402: Obtain the current resource usage of all physical nodes;

[0178] In this embodiment, there are 5 physical nodes, and the resources of each physical node include GPU, CPU, memory, and disk.

[0179] Step 403: Set the weight ratios for the resources of each physical node;

[0180] For each physical node, set the weight ratios for GPU, CPU, memory, and disk. The setting principle is that the sum of the weight values of the four (GPU, CPU, memory, and disk) is 1; the weight value of CPU is greater than that of disk; the weight value of memory is greater than that of CPU; and at the same time, since this is a memory-intensive application, the weight value of GPU is set to 0. Set the weight values of these resource metrics according to such an allocation principle.

[0181] Step 404: Calculate the free resource scores of the physical nodes;

[0182] Obtain the weight values of the four resources. For each physical node, multiply the weight value of GPU, which is 0, by the number of unused GPUs, multiply the weight value of CPU by the ratio of unused CPU, multiply the weight value of memory by the ratio of unused memory, and multiply the weight value of disk by the ratio of unused disk. Then, add the products of these four (GPU, CPU, memory, and disk) together to obtain the weight value of each physical node as its free resource score.

[0183] Step 405: Sort the physical nodes according to the free resource scores of each physical node;

[0184] Step 406: Label the physical nodes;

[0185] Obtain the Mac address of the current physical node, and then perform decimal conversion (e.g., 48-89-E7-2A-60-11–72137231429617) to get the converted value Num1.

[0186] The Redis distributed container cluster contains two roles, Role, namely Master and Worker (the values are 0 and 1 respectively). Mark the physical node with the least free resource score as the Master role, and then mark all the remaining physical nodes as the Worker role.

[0187] In addition, automatically obtain the timestamp T at the time of node allocation (e.g., 20191211141125).

[0188] Then sort according to the free resource scores of the physical nodes to obtain the index Index of the physical nodes.

[0189] Using Num1, Role, T, and the allocated physical node index as the label of the physical node Num1_Role_T_Index. In a specific example, the label of the physical node is 72137231429617_1_20191211141125_0.

[0190] Repeat the above process to obtain the labels of all physical nodes.

[0191] Step 407: Label the containers;

[0192] Randomly number the Pods of all containers in the Redis cluster by role. For example, the PodNum of the third container in the Worker role is 103. Combine PodNum with the timestamp T as the current container key, and then perform a modulo operation on the total number of physical nodes of each role in the cluster using the key to obtain the physical node Pod_index corresponding to this container;

[0193] Obtain the Mac address of the node corresponding to this Pod_index, the role Role, and then combine Mac, Role, the timestamp T, and Pod_index together as the label Pod_label of this container.

[0194] Repeat the above process to obtain the labels of all containers.

[0195] Step 408: Match the Pod_label with the physical node label. After successful matching, schedule this container to the matched physical node and bind it. Loop this operation until all containers are scheduled.

[0196] After scheduling the containers required by the user, start the distributed Redis cluster to execute the leaderboard application, and then return the execution result to the user. Kubernetes stops running all containers related to this application on each physical node. Then, destroy all containers allocated for the leaderboard application on all physical nodes, release the resources occupied by the containers, and provide them for the user to execute the next application.

[0197] In this embodiment, the user only needs to determine the application type, the number of Pods, etc., and can automatically select appropriate physical nodes to quickly create containers and build a cluster. This embodiment can dynamically deploy containers to physical nodes in order according to the resource idle status based on the resource utilization of physical nodes, which can ensure the balanced utilization of resources and greatly reduce the time cost of cluster construction. At the same time, after the application is executed, the containers are automatically destroyed to release the physical resources, ensuring that the resources can be reused.

[0198] Embodiment Five

[0199] This embodiment provides a cluster node resource scheduling device for scheduling physical nodes in a cluster to create a container cluster for running user applications, such as Figure 6 As shown, this embodiment includes:

[0200] A first acquisition module 61, configured to acquire parameters of a user application, where the parameters of the user application include at least one of the following: a training data set, an expected training duration, an application type, the number of containers, the graphics processing unit (GPU) model in the cluster, and a machine learning model;

[0201] A second acquisition module 62, configured to acquire resource usage information of each physical node in the cluster;

[0202] Wherein, the resources of the physical node include a GPU, a CPU, memory, and a disk;

[0203] The resource usage information of the physical node includes: GPU load, CPU usage rate, memory usage rate, and disk usage rate.

[0204] A processing module 63, configured to calculate a score for each physical node according to the resource usage information of each physical node and the parameters, and allocate containers to each physical node according to the score of each physical node to create a container cluster.

[0205] Optionally, the processing module 63 includes:

[0206] A determination sub-module, configured to determine a weight value for each resource according to the application type;

[0207] A calculation sub-module, configured to calculate a score for each physical node according to the weight value of each resource and the number of free resources of each resource.

[0208] Optionally, the application type includes GPU-intensive applications, CPU-intensive applications, and memory-intensive applications.

[0209] For GPU-intensive applications, the sum of the GPU weight value, the CPU weight value, the memory weight value, and the disk weight value is equal to 1, the CPU weight value is greater than the memory weight value, and the GPU weight value is greater than the sum of the CPU weight value, the memory weight value, and the disk weight value;

[0210] For CPU-intensive applications, the sum of the GPU weight value, the CPU weight value, the memory weight value, and the disk weight value is equal to 1, the memory weight value is greater than the disk weight value, the CPU weight value is greater than the memory weight value, and the GPU weight value is 0;

[0211] For memory-intensive applications, the sum of the GPU weight value, the CPU weight value, the memory weight value, and the disk weight value is equal to 1, the CPU weight value is greater than the disk weight value, the memory weight value is greater than the CPU weight value, and the GPU weight value is 0.

[0212] Optionally, the number of free GPUs, Free gpu(i), of the GPU is calculated using the following formula:

[0213] Free gpu(i) = (free GPU num(i) + ∑min(free GPU memory ratio, (1 - GPUusage))

[0214] where free GPU num(i) is the number of free GPUs of the physical node, free GPU memory ratio is the memory free ratio of the non - free GPUs of the physical node, and GPU usage is the computing load of the non - free GPUs of the physical node. If the GPU memory usage rate exceeds the first threshold or the GPU computing load exceeds the second threshold, it is considered that this GPU has no free resources.

[0215] Optionally, the processing module 63 is specifically configured to schedule containers to different physical nodes according to the score ranking of each physical node and the cluster role of the containers.

[0216] Optionally, the processing module 63 includes:

[0217] A first tagging sub - module, configured to tag physical nodes according to the physical address of each physical node, the timestamp at the time of scheduling, the cluster role of the container, and the score ranking of the physical node;

[0218] A second tagging sub - module, configured to determine the physical node corresponding to the container according to the cluster role of each container and the timestamp at the time of scheduling, and tag the container according to the physical address of the physical node corresponding to the container, the cluster role of the container, the timestamp at the time of scheduling, and the score ranking of the corresponding physical node;

[0219] A matching module, configured to match the container tag with the physical node tag, and schedule and bind the container to the matched physical node.

[0220] Optionally, the apparatus further includes:

[0221] A judgment module, configured to determine whether the application type is a GPU - intensive application;

[0222] A calculation module, configured to, if the application type is a GPU - intensive application, calculate the required number of GPUs according to the training data set, the expected training duration, and the GPU model;

[0223] An adjustment module, configured to compare the number of idle GPUs with the number of required GPUs. If the number of idle GPUs is less than the number of required GPUs, adjust the expected training duration and the training dataset until the number of idle GPUs is not less than the number of required GPUs.

[0224] Optionally, the apparatus further includes:

[0225] An operation module, configured to run a user application on the container cluster;

[0226] A release module, configured to obtain the output result of the user application after the operation is completed, and release the resources occupied by the container cluster.

[0227] In this embodiment, the score of each physical node is calculated according to the resource usage information of each physical node and the parameters of the user application, and containers are allocated to each physical node according to the score of each physical node to create a container cluster. According to the resource utilization of the physical nodes, this embodiment deploys the containers to the physical nodes in order according to the resource idle status of the physical nodes, which can reasonably schedule the containers to different physical nodes, improving the processing performance while making more efficient use of resources.

[0228] Embodiment Six

[0229] An embodiment of the present invention further provides a cluster node resource scheduling device 50, as Figure 7 shown, including:

[0230] A processor 52; and

[0231] A memory 54, in which computer program instructions are stored,

[0232] wherein, when the computer program instructions are run by the processor, the processor 52 is caused to execute the following steps:

[0233] Obtain the parameters of the user application, where the parameters of the user application include at least one of the following: training dataset, expected training duration, application type, number of containers, graphics processing unit (GPU) model in the cluster, and machine learning model;

[0234] Obtain the resource usage information of each physical node in the cluster;

[0235] Calculate the score of each physical node according to the resource usage information of each physical node and the parameters, and allocate containers to each physical node according to the score of each physical node to create a container cluster.

[0236] In this embodiment, the scores of each physical node are calculated according to the resource usage information of each physical node and the parameters of the user application, and containers are allocated to each physical node according to the scores of each physical node to create a container cluster. According to the resource utilization of the physical nodes, this embodiment deploys the containers to the physical nodes in order according to the resource idle status of the physical nodes, which can reasonably schedule the containers to different physical nodes, improving the processing performance while more efficiently utilizing the resources.

[0237] Further, as Figure 7 shown, the cluster node resource scheduling device 50 further includes a network interface 51, an input device 53, a hard disk 55, and a display device 56.

[0238] The above-mentioned various interfaces and devices can be interconnected through a bus architecture. The bus architecture can include any number of interconnected buses and bridges. Specifically, one or more central processing units (CPUs) represented by the processor 52 and various circuits of one or more memories represented by the memory 54 are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. It can be understood that the bus architecture is used to implement the connection and communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are well known in the art and will not be described in detail herein.

[0239] The network interface 51 can be connected to a network (such as the Internet, a local area network, etc.), obtain relevant data from the network, and can be stored in the hard disk 55.

[0240] The input device 53 can receive various instructions input by an operator and send them to the processor 52 for execution. The input device 53 can include a keyboard or a pointing device (for example, a mouse, a trackball, a touchpad, or a touch screen, etc.).

[0241] The display device 56 can display the results obtained by the processor 52 executing instructions.

[0242] The memory 54 is used to store the programs and data necessary for the operation of the operating system, as well as data such as intermediate results in the calculation process of the processor 52.

[0243] It can be understood that the memory 54 in the embodiments of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. The memory 54 of the devices and methods described herein is intended to include, but is not limited to, these and any other suitable types of memories.

[0244] In some embodiments, the memory 54 stores the following elements, executable modules, or data structures, or subsets or extensions thereof: an operating system 541 and application programs 542.

[0245] Among them, the operating system 541 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application programs 542 include various application programs, such as a browser, etc., for implementing various application services. The program for implementing the method of the embodiments of the present invention can be included in the application programs 542.

[0246] When the above-mentioned processor 52 calls and executes the application programs and data stored in the memory 54, specifically, it obtains parameters of a user application, and the parameters of the user application include at least one of the following: a training data set, an expected training duration, an application type, the number of containers, the graphics processing unit (GPU) model in the cluster, and a machine learning model; obtains resource usage information of each physical node in the cluster; calculates a score for each physical node according to the resource usage information of each physical node and the parameters, and allocates containers to each physical node according to the scores of each physical node to create a container cluster.

[0247] The method disclosed in the above embodiments of the present invention can be applied to or implemented by the processor 52. The processor 52 may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed by the integrated logic circuit in hardware or instructions in software form in the processor 52. The above-mentioned processor 52 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. This storage medium is located in the memory 54, and the processor 52 reads the information in the memory 54 and combines its hardware to complete the steps of the above method.

[0248] It can be understood that these embodiments described herein can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, or other electronic units for performing the functions described in this application, or a combination thereof.

[0249] For software implementation, the technologies described herein can be implemented by modules (such as procedures, functions, etc.) that execute the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented inside or outside the processor.

[0250] Furthermore, the resources of the physical node include GPU, CPU, memory, and disk;

[0251] The resource usage information of the physical node includes: GPU load, CPU usage rate, memory usage rate, and disk usage rate.

[0252] Furthermore, the processor 52 determines the weight value of each resource according to the application type; and calculates the score of each physical node based on the weight value of each resource and the number of free resources of each resource.

[0253] Further, the application types include GPU-intensive applications, CPU-intensive applications, and memory-intensive applications.

[0254] For GPU-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1. The CPU weight value is greater than the memory weight value, and the GPU weight value is greater than the sum of the CPU weight value, memory weight value, and disk weight value.

[0255] For CPU-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1. The memory weight value is greater than the disk weight value, the CPU weight value is greater than the memory weight value, and the GPU weight value is 0.

[0256] For memory-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1. The CPU weight value is greater than the disk weight value, the memory weight value is greater than the CPU weight value, and the GPU weight value is 0.

[0257] Optionally, the number of free GPUs Free gpu(i) of the GPU is calculated using the following formula:

[0258] Free gpu(i) = (free GPU num(i) + ∑min(free GPU memory ratio, (1 - GPUusage))

[0259] where free GPU num(i) is the number of free GPUs of the physical node, free GPU memory ratio is the memory free rate of the non-free GPUs of the physical node, and GPU usage is the computing load of the non-free GPUs of the physical node. If the GPU memory usage rate exceeds the first threshold or the GPU computing load exceeds the second threshold, it is considered that this GPU has no free resources.

[0260] Further, the processor 52 schedules the containers to different physical nodes according to the score ranking of each physical node and the cluster role of the containers.

[0261] Further, the processor 52 tags the physical nodes according to the physical address of each physical node, the timestamp at the time of scheduling, the cluster role of the containers, and the score ranking of the physical nodes.

[0262] Determine the physical node corresponding to the container according to the cluster role of each container and the timestamp at the time of scheduling. Tag the container according to the physical address of the physical node corresponding to the container, the cluster role of the container, the timestamp at the time of scheduling, and the score ranking of the corresponding physical node. Match the container tag with the physical node tag, and schedule the container to the matched physical node and bind them.

[0263] Further, the processor 52 determines whether the application type is a GPU-intensive application; if so, calculates the required number of GPUs according to the training data set, the desired training duration, and the GPU model; compares the number of idle GPUs with the required number of GPUs, and if the number of idle GPUs is less than the required number of GPUs, adjusts the desired training duration and the training data set until the number of idle GPUs is not less than the required number of GPUs.

[0264] Further, the processor 52 runs the user application on the container cluster; after the run is completed, obtains the output result of the user application, and releases the resources occupied by the container cluster.

[0265] Embodiment Seven

[0266] The embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is run by a processor, the processor is caused to execute the following steps:

[0267] Obtain the parameters of the user application, where the parameters of the user application include at least one of the following: training data set, desired training duration, application type, number of containers, graphics processing unit (GPU) model in the cluster, and machine learning model;

[0268] Obtain the resource usage information of each physical node in the cluster;

[0269] Calculate the score of each physical node according to the resource usage information of each physical node and the parameters, and allocate containers to each physical node according to the score of each physical node to create a container cluster.

[0270] In this embodiment, the score of each physical node is calculated according to the resource usage information of each physical node and the parameters of the user application, and containers are allocated to each physical node according to the score of each physical node to create a container cluster. According to the resource utilization situation of the physical nodes, this embodiment deploys the containers to the physical nodes in order according to the resource idle status of the physical nodes, can reasonably schedule the containers to different physical nodes, and improves the processing performance while more efficiently utilizing the resources.

[0271] Further, when the computer program is run by the processor, the processor is further caused to execute the following steps:

[0272] Determine the weight value of each resource according to the application type;

[0273] Calculate the score of each physical node according to the weight value of each resource and the number of idle resources of each resource.

[0274] Further, when the computer program is run by a processor, the processor is further caused to perform the following steps:

[0275] Schedule containers to different physical nodes according to the score ranking of each physical node and the cluster roles of the containers.

[0276] Further, when the computer program is run by a processor, the processor is further caused to perform the following steps:

[0277] Label physical nodes according to the physical addresses of each physical node, the timestamp at the time of scheduling, the cluster roles of the containers, and the score ranking of the physical nodes;

[0278] Determine the physical nodes corresponding to the containers according to the cluster roles of each container and the timestamp at the time of scheduling, and label the containers according to the physical addresses of the physical nodes corresponding to the containers, the cluster roles of the containers, the timestamp at the time of scheduling, and the score ranking of the corresponding physical nodes;

[0279] Match the container labels with the physical node labels, and schedule the containers to the matching physical nodes and bind them.

[0280] Further, when the computer program is run by a processor, the processor is further caused to perform the following steps:

[0281] Determine whether the application type is a GPU-intensive application;

[0282] If so, calculate the required number of GPUs according to the training data set, the expected training duration, and the GPU model;

[0283] Compare the number of idle GPUs with the required number of GPUs. If the number of idle GPUs is less than the required number of GPUs, adjust the expected training duration and the training data set until the number of idle GPUs is not less than the required number of GPUs.

[0284] Further, when the computer program is run by a processor, the processor is further caused to perform the following steps:

[0285] Run user applications on the container cluster;

[0286] After the run is completed, obtain the output results of the user applications and release the resources occupied by the container cluster.

[0287] The above are the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for scheduling cluster node resources, characterized in that, Used to schedule physical nodes in a cluster to create a container cluster to run user applications, including: Obtain parameters of the user application, where the parameters of the user application include at least one of the following: training data set, expected training duration, application type, number of containers, graphics processing unit (GPU) model in the cluster, and machine learning model; Obtain the resource usage information of each physical node in the cluster; Calculate the score of each physical node according to the resource usage information of each physical node and the parameters, and allocate containers to each physical node according to the score of each physical node to create a container cluster; Among them, the resources of the physical node include GPU, CPU, memory, and disk; The resource usage information of the physical node includes: GPU load, CPU usage rate, memory usage rate, and disk usage rate; Among them, the calculating the score of each physical node according to the resource usage information of each physical node and the parameters includes: Determine the weight value of each resource according to the application type; Calculate the score of each physical node according to the weight value of each resource and the free number of each resource; Among them, the application type includes GPU-intensive applications, CPU-intensive applications, and memory-intensive applications. For GPU-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the CPU weight value is greater than the memory weight value, and the GPU weight value is greater than the sum of the CPU weight value, memory weight value, and disk weight value; For CPU-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the memory weight value is greater than the disk weight value, the CPU weight value is greater than the memory weight value, and the GPU weight value is 0; For memory-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the CPU weight value is greater than the disk weight value, the memory weight value is greater than the CPU weight value, and the GPU weight value is 0; Among them, the free number of GPUs Free gpu(i) is calculated using the following formula: Free gpu(i) = (free GPU num(i) + ∑min(free GPU memory ratio, (1 - GPU usage)) Among them, free GPU num(i) is the number of free GPUs of the physical node, free GPU memory ratio is the memory free rate of the non-free GPUs of the physical node, and GPU usage is the computing load of the non-free GPUs of the physical node. If the GPU memory usage rate exceeds the first threshold or the GPU computing load exceeds the second threshold, it is considered that this GPU has no free resources.

2. The cluster node resource scheduling method according to claim 1, wherein The allocating containers to each physical node according to the score of each physical node includes: Scheduling containers to different physical nodes according to the score ranking of each physical node and the cluster role of the containers.

3. The cluster node resource scheduling method according to claim 2, wherein, The scheduling containers to different physical nodes according to the score ranking of each physical node and the cluster role of the containers includes: Tag the physical nodes according to the physical addresses of each physical node, the timestamps at the time of scheduling, the cluster roles of the containers, and the score sorting of the physical nodes; Determine the physical node corresponding to the container according to the cluster role of each container and the timestamp at the time of scheduling, and tag the container according to the physical address of the physical node corresponding to the container, the cluster role of the container, the timestamp at the time of scheduling, and the score sorting of the corresponding physical node; Match the container tags with the physical node tags, and schedule the container to the matched physical node and bind them.

4. The cluster node resource scheduling method according to any one of claims 1-3, characterized in that Before calculating the score of each physical node according to the resource usage information of each physical node and the parameter, the method further includes: Determine whether the application type is a GPU-intensive application; If so, calculate the required number of GPUs according to the training data set, the expected training duration, and the GPU model; Compare the number of idle GPUs with the required number of GPUs. If the number of idle GPUs is less than the required number of GPUs, adjust the expected training duration and the training data set until the number of idle GPUs is not less than the required number of GPUs.

5. The cluster node resource scheduling method according to claim 4, wherein After creating the container cluster, the method further includes: Run the user application on the container cluster; Obtain the output result of the user application after running, and release the resources occupied by the container cluster.

6. A cluster node resource scheduling device, characterized in that, Used to schedule physical nodes in the cluster to create a container cluster to run a user application, including: A first acquisition module, configured to acquire parameters of the user application, where the parameters of the user application include at least one of the following: training data set, expected training duration, application type, number of containers, graphics processing unit GPU model in the cluster, and machine learning model; A second acquisition module, configured to acquire the resource usage information of each physical node in the cluster; A processing module, configured to calculate the score of each physical node according to the resource usage information of each physical node and the parameter, and allocate containers to each physical node according to the score of each physical node to create a container cluster; Wherein, the resources of the physical node include GPU, CPU, memory, and disk; The resource usage information of the physical node includes: GPU load, CPU usage rate, memory usage rate, and disk usage rate; Wherein, the processing module includes: A determination sub-module, configured to determine the weight value of each resource according to the application type; A calculation sub-module, configured to calculate the score of each physical node according to the weight value of each resource and the number of idle resources of each resource; Wherein, the application type includes GPU-intensive applications, CPU-intensive applications, and memory-intensive applications, For GPU-intensive applications, the sum of the GPU weight value, the CPU weight value, the memory weight value, and the disk weight value is equal to 1, the CPU weight value is greater than the memory weight value, and the GPU weight value is greater than the sum of the CPU weight value, the memory weight value, and the disk weight value; For CPU-intensive applications, the sum of the GPU weight value, the CPU weight value, the memory weight value, and the disk weight value is equal to 1, the memory weight value is greater than the disk weight value, the CPU weight value is greater than the memory weight value, and the GPU weight value is 0; For memory-intensive applications, the sum of the GPU weight value, CPU weight value, memory weight value, and disk weight value is equal to 1, the CPU weight value is greater than the disk weight value, the memory weight value is greater than the CPU weight value, and the GPU weight value is 0; Among them, the number of free GPUs Free gpu(i) of the GPU is calculated using the following formula: Free gpu(i) = (free GPU num(i) + ∑min(free GPU memory ratio, (1 - GPU usage)) Among them, free GPU num(i) is the number of free GPUs of the physical node, free GPU memory ratio is the memory free ratio of the non-free GPUs of the physical node, and GPU usage is the computing load of the non-free GPUs of the physical node. If the GPU memory usage exceeds the first threshold or the GPU computing load exceeds the second threshold, it is considered that this GPU has no free resources.

7. The cluster node resource scheduling device according to claim 6, wherein The processing module is specifically configured to schedule containers to different physical nodes according to the score ranking of each physical node and the cluster role of the containers.

8. The cluster node resource scheduling device according to claim 7, wherein The processing module includes: The first tagging sub-module is used to tag physical nodes according to the physical address of each physical node, the timestamp at the time of scheduling, the cluster role of the container, and the score ranking of the physical node; The second tagging sub-module is used to determine the physical node corresponding to the container according to the cluster role of each container and the timestamp at the time of scheduling, and tag the container according to the physical address of the physical node corresponding to the container, the cluster role of the container, the timestamp at the time of scheduling, and the score ranking of the corresponding physical node; The matching module is used to match the container tags with the physical node tags, and schedule the container to the matched physical node and bind them.

9. The cluster node resource scheduling device according to any one of claims 6-8, characterized in that, The device further includes: The judgment module is used to determine whether the application type is a GPU-intensive application; The calculation module is used to, if the application type is a GPU-intensive application, calculate the required number of GPUs according to the training data set, the expected training duration, and the GPU model; The adjustment module is used to compare the number of free GPUs with the required number of GPUs. If the number of free GPUs is less than the required number of GPUs, adjust the expected training duration and the training data set until the number of free GPUs is not less than the required number of GPUs.

10. The cluster node resource scheduling device according to claim 9, wherein The device further includes: The running module is used to run user applications on the container cluster; The release module is used to obtain the output result of the user application after running is completed, and release the resources occupied by the container cluster.

Images