A distributed machine learning task GPU resource scheduling and allocation method and system

Abstract

The present invention discloses a distributed machine learning training task GPU resource scheduling allocation method and system, the method includes: S1: allocating a parameter server and a working node for each training job; S2: fitting a training speed model by collecting the training speed of the job under different parameter server and working node number combinations; S3: collecting the loss value of each training iteration of the training job, fitting the loss value iteration curve of the job, and calculating the remaining completion time of the training job; S4: the node allocation module allocates the appropriate number of working nodes and parameter servers for each job according to the training speed model; S5: the job placement module selects the placement strategy with the lowest cost based on the distributed communication delay and the queuing waiting time according to the short job priority strategy; the system includes a node allocation module and a job placement module. The present invention can achieve the goal of reducing the average completion time of cluster training jobs.

Description

Technical Field

[0001] The present invention relates to a method for implementing GPU resource scheduling and allocation, and in particular to a method and system for resource scheduling and allocation for machine learning training tasks using a parameter server architecture in a GPU data center that deploys virtualization technology on a large scale. Background Art

[0002] As artificial intelligence technology matures, more and more machine learning applications have emerged. Internet companies usually choose to deploy machine learning training tasks on cloud computing servers. Machine learning training tasks are floating-point computing intensive, so they rely heavily on powerful but expensive GPU resources.

[0003] Currently, the GPU of a single node is usually unable to cope with massive amounts of training data, so machine learning training tasks generally adopt a distributed parameter server architecture. Under this architecture, the training data of the model is divided into smaller scales and distributed to multiple worker nodes for training. Each worker node communicates with the parameter server node to update the gradient / parameters in each training cycle. Currently, there are two main training modes for the parameter server architecture: 1) asynchronous training, in which the training progress of different worker nodes in the training job is not synchronized, and the parameter server updates its model partition each time it receives the gradient of a worker node; 2) synchronous training, in which the training progress of all worker nodes in the training job is synchronized in each step, and the parameter server updates the parameters after collecting gradients from all worker nodes.

[0004] Therefore, distributed machine learning frameworks usually need to schedule tasks on multiple GPUs at the same time, which is called group scheduling, which increases the risk of resource fragmentation and low utilization in shared clusters. In addition, multi-GPU training also means synchronization of model parameters across nodes, so the network communication delay between nodes will greatly affect the completion time of the training job. Therefore, if you want to communicate faster between machines, you need to ensure that the node placement scheduling achieves better locality.

[0005] As for cloud computing resource providers, they usually use traditional cluster schedulers, such as Kubernetes or YARN. Such schedulers simply treat machine learning task loads as a big data processing job, assign the job to a machine with a GPU when it starts, and ensure exclusive access to the machine during the execution of the job. However, machine learning training tasks are quite different from previous workloads, and current data centers generally lack insights into the characteristics of machine learning workloads, resulting in the current data center resource supply model for machine learning loads still being extensive.

[0006] Coarse-grained resource allocation will lead to resource waste in the data center, resulting in increased energy consumption and costs in the data center. Compared with ordinary computing resources, the cost of wasting computing resources in machine learning is much greater. At the same time, machine learning task loads usually adopt a distributed structure. If the scheduler lacks local awareness of distributed nodes, it will increase the waiting time and communication time of the job, reduce the efficiency of the cluster, slow down the completion time of the training job, and even cause the job to fail. Therefore, it is necessary to design a reasonable machine learning task scheduling model based on the characteristics of machine learning training tasks.

[0007] Currently, GPU schedulers in data centers are all based on manual configuration when users submit jobs. If they are not configured properly, training performance and resource efficiency will be greatly reduced. For example, allocating too many GPUs will cause other jobs to wait in line for GPUs for too long and result in inefficient use of each GPU, while allocating too few GPUs will result in long runtimes and inefficient training. However, it is difficult to make optimal scheduling decisions in large-scale GPU clusters because they depend on the dynamic cluster load when the job is running. Summary of the invention

[0008] In view of the above problems, the present invention manages the resource quota of each node in the job and the node placement position based on the resource allocation specified by the user, shortening the completion time of the overall machine learning model training. The specific technical solution is as follows:

[0009] One aspect of the present invention provides a method for GPU resource scheduling and allocation for distributed machine learning training tasks, comprising the following steps:

[0010] S1: The scheduling system assigns a parameter server and a worker node to each job to be assigned in the training job queue and places it in the server cluster to prevent starvation.

[0011] S2: The scheduling system tries to run each training job in the cluster with different combinations of parameter servers and worker nodes. By collecting the training speed of the job under different combinations of parameter servers and worker nodes, the training speed model S(p,w) of the training job is fitted.

[0012] S3: The scheduling system monitoring system collects the loss value of each training iteration of the training job, which is used to fit the loss value iteration curve of the job and calculate the remaining number of iterations for the training model to converge.

[0013] S4: The node allocation module uses a greedy strategy to allocate the appropriate number of working nodes and parameter servers for each job according to the training speed model S(p,w) of the training job, and adds the allocated jobs to the queue to be placed.

[0014] S5: The job placement module schedules the training jobs in the placement queue in order of the remaining completion time T from small to large. The job placement module calculates the costs of different placement strategies based on the communication delay and queue waiting time of the distributed training job, and selects the placement strategy with the lowest cost to deploy the job to the GPU cluster.

[0015] S6. Repeat steps S1 to S5 at fixed scheduling intervals.

[0016] In step S1, when there are training jobs to be scheduled in the training job queue, the scheduling system checks the available resources of the nodes according to the resource requirements attached to the training jobs submitted by the user. If there are available nodes that can allocate the required resources, a parameter server and a working node are allocated to the training job to avoid starvation caused by the job waiting for too long. The job that is temporarily allocated a parameter server and a working node is marked as a job to be adjusted.

[0017] In step S2, the training speed model of the machine learning training job is described as follows. For a training job, the number of parameter servers allocated is p, the number of working nodes is w, and the training data set is guaranteed to be evenly distributed on the w working nodes. Then the relationship between the time T required for a training step of a working node and the number of nodes allocated is as follows:

[0018]

[0019] The specific meanings of the symbols in the above formula are shown in the following table:

[0020] Table 1 List of homework training time symbols

[0021] symbol meaning T Time required for one training step on a worker node <![CDATA[t f ]]> Time for the forward propagation phase on the worker nodes <![CDATA[t b ]]> Time for the backpropagation phase on the worker nodes M The model size of the working phase is the total number of bytes of parameters B Bandwidth value of parameter server w The number of worker nodes for the job p The number of parameter servers for the job <![CDATA[t c ]]> Time required to update parameters on the parameter server <![CDATA[θ w ]]> Communication delay coefficient of working nodes <![CDATA[θ p ]]> Communication delay coefficient of parameter server

[0022] Therefore, the number of training steps per unit time using p parameter servers and w worker nodes is w / T, and the training speed of the job is modeled as follows:

[0023]

[0024] Furthermore, the step S2 comprises the following steps:

[0025] S21: The scheduling system intercepts data within 500MB from the training data set submitted by the user for the pending adjustment job in the job queue that has been assigned 1 parameter server and 1 working node.

[0026] S22: The scheduling system arranges the training job in the cluster with different combinations of parameter server number p and worker node number w to train the intercepted training data set. The scheduling system monitors the training process, ends the training when the number of training iterations reaches 10, and records the average training time under the current node allocation.

[0027] S23: Collect the average training time of training jobs under different numbers of node configurations, and use the least bisection method to fit the training speed model S(p,w) of the machine learning job.

[0028] In step S3, the scheduling system is integrated with the distributed machine learning training framework to collect the loss value and iteration number of each training iteration of the job to be adjusted, and fits the loss curve based on the loss value and the number of steps. Since most training tasks use the stochastic gradient descent method as the gradient update algorithm, based on this premise, the loss value iteration curve model is constructed as follows:

[0029]

[0030] Where l is the loss value of training, k is the number of iterations of training completed, and α, β, and γ are the non-negative coefficients of the model.

[0031] Furthermore, step S3 includes the following steps:

[0032] S31: The scheduling system monitors each training job and collects the number of completed iterations and loss values ​​after a training iteration of the job is completed. The data is then preprocessed and the points that exceed the loss value range are replaced with the average loss of the most recent steps.

[0033] S32: Use the non-negative least squares solver to calculate the loss curve model coefficients α, β, and γ that fit the current job based on the number of iterations and loss value points (k, l) collected by the scheduling system, where k is the number of iterations completed by the job and l is the loss value obtained from the current training iteration. Then calculate the number of steps required for the model to converge based on the convergence threshold predefined when the user submits. Assume that the convergence threshold given by the user is l th , then the condition that the loss function meets to converge is Based on this formula, the total number of iterations required for training convergence is:

[0034] S33: By combining the job performance model S(p,w) in step S2, the training iteration time of each working node is obtained The number of iterations that the job has completed is k, and the time required to complete the job training can be calculated as T = (k total -k) / S(p,w).

[0035] In step S4, the scheduling system uses a greedy strategy to find the optimal number of parameter server nodes p and the number of working nodes w for each training job to be adjusted. Suppose the set of training jobs to be adjusted in the current job queue is J, where the time required for training completion of job j calculated in step S3 is T j , then the algorithm optimization goal is:

[0036] min∑ j∈J T j

[0037] in At the same time, resource availability must also be considered when allocating nodes. The constraints are as follows:

[0038]

[0039] p j ∈Z + ,

[0040] Among them, the occupation of resource r by the parameter server of job j is The occupation of resource r by the working node of job j is The resource capacity constraint of the job needs to satisfy that the occupation of resource r does not exceed the total capacity C of resource r r .

[0041] This problem is a nonlinear integer programming problem and cannot be solved using a linear programming solver. Blindly seeking the optimal solution will delay scheduling decisions because the solution search space is too large. Therefore, the present invention adopts a greedy strategy to narrow the solution search space, sacrificing the scheduling effect in exchange for better scheduling efficiency.

[0042] Assume that the number of worker nodes assigned to job j is w, and the number of parameter servers assigned is p, then the remaining training completion time for the current job j is T = R j / S j (p,w), allocating an additional parameter server node or an additional GPU computing node to the job will shorten the training time of the job by ΔT, ΔT p The shortened training time for allocating 1 additional parameter server node, ΔT w The shortened training time for allocating 1 additional worker node is:

[0043]

[0044] The resource allocation benefit is the shortened training time after node allocation divided by the resource usage. p Resource allocation benefit for allocating 1 additional parameter server node, Ben wThe resource allocation benefit generated by allocating 1 additional worker node is:

[0045]

[0046] Among them Ben p Allocating revenue to parameter servers, Ben w To allocate benefits to the working nodes, the greedy strategy traverses all jobs each time and calculates the larger value of the resource allocation benefits generated by the additional parameter server nodes and working nodes allocated to each job as the resource allocation benefit Ben of training job j j , the complete calculation formula is as follows:

[0047]

[0048] Furthermore, the step S4 comprises the following steps:

[0049] S41: The scheduling system calculates the parameter server allocation revenue Ben for each training job to be adjusted p , and distribute the income to the working nodesBen w , select the larger one as the resource allocation benefit Ben = max (Ben p ,Ben w ).

[0050] S42: After the calculation is completed, the scheduling system compares the resource allocation benefits Ben of all jobs and selects the job with the largest resource allocation benefit. If the parameter server allocation benefit of the job is greater than the work node allocation benefit, that is, Ben p >Ben w , then an additional parameter server is assigned to the job, otherwise an additional worker node is assigned.

[0051] S43: Repeat steps S41 to S42 until the available resources of the cluster are insufficient to continue deploying new nodes.

[0052] S44: The scheduling system records the number of working nodes and parameter servers allocated to each job to be scheduled, marks it as a job to be deployed, and deploys it in the next step.

[0053] Furthermore, step S5 comprises the following steps:

[0054] S51: For each job to be deployed in the job queue, according to the principle of short job first, the remaining completion time T of each training job is calculated, and the jobs to be deployed are sorted from small to large according to T, and saved in a queue of jobs to be deployed.

[0055] S52: The scheduling system sorts all servers in the server cluster from largest to smallest according to the remaining available resources, and maintains a priority queue for available resources of the servers.

[0056] S53: The scheduling system selects the first job j in the queue of jobs to be deployed for placement scheduling, traverses the priority queue of available resources of the server, selects the first server s in the queue, and adds it to the deployment server set S of job j.

[0057] S54: Query whether the available resources of the deployment server set can meet the resource requirements of job j. If so, calculate the communication delay of the deployment server set S. Add deployment plan cost key-value pair Go to the deployment plan list DeploymentPlans and jump to step S56, otherwise go to step S55.

[0058] S55: Continue to traverse the queue of jobs to be deployed and select a server s that is not in the deployment server set S of job j. ′ Join the deployment server set S and jump back to step S54.

[0059] S56: Traverse other machine learning jobs j that are already running on the servers in the deployment server set S other , sort them in ascending order according to the remaining completion time, and save them to other job sets J other middle.

[0060] S57: Traverse set J other Homework other , if j other The server s occupies resources r. Compare other servers in the set S. If s′.resoure≤r,s′∈S and s′≠s, then delete s′ from the set S and record it as S′. Calculate the communication delay and homework other The remaining completion time T j , and deploy the cost key-value pair Add to the deployment plan list DeploymentPlans.

[0061] S58: Traverse all key-value pairs in the deployment plan list DeploymentPlans and select the minimum cost as the deployment plan, deploy job j to the deployment server set corresponding to the minimum cost, delete job j from the queue of jobs to be deployed, and if there are no available resources in the cluster at this time, deploy it in the next scheduling interval, otherwise jump to step S53.

[0062] Furthermore, in step S54, the communication delay of job j deployed on the server set S is The calculation is as follows:

[0063] The scheduling system attempts to transmit the parameter data generated by the training of job j between nodes of different server combinations in the deployment server set S. Each different server combination attempts to transmit 10 times and records the average transmission time. Then the communication delay of job j deployed on server set S is Where k is the number of remaining convergence iterations for job j.

[0064] Another aspect of the present invention provides a GPU resource scheduling and allocation system for distributed machine learning training jobs, including a node allocation module and a job placement module. The node allocation module considers the benefits of node allocation on job training speed improvement. The job placement module has job completion time awareness, weighs the communication delay of the computing server cluster and the queuing waiting time, so as to minimize the average completion time of the machine learning training tasks in the GPU cluster.

[0065] When a new task arrives at the scheduling system, the scheduling system first allocates 1 working node and 1 parameter server to the new task to prevent starvation. The node allocation module calculates the training speed model and selects the current largest node allocation benefit plan based on the training speed model to allocate additional nodes to the training job, thereby maximizing the node allocation benefit of the scheduling system and reducing the average completion time of all training jobs.

[0066] When the node allocation is determined, the job is transferred to the job placement module. The job placement module arranges placement decisions for the job according to the calculated remaining completion time and in the order of short jobs first. Compared with the first-in-first-out strategy, this strategy can avoid the phenomenon of long jobs being deployed first and short jobs having long waiting times, and can effectively reduce the overall job waiting time of the cluster. The job placement module calculates the communication delay and waiting time generated by placing the job in different server sets as the placement cost, and selects the placement strategy with the lowest placement cost, which balances the communication delay and queue waiting time of distributed machine learning training, and further reduces the average completion time of distributed machine training tasks.

[0067] Furthermore, the node allocation module calculates the training speed improvement after allocating an additional parameter server or an additional working node to each job, selects the job with the largest improvement in the current cluster, and allocates an additional parameter server to the job if the speed improvement of allocating an additional parameter server to the job is greater than that of allocating a working node, otherwise allocates an additional working node.

[0068] Furthermore, in the job placement module, the jobs to be deployed are sorted in ascending order according to the remaining completion time. Each time, the scheduling system selects the first job in the queue of jobs to be deployed for placement scheduling, traverses the server available resource priority queue, selects the first server in the queue, and adds it to the deployment server set S of the job; queries whether the available resources of the deployment server set can meet the resource requirements of the job. If not, continue to traverse the queue of jobs to be deployed and select server s that is not in the deployment server set S of job j. ′ Join the deployment server set S until the server set S meets the deployment resource requirements.

[0069] When the server set S meets the deployment resource requirements of job j, the communication delay Delay of the server set S is calculated S , add the deployment plan cost key-value pair {plan:S,cost:Delay S} to the deployment plan list DeploymentPlans; traverse the other machine learning jobs j that are already running on the servers in the deployment server set S according to the remaining completion time from small to large other If j other The server s occupies r resources. Compare other servers in the set S. If the resources occupied by other servers s′ are less than or equal to r, then s′ in the set S is deleted and recorded as S′. The communication delay Delay is calculated. S′ and homework other The remaining completion time T j , and deploy the cost key-value pair Add to the deployment plan list DeploymentPlans. Traverse all key-value pairs in the deployment plan list DeploymentPlans and select the deployment plan with the smallest cost. Deploy the job to the deployment server set corresponding to the smallest cost, delete the job from the queue of jobs to be deployed, and if there are no available resources in the cluster at this time, deploy it in the next scheduling interval, otherwise continue to deploy the next job.

[0070] The advantages and beneficial effects of the present invention are as follows: the node allocation module of the present invention perceives the relationship between the node allocation and training speed of the job, and adopts a greedy algorithm to select the node allocation with the greatest speed improvement benefit, thereby maximizing the speed improvement benefit of the node allocation while ensuring the scheduling efficiency. The node placement module adopts a placement order that prioritizes short jobs based on the perception of completion time, effectively reducing the average waiting time of the job. In addition, the node placement module has communication delay perception, fully considers the impact of the delay of distributed machine learning training tasks on training efficiency, and weighs it with the queuing waiting time of the job, ensuring that the cost of the placement strategy is minimized, thereby achieving the goal of reducing the average completion time of the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is the overall architecture diagram of the present invention. DETAILED DESCRIPTION

[0072] The specific implementation modes of the present invention are described in detail below with reference to the accompanying drawings.

[0073] When a user submits a new machine learning training task, the training task enters the waiting job queue and is initially assigned 1 parameter server and 1 working node. Figure 1 As shown in the figure, there are a total of n jobs to be assigned in the job queue to be assigned, and they are represented as the initial state of p:1, w:1, which means that the current job is assigned to 1 parameter server and 1 working node.

[0074] The node allocation module calculates the training speed model for the jobs in the waiting job queue, generates node allocation decisions based on the training speed model, and adds the jobs to the waiting job queue in the order of their remaining completion time. Figure 1 As shown in Figure 1, after the node allocation module generates a node allocation decision, the number of parameter servers and working nodes assigned to the jobs in the to-be-placed job queue has changed. The number of parameter servers for job 1 is 1, and the number of working nodes is 3. The number of parameter servers and working nodes for training job 2 is both 2. The job placement module generates job placement decisions for the jobs of the to-be-placed module and determines the server cluster location for job placement, as shown in Figure 1. Figure 1 As shown, training job 1 is placed in GPU server 1, and training job 2 is placed in GPU servers 1 and 2.

[0075] The specific implementation of the present invention includes the following steps:

[0076] (1) The scheduling system assigns a parameter server and a worker node to each job to be assigned in the training job queue and places it in the server cluster to prevent starvation.

[0077] (2) The scheduling system attempts to run each training job in the cluster with different combinations of parameter servers and worker nodes, and fits the machine learning training speed model S(p,w) through the training speed of the job under different combinations of parameter servers and worker nodes.

[0078] S(p,w) is

[0079] Where p is the number of parameter servers for the job; w is the number of worker nodes for the job; t f is the time of the forward propagation phase on the worker node; t b is the time of the back-propagation phase on the worker node; tc is the time required to update the parameters on the parameter server; M is the model size in the working phase, i.e., the total number of bytes of parameters; B is the bandwidth value of the parameter server; θ w is the communication delay coefficient of the working node; θ p is the communication delay coefficient of the parameter server.

[0080] Furthermore, the scheduling system intercepts data within 500MB from the training data set submitted by the user for the pending jobs in the job queue that have been assigned 1 parameter server and 1 working node. The scheduling system arranges the training jobs in the cluster with different combinations of parameter server numbers p and working node numbers w to train the intercepted training data set. Among them, in order to ensure the effectiveness of fitting the training speed model and the fitting efficiency of the model, the range of parameter server number p is set to [1,3], and the range of working node w is [1,5]. The scheduling system monitors the training process, ends the training when the number of training iterations reaches 10, and records the average training time under the current node allocation. The average training time of the training job under different number of node configurations is collected, and the minimum bisection method is used to fit the training speed model S(p,w) of the machine learning job.

[0081] (3) The scheduling system monitoring system collects the loss value of each training iteration of the training job, which is used to fit the loss value iteration curve of the job and calculate the remaining number of iterations for the training model to reach convergence.

[0082] The convergence curve model of the operation is: Where l is the loss value of the training, k is the number of iterations completed by the training job, and α, β, and γ are the non-negative coefficients of the model.

[0083] Furthermore, the scheduling system monitors each training job, and collects the number of completed iterations and loss values ​​after a training iteration of the job is completed. The data is then preprocessed, and points that exceed the loss value range are replaced with the average loss of the first two steps and the last two steps. The scheduling system uses a non-negative least squares solver based on the number of completed iterations and loss value points (k, l) collected by the scheduling system, where k is the current number of completed iterations of the job, and l is the loss value obtained in the current training iteration. Calculate the loss curve model coefficients α, β, and γ that fit the current job. Then calculate the number of steps required for the model to converge based on the convergence threshold predefined when the user submits it. By combining the job performance model S(p, w) in step S2, the training time for one step of each working node is obtained. The job has completed k iterations so far, and the time required to complete the job training can be calculated.

[0084] (4) The node allocation module uses a greedy strategy to allocate an appropriate number of working nodes and parameter servers to each job according to the training speed model S(p,w) of the training job, and adds the allocated jobs to the queue to be placed.

[0085] Furthermore, the node allocation strategy is described as follows:

[0086] The scheduling system calculates the parameter server allocation revenue for each training job to be adjusted. p , and distribute the income to the working nodesBen w , select the larger one as the resource allocation benefit Ben = max (Ben p ,Ben w ); After the calculation is completed, the scheduling system compares the resource allocation benefits Ben of all jobs and selects the job with the largest resource allocation benefit. If the parameter server allocation benefit of the job is greater than the working node allocation benefit, that is, Ben p >Ben w , an additional parameter server is allocated to the job, otherwise an additional worker node is allocated; repeat the above steps until the available resources of the cluster are insufficient to continue deploying new nodes.

[0087] (5) The job placement module schedules the training jobs in the placement queue in ascending order of the remaining completion time T. The job placement module calculates the cost of different placement strategies based on the communication delay and queue waiting time of the distributed training job, and selects the placement strategy with the lowest cost to deploy the job to the GPU cluster.

[0088] Furthermore, the job allocation strategy is described as follows:

[0089] The job placement module sorts the jobs to be deployed in ascending order of remaining completion time. The scheduling system selects the first job in the queue of jobs to be deployed for placement scheduling each time, traverses the server available resource priority queue, selects the first server in the queue, and adds it to the deployment server set S of the job; queries whether the available resources of the deployment server set can meet the resource requirements of the job. If not, continue to traverse the queue of jobs to be deployed and select server s that is not in the deployment server set S of job j. ′ Join the deployment server set S until the server set S meets the deployment resource requirements.

[0090] When the server set S meets the deployment resource requirements of job j, the communication delay Delay of the server set S is calculated S , add the deployment plan cost key-value pair {plan:S,cost:Delay S} to the deployment plan list DeploymentPlans; traverse the other machine learning jobs j that are already running on the servers in the deployment server set S according to the remaining completion time from small to large other If j other The server s occupies r resources. Compare other servers in the set S. If the resources occupied by other servers s′ are less than or equal to r, then s′ in the set S is deleted and recorded as S′. The communication delay Delay is calculated. S′ and homework other The remaining completion time T j , and deploy the cost key-value pair Add to the deployment plan list DeploymentPlans. Traverse all key-value pairs in the deployment plan list DeploymentPlans and select the deployment plan with the smallest cost. Deploy the job to the deployment server set corresponding to the smallest cost, delete the job from the queue of jobs to be deployed, and if there are no available resources in the cluster at this time, deploy it in the next scheduling interval, otherwise continue to deploy the next job.

[0091] (6) At fixed scheduling intervals, repeat steps (1) to (5).

[0092] To implement the GPU resource scheduling system proposed in the present invention, the scheduling system needs to monitor the allocation of each training job to the working node and parameter server, as well as the loss value generated by each training iteration of the job, and the number of iterations currently executed by the job. After the job is assigned to the node, the monitoring system also needs to record the training speed model S(p,w) of each job.

Claims

1. A distributed machine learning training task GPU resource scheduling and allocation method, characterized in that The method comprises the following steps: S1: The scheduling system assigns a parameter server and a worker node to each job to be assigned in the training job queue and places them in the server cluster to prevent starvation; S2: The scheduling system tries to run each training job in the cluster with different combinations of parameter servers and worker nodes. The training speed model S(p,w) of the training job is fitted by collecting the training speed of the job under different combinations of parameter servers and worker nodes. S3: The scheduling system monitoring system collects the loss value of each training iteration of the training job, which is used to fit the loss value iteration curve of the job and calculate the remaining number of iterations for the training model to reach convergence; S4: The node allocation module uses a greedy strategy to allocate an appropriate number of working nodes and parameter servers to each job according to the training speed model S(p,w) of the training job, and adds the allocated jobs to the queue to be placed; S5: The job placement module schedules the training jobs in the placement queue in the order of the remaining completion time T from small to large. The job placement module calculates the cost of different placement strategies based on the communication delay and queue waiting time of the distributed training job, and selects the placement strategy with the lowest cost to deploy the job to the GPU cluster; S6. Repeat steps S1 to S5 at fixed scheduling intervals.

2. A distributed machine learning training task GPU resource scheduling and allocation method according to claim 1, characterized in that The step S2 comprises the following steps: S21: The scheduling system intercepts data within 500MB from the training data set submitted by the user for the job to be adjusted that has been assigned a parameter server and a working node in the job queue; S22: The scheduling system arranges the training job in the cluster with different combinations of the number of parameter servers p and the number of working nodes w to train the intercepted training data set; The scheduling system monitors the training process, ends the training when the number of training iterations reaches 10, and records the average training time under the current node allocation; S23: Collect the average training time of training jobs under different numbers of node configurations, and use the least bisection method to fit the training speed model S(p,w) of the machine learning job: Where p is the number of parameter servers for the job, w is the number of worker nodes for the job, and t f is the time of the forward propagation phase on the worker node, t b is the time of the back-propagation phase on the worker node, t c is the time required to update the parameters on the parameter server, M is the model size in the working phase, i.e., the total number of bytes of parameters, B is the bandwidth value of the parameter server, and θ w is the communication delay coefficient of the working node, θ p is the communication delay coefficient of the parameter server.

3. A distributed machine learning training task GPU resource scheduling and allocation method according to claim 1, characterized in that The step S3 comprises the following steps: S31: The scheduling system monitors each training job and collects the number of completed iterations and loss values ​​after a training iteration of the job is completed; Replace points that exceed the loss value range with the average loss of the most recent steps; S32: Using a non-negative least squares solver, according to the number of completed iterations and loss value points (k, l) collected by the scheduling system, calculate the loss curve model coefficients α, β, and γ that fit the current job; where k is the number of completed iterations of the job, and l is the loss value obtained in the current training iteration; Calculate the number of steps required for the model to converge based on the convergence threshold predefined by the user during submission: Assume the convergence threshold given by the user is l th , then the condition that the loss function meets to converge is Based on this formula, the total number of iterations required for training convergence is: S33: By combining the training speed model S(p,w) in step S2, the training iteration time of each working node is obtained The number of completed iterations of the job is k, and the time required to complete the job training is calculated as T = (k total -k) / S(p,w).

4. A distributed machine learning training task GPU resource scheduling and allocation method according to claim 1, characterized in that The step S4 comprises the following steps: S41: The scheduling system calculates the parameter server allocation revenue Ben for each training job to be adjusted p , and distribute the income to the working nodesBen w , select the larger one as the resource allocation benefit Ben = max (Ben p ,Ben w ); S42: The scheduling system compares the resource allocation benefits Ben of all jobs and selects the job with the largest resource allocation benefit; if the parameter server allocation benefit of the job is greater than the work node allocation benefit, that is, Ben p >Ben w , then an additional parameter server is assigned to the job, otherwise an additional worker node is assigned; S43: Repeat steps S41 to S42 until the cluster's available resources are insufficient to continue deploying new nodes; S44: The scheduling system records the number of working nodes and parameter servers allocated to each job to be scheduled, marks it as a job to be deployed, and deploys it in the next step.

5. A distributed machine learning training task GPU resource scheduling and allocation method according to claim 1, characterized in that The step S5 comprises the following steps: S51: for each job to be deployed in the job queue, according to the principle of short job priority, the remaining completion time T of each training job is calculated, and the jobs to be deployed are sorted from small to large according to T, and stored in a queue of jobs to be deployed; S52: The scheduling system sorts all servers in the server cluster from largest to smallest according to the available remaining resources, and maintains a priority queue of available server resources; S53: The scheduling system selects the first job j in the queue of jobs to be deployed for placement scheduling, traverses the priority queue of available resources of the server, selects the first server s in the queue, and adds it to the deployment server set S of job j; S54: Query whether the available resources of the deployment server set can meet the resource requirements of job j. If so, calculate the communication delay of the deployment server set S. Save the deployment plan cost key-value pair Go to the deployment plan list DeploymentPlans and jump to step S56, otherwise go to step S55; S55: Continue to traverse the queue of jobs to be deployed and select a server s that is not in the deployment server set S of job j. ′ Join the deployment server set S and jump back to step S54; S56: Traverse other machine learning jobs j that are already running on the servers in the deployment server set S other , sort them in ascending order according to the remaining completion time, and save them to other job sets J other middle; S57: Traverse set J other Homework other , if j other The server s occupies resources r. Compare other servers in the set S. If s′.resoure≤r,s′∈S and s′≠s, then delete s′ from the set S and record it as S′. Calculate the communication delay and homework other The remaining completion time T j , and deploy the cost key-value pair Added DeploymentPlans to the deployment plan list; S58: Traverse all key-value pairs in the deployment plan list DeploymentPlans and select the minimum cost as the deployment plan, deploy job j to the deployment server set corresponding to the minimum cost, delete job j from the queue of jobs to be deployed, and if there are no available resources in the cluster at this time, deploy it in the next scheduling interval, otherwise jump to step S53.

6. A distributed machine learning training task GPU resource scheduling and allocation method according to claim 5, characterized in that: The communication delay of job j deployed on server set S The calculation is as follows: The scheduling system attempts to transmit the parameter data generated by the training of job j between nodes of different server combinations in the deployment server set S. Each different server combination attempts to transmit 10 times and records the average transmission time. s1,s2∈S and s1≠s2, then the communication delay of job j deployed on the server set S is Where k is the number of remaining convergence iterations for job j.

7. A GPU resource scheduling and allocation system for distributed machine learning training jobs, comprising a node allocation module and a job placement module, characterized in that: The node allocation module considers the benefits of node allocation on job training speed, and the job placement module has job completion time perception, weighs the communication delay of the computing server cluster and the queuing waiting time, so as to minimize the average completion time of the machine learning training task in the GPU cluster; When a new task arrives at the scheduling system, the scheduling system first allocates a working node and a parameter server to the new task to prevent starvation. The node allocation module calculates the training speed model and selects the current largest node allocation benefit scheme based on the training speed model to allocate additional nodes to the training job, thereby maximizing the node allocation benefit of the scheduling system and reducing the average completion time of all training jobs. When the node allocation is determined, the job is transferred to the job placement module, which makes placement decisions for the job in the order of shortest job first, based on the calculated remaining completion time; The job placement module calculates the communication delay and waiting time caused by placing the job to different server sets as the placement cost, and selects the placement strategy with the lowest placement cost; In the node allocation module, the training speed improvement after allocating an additional parameter server or an additional working node to each job is calculated; the job with the largest improvement in the current cluster is selected, and if the speed improvement of allocating an additional parameter server to the job is greater than that of allocating a working node, an additional parameter server is allocated to the job, otherwise an additional working node is allocated.

8. The GPU resource scheduling and allocation system for distributed machine learning training operations according to claim 7, characterized in that: In the job placement module, the jobs to be deployed are sorted in ascending order according to the remaining completion time. The scheduling system selects the first job in the queue of jobs to be deployed for placement scheduling each time, traverses the priority queue of available resources of the server, selects the first server in the queue, and adds it to the deployment server set S of the job; Query whether the available resources of the deployment server set can meet the resource requirements of the job. If not, continue to traverse the queue of jobs to be deployed and select server s that is not in the deployment server set S of job j. ′ Join the deployment server set S until the server set S meets the deployment resource requirements; When the server set S meets the deployment resource requirements of job j, the communication delay Delay of the server set S is calculated S , add the deployment plan cost key-value pair {plan:S,cost:Delay S } to the deployment plan list DeploymentPlans; traverse the other machine learning jobs j that are already running on the servers in the deployment server set S according to the remaining completion time from small to large other If j other The server s occupies r resources. Compare other servers in the set S. If the resources occupied by other servers s′ are less than or equal to r, then s′ in the set S is deleted and recorded as S′. The communication delay Delay is calculated. S′ and homework other The remaining completion time T j , and deploy the cost key-value pair Add the deployment plan list DeploymentPlans; Traverse all key-value pairs in the deployment plan list DeploymentPlans and select the deployment plan with the smallest cost. Deploy the job to the deployment server set corresponding to the smallest cost and delete the job from the queue of jobs to be deployed. If there are no available resources in the cluster at this time, deploy it in the next scheduling interval. Otherwise, continue to deploy the next job.

Images