Deep learning job priority scheduling method and deep learning job system

By using time estimate and job area scoring mechanisms in deep learning job priority scheduling, the problem of insufficient failure impact and interpretability in multi-machine scheduling in the existing technology is solved, and shorter averages are achieved in large-scale clusters Response time and higher scheduling efficiency.

CN113568725BActive Publication Date: 2024-11-22INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110794626.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-14
Publication Date
2024-11-22
Estimated Expiration
2041-07-14

AI Technical Summary

Technical Problem

The existing priority strategies cannot effectively consider the impact of cluster failures in multi-machine scheduling, resulting in the average response time being not minimal and poor interpretability, so they cannot adapt to failure situations in large-scale clusters, affecting user service quality.

Method used

The deep learning job priority scheduling method based on time estimates is used to obtain the failure probability of the GPU cluster and the remaining running time of the job without the fault, and estimate the remaining execution time of the job and the number of resources. The product is used as the work area. The job with the smallest area is selected to set the highest priority, and the priority is evaluated in combination with the heuristic algorithm.

Benefits of technology

It realizes the reduction of the average response time in large-scale clusters, improves scheduling efficiency, has good interpretability and adaptability, and can optimize the turnover time of the job while considering cluster failures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113568725B_ABST
    Figure CN113568725B_ABST
Patent Text Reader

Abstract

The present invention proposes a deep learning job priority scheduling method, including: obtaining the predicted working parameters of all available GPUs in a GPU cluster and the predicted job parameters of all jobs in the waiting queue of the GPU cluster within any job scheduling cycle; estimating the remaining execution time of each job with the predicted working parameters and the predicted job parameters; taking the product of the remaining execution time of any job and the estimated number of resources of the job as the job area of ​​the job; selecting the job with the smallest job area among all jobs and setting it to have the highest priority within the current job cycle. The present invention also proposes a deep learning job system and a data processing device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a deep learning job priority scheduling method based on time estimation and a deep learning job system. Background Art

[0002] Priority algorithms originally came from the process scheduling problem in operating systems. In early operating systems, since a single process generally only runs on a single CPU, only single-machine priority algorithms were designed. The main purpose of these priority algorithms is to avoid multiple jobs requesting CPU resources at the same time, while optimizing some common performance indicators as much as possible, such as job average response time, average turnaround time, response ratio, etc.

[0003] According to different scenarios and optimization goals, various priority algorithms have made targeted optimizations.

[0004] For example, if you want to minimize the average waiting time of all jobs, you can use the short job priority algorithm, that is, let the priority of the job decrease monotonically with the length of the job execution time. The shorter the job, the higher the priority. Mathematical derivation can be used to strictly prove that this algorithm is optimal, but from the perspective of fairness, this algorithm has obvious starvation phenomenon. In addition, there are also some classic priority algorithms as follows:

[0005] SJF (Short Job First): This is a classic algorithm for operating system task scheduling, which gives priority to tasks with short execution time. This algorithm can usually achieve good response time, but it is not suitable for GPU cluster scheduling. The main reason is that a single process generally runs only on a single CPU, while deep learning training jobs often run on multiple GPU devices.

[0006] LRF (Less Resource First): This is a widely used scheduling algorithm for resource-sensitive tasks. This algorithm only focuses on the resource usage of the job. Jobs with lower resource usage have higher priority and can be scheduled first.

[0007] RAND (Random Priority Algorithm): Random priority algorithm is also a priority strategy, that is, the next task scheduled in the queue is completely random or has a certain degree of randomness.

[0008] To solve the starvation problem, you can choose the highest response ratio priority algorithm. In this algorithm, the response ratio is defined as the waiting time of the job divided by the time required to execute, and the priority of the job increases monotonically with the response ratio. In this way, if the priority of a single job changes with the waiting time, when the waiting time is too long, the response ratio becomes very high, so it will be scheduled, thus avoiding job starvation.

[0009] If you want to execute some very real-time jobs and pay special attention to fairness, then there is a time slice rotation algorithm that allows the process to be interrupted during execution. This algorithm divides time into multiple small time slices. By assigning each time slice to a process, it can create the illusion that each process is running continuously, reducing the response time. And because the allocation of time slices is uniform for each job, this algorithm has strong fairness. In fact, this idea of ​​time slicing and using preemption can also be used in deep learning training jobs. In their work, Yujeong Choi et al. chose to implement the preemption function on the NPU. According to the urgency of each job, they chose whether to let the job in the waiting queue preempt the job being executed, thereby reducing the response time while achieving fairness.

[0010] If you simply want to reduce the complexity of the scheduling algorithm and shorten the time overhead caused by the scheduling algorithm itself, you can also use the first-come, first-served algorithm. This algorithm simply sorts the jobs according to the arrival time. The jobs that arrive earlier have a higher priority and are therefore executed first; the jobs that arrive later are the opposite. This algorithm does not require repeated sorting and insertion operations on the waiting queue, so it takes very little time.

[0011] These algorithms have good effects on their optimization goals in the case of single-machine scheduling, but because they cannot use the information about the number of CPUs that a job needs to occupy, they are no longer fully applicable in multi-machine scheduling. For multi-machine algorithms, there are more factors that need to be considered.

[0012] There are two scenarios for multi-machine scheduling, namely, each scheduled job only occupies one processor and each can occupy multiple processors. The former is a special case of the latter. But even for the former, finding a scheduling algorithm to optimize certain important indicators is NP-complete. Although the optimal problem is difficult to solve at this stage, under the condition of giving up the search for the optimal solution, there are still a considerable number of heuristic scheduling algorithms available for selection in various different demand scenarios.

[0013] There are many related works on priority. One of the more common practices is to use the backfilling method. Under certain conditions, jobs that occupy fewer resources are executed before jobs that occupy more resources with higher priority, occupying idle resources in order to improve cluster utilization. Eric Gaussier and others used backfilling in their research work, and based on various simple heuristic priority algorithms, they used a dynamic "multi-armed tiger machine" learning method to make the priority algorithm gradually converge to the one that best matches the workload characteristics, thereby reducing the average waiting time of the job. A priority scheduling strategy for a secondary queue is proposed in the prior art. Specifically, the scores of all nodes in the cluster are calculated according to fixed rules to generate a first node priority queue, and a Pod priority queue is obtained by a dynamic priority algorithm. The two queues filter the nodes that cannot be scheduled to generate a second node priority queue, and select the node with the highest priority from them to bind to the Pod popped out of the Pod priority queue. If the binding is successful, the next Pod scheduling cycle will be entered. If it fails, the built-in priority algorithm will be used to select the preferred Node from the second node priority queue. If it fails again, there is no suitable Node for the Pod to run, and the next Pod scheduling cycle will be entered. The invention [ ] uses a neural network to obtain the weights of various indicators of the jobs in the cluster in a black box manner, and then prioritizes them according to the weights.

[0014] Other algorithms include the Gang Scheduling method, which is designed to imitate the time slice round-robin algorithm under the single-machine state. Like the time slice round-robin algorithm, the Gang Scheduling method divides time into smaller time slices, performs each operation on each processor separately, and then allocates a certain number of processors to each job in a certain time slice according to its needs. Like the time slice round-robin algorithm, this algorithm can also improve fairness and reduce response time.

[0015] There are two main ways to deal with errors: reducing error overhead and reducing the probability of errors.

[0016] At the software level, the most common method to reduce error overhead is to use Checkpoint technology, that is, insert savepoints periodically during job execution. When an error occurs in the job, it can be restarted from the savepoint, thus avoiding the job from the beginning to re-execute and waste a lot of time. However, the act of saving Checkpoints itself will cause a high time overhead, so it is necessary to choose an appropriate frequency for saving. Two examples in related work are as follows. Han Li et al. examined in detail the data transmission path between multiple machine nodes during job execution, and only saved Checkpoints when data needed to be transmitted across nodes, ensuring that each node only needed to re-execute the content executed on the node after an error occurred, without having to re-execute the content on other non-error nodes. This method reduces the frequency of saving Checkpoints while avoiding high error overhead caused by too few Checkpoints. Ismail Akturk et al. proposed that the value of saving Checkpoints should be weighed against the time overhead. For some Checkpoints that have a large time overhead, they should choose not to save them, and use a small part of the program to re-acquire them after an error to avoid a lot of access overhead.

[0017] There are also many related works to reduce the probability of errors. It is particularly worth mentioning that Ajeya Naithani et al. chose to use scheduling methods to alleviate the error problem, hoping to reduce the frequency of errors. Through experiments, they found that in heterogeneous processor architectures, different jobs have different error probabilities when executed on different types of processors. In order to reduce the average number of errors, jobs executed on different types of processors can be continuously exchanged to find a placement method that has the least frequent errors.

[0018] The existing priority strategies have the following problems: 1) They do not consider the impact of cluster failures and only schedule the cluster in a fault-free state; 2) They are unable to minimize the average response time of jobs in the cluster. The average response time is an important indicator of user service quality, which reflects the time interval from when a user submits a job to when he receives the result; 3) Some existing priority strategies use machine learning algorithms to assist in determining the priority of jobs, and the interpretability of such priority algorithms is poor. Summary of the invention

[0019] In view of the above problems, the present invention proposes a deep learning job priority scheduling method based on time estimation, including: within any job scheduling cycle, obtaining the predicted working parameters of all available GPUs in the GPU cluster and the predicted job parameters of all jobs in the waiting queue of the GPU cluster; estimating the remaining execution time of each job with the predicted working parameters and the predicted job parameters; taking the product of the remaining execution time of any job and the estimated number of resources of the job as the job area of ​​the job; selecting the job with the smallest job area among all jobs, and setting it to have the highest priority in the current job cycle.

[0020] The deep learning job priority scheduling method of the present invention, wherein the predicted working parameters include: a failure parameter λ reflecting the average failure probability p of all available GPUs, λ satisfies p=1-e -λ , and the average recovery time after GPU failure δ; the predicted job parameters include: the theoretical remaining running time T of each job without failure, the estimated number of resources n, the job scheduling cycle τ; the remaining execution time of any job Among them, τ0 is the estimated execution time of the job within a job scheduling cycle τ, p' is the probability of failure-free operation within a job scheduling cycle τ. Within a job scheduling cycle τ, p' = e -nλτ .

[0021] The deep learning job priority scheduling method described in the present invention executes all deep learning jobs in at least one job scheduling cycle.

[0022] The present invention also proposes a deep learning job system, including: a parameter acquisition module, used to obtain the predicted working parameters of all available GPUs in the GPU cluster and the predicted job parameters of all jobs in the waiting queue of the GPU cluster within any job scheduling cycle; and use the predicted working parameters and the predicted job parameters to estimate the remaining execution time of each job; a priority scheduling module, used to set the priority of the current job cycle according to the job area of ​​the job; wherein the product of the remaining execution time of any job and the estimated number of resources of the job is used as the job area of ​​the job, and the job with the smallest job area among all jobs is selected and set to have the highest priority within the current job cycle.

[0023] The deep learning operation system of the present invention, wherein the predicted operation parameters include: a failure parameter λ reflecting the average failure probability p of all available GPUs, λ satisfies p=1-e -λ , and the average recovery time after GPU failure δ; the predicted job parameters include: the theoretical remaining running time T of each job without failure, the estimated number of resources n, the job scheduling cycle τ; the remaining execution time of any job Among them, τ0 is the estimated execution time of the job within a job scheduling cycle τ, p' is the probability of failure-free operation within a job scheduling cycle τ. Within a job scheduling cycle τ, p' = e -nλτ .

[0024] The deep learning operation system described in the present invention executes all deep learning operations in at least one operation scheduling cycle.

[0025] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that when the computer-executable instructions are executed by a processor, the deep learning job priority scheduling method as described above is implemented.

[0026] The present invention also proposes a data processing device, comprising: a GPU cluster; a processor; and a computer-readable storage medium. When the processor retrieves and executes computer-executable instructions in the computer-readable storage medium, the deep learning job executed on the GPU cluster is scheduled. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart for obtaining the job priority strategy in the present invention.

[0028] Figure 2 This is a diagram of the estimated job running time without checkpoints.

[0029] Figure 3 This is a diagram showing the estimated running time of a job with checkpoints.

[0030] Figure 4 It is a schematic diagram of a data processing device of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation method described herein is only used to explain the present invention and is not used to limit the present invention.

[0032] When the inventors were conducting research on large-scale distributed deep learning job priority strategies, they found that it was unreasonable to use the short job priority strategy used in traditional CPU clusters directly in GPU clusters. The main reason is that a single process generally only runs on a single CPU, while deep learning training jobs often run on multiple GPU devices. Therefore, the inventors proved through analysis and rigorous mathematical derivation that the strategy of setting the highest priority for the job with the smallest product of job running time and the number of GPU resources required for the job can obtain the best average turnaround time, and the product of the job running time and the number of GPU resources required for the job is defined as the job area. Therefore, the present invention adopts the small area priority strategy (SAF).

[0033] In addition, the simple small area priority strategy does not take into account that the remaining running time of the job may change greatly due to cluster failure, resulting in inaccurate calculation of the job area and affecting the scheduling effect. After a large number of experiments and mathematical modeling studies, the inventor found that the solution to this defect can be improved by first estimating the remaining execution time of the training job based on the cluster failure situation. Therefore, a deep learning job priority scheduling method based on time estimation is proposed. The difficulty lies in how to estimate the actual running time under cluster failure interference based on the theoretical execution time.

[0034] The purpose of the present invention is to solve the problem that the existing cluster deep learning job priority strategy cannot guarantee the optimal average turnaround time from two aspects. On the one hand, strict mathematical proof can deduce that the existing priority strategy cannot minimize the average turnaround time in the case of large clusters and multiple jobs, while the small area strategy of the present invention can optimize the average turnaround time. On the other hand, the failure of large clusters will increase with the increase of cluster size, and the priority strategy that does not consider cluster failures cannot adapt to large-scale clusters. Therefore, the present invention proposes a technology for estimating the remaining time of deep learning training jobs based on cluster failure information, and applies it to the small area priority priority strategy, which can minimize the average response time of jobs in the cluster.

[0035] The scheduling method of the present invention uses cluster fault information to estimate the remaining execution time of deep learning training jobs, and can estimate the job execution time under cluster fault interference based on the job theoretical execution time; the method adopts a small area first (SAF) scheduling strategy, which can reduce the average response time of deep learning training jobs in the cluster; and a heuristic algorithm is used in the priority acquisition process, which has complete explainability.

[0036] Compared with SJF (Shortest Job First Algorithm), FCFS (First In First Out Algorithm), LRF (Low Resource First Algorithm), RAND (Random Priority Algorithm) and other algorithms, the present invention has the best average turnaround time. Theoretical proof is as follows:

[0037] First, define two constraints:

[0038] Constraint 1: The number of cards in the cluster is much larger than the number of cards required by a single job. Under this condition, it can be asserted that as long as there are jobs in the job queue, the cluster utilization can be maintained at a high level close to 1. The following lemma holds:

[0039] Lemma 1: Let the total number of GPUs in the cluster be n cluster , and there is another long enough work queue {job i ,i>0}, where job i The number of GPUs occupied is n jobi , remember n max :=maxn jobi . At any time, if there are still jobs in the work queue, the utilization U of the GPU in the cluster is:

[0040]

[0041] In particular, if n cluster >>maxn jobi , then 1-U<<1.

[0042] Proof: Without loss of generality, assume that the priority of the job in the work queue decreases with the job index. Now schedule the jobs in the queue to the cluster one by one according to their priority until the highest priority job remaining in the work queue cannot be scheduled. Using the proof by contradiction, assume

[0043]

[0044] Then let the current job with the highest priority be indexed as k, there should be

[0045] n jobk ≤n ma x≤n cluster -n occupied

[0046] Therefore, the number of idle GPUs is not less than the number of GPUs required by job k, and the job can be scheduled. This contradicts the scheduling stop. Therefore, the original proposition holds.

[0047] If cluster >>n max , then 1-U<n max / n cluster <<1.

[0048] In fact, cluster >>n maxThis constraint is generally true in large-scale clusters. Therefore, it can be guaranteed that when there are enough jobs, almost all GPU cards in the cluster are occupied or in error state, and rarely idle due to competition.

[0049] Constraint 2: The total execution time of all jobs is much longer than the execution time of a single job. This condition is essentially similar to the previous condition. Let the total execution time of k jobs with adjacent priorities starting from the mth job in the execution queue be t m,k , the time from the first job to the last job is t k , and job i The execution time is t jobi . Remember max :=maxt jobi . Number of jobs

[0050] t m,k >>t max

[0051] Then we can assert that when executing this batch of jobs, there are at least t m,k -2t max The time can ensure that the proportion of GPUs in the cluster occupied by these k jobs satisfies the inequality U obeys in Lemma 1. Furthermore, it can ensure that the cluster m,k The average occupancy rate of these k jobs in the time length is within a certain range. Suppose that after scheduling using a certain priority algorithm, each job i Respectively in b i The execution is scheduled to start at the time e i The execution is completed at the moment. Then we have the following Lemma 2.

[0052] Lemma 2: Let the total execution time of k jobs with adjacent priorities starting from the mth job in the execution queue be t m,k , then the average utilization

[0053]

[0054] There should be an inequality

[0055]

[0056] Proof: Since we are discussing k jobs with adjacent priorities, we can guarantee that m Before time b, the first m-1 jobs have been scheduled. m +t max After that, the first m-1 jobs must have been completed. m+k-1 -t maxBefore, the jobs after m+k must not have been scheduled. m +t max ,e m+k-1 -t max ], there must be only these k jobs running on the cluster.

[0057] According to Lemma 1, the utilization rate of the cluster is higher than 1-n in this time period. max / n cluster Therefore, during this period, the sum of the areas of these operations should satisfy

[0058]

[0059] thereby

[0060]

[0061] The lemma holds.

[0062] So far, it has been proved that under the two constraints, the cluster can maintain high utilization during the execution of any consecutive fragments in the work queue.

[0063] The following proves that the scheduling method of the present invention is optimal in terms of average turnaround time:

[0064] This proof can be transformed into finding an approximate optimal algorithm under constraints. To find an approximate optimal algorithm, we need to find several properties that the optimal algorithm satisfies. The following will explain the conditions that the approximate optimal priority algorithm needs to satisfy under the above two constraints. Here, we can make a simplified assumption that all jobs arrive at the initial time, and the information of all jobs can be obtained at the initial time. For the situation where jobs arrive at different times, the priority algorithm can be used to set the priority of each job at the time of arrival.

[0065] Now assume that the optimal priority algorithm has been found. Under this algorithm, each job i Respectively in b i The execution is scheduled to start at the time e i The time taken by the algorithm from scheduling the first job to completing the execution of all jobs is t all Now we divide all jobs into several sets according to the time when they start executing.

[0066]

[0067] First, estimate the size relationship between the number of jobs in each set. By considering two of these sets and swapping the priorities of the two sets as a whole, we can derive the following lemma:

[0068] Lemma 3: Under the above notation, for k1<k2<N, let The number of elements in is There should be

[0069]

[0070] Proof: After the above division, N mutually disjoint sets are obtained. Select the k1th and k2th sets, where the k1th set contains the priority from arrive The k2th set contains the jobs with priorities from arrive Assume that under this optimal priority algorithm, the average turnaround time of all jobs is l0.

[0071] Now swap the priorities of these two sets as a whole, that is, move the priorities of all jobs in the k1th set back to the k2th set. to and move all jobs in the k2th set forward to the to Now, estimate how the average turnaround time for all jobs changes after re-prioritization.

[0072] Now consider a situation where for each job in the k2th set, the start time of the job is advanced by an equal value For each job in the k1+1th to k2-1th sets, the start time of the job is delayed by 2t. max ; For the k1th set, the start time of each job in it is delayed For the sets after k2+1, the job start time is delayed by 4t max Since all jobs in each part are delayed or advanced by the same length of time, there is no conflict between jobs in each part. According to the above time change value, the earliest start time of the job in the k2th set is t all (k1-1) / N+t max , so it must be later than the end time of the jobs in the k1-1th and previous sets, and these two parts of the jobs will not conflict. Similarly, it can be calculated that all the above jobs will not conflict. This shows that the above situation is a feasible scheduling scheme. Under this scheduling scheme, the average turnaround time can be calculated as

[0073]

[0074] Also because

[0075]

[0076] So there is

[0077]

[0078] In this case, the order of all job start times is the same as the adjusted priority order, and there is no conflict. Therefore, according to the adjusted priority order, each job has the opportunity to be scheduled at an earlier time. In other words, according to the adjusted priority order, the average turnaround time of all jobs must not be longer than l1. However, according to the initial assumption, l0 is the average turnaround time obtained under the optimal priority algorithm, so it must be shorter than l1, so it can be guaranteed

[0079]

[0080] This is equivalent to the expression to be proved by the lemma.

[0081] According to the above proof, the number of jobs in the later set will not be much more than that in the earlier set. And when the number of jobs is large enough to meet the second constraint, the number of jobs in the later set will not be greater than that in the earlier set.

[0082] At the same time, it can be found that the time spent from the start of the first job to the end of the last job in each set is t all / N to t all / N+t max , which is almost unchanged for each set. According to Lemma 2, we can infer that the total area of ​​the jobs contained in each set is similar. Since the number of jobs in the later sets is less than that in the earlier sets, the average area occupied by each job must be larger.

[0083] But this is not enough to explain the problem. This is because if most of the area in a set is occupied by a job that occupies a particularly large area, then even if most of the jobs in the set are very small, the average area occupied by each job may be larger than that of other sets. To find a superior algorithm, it is also necessary to study the distribution of various types of jobs with different occupied areas in each set.

[0084] Therefore, according to the above three lemmas, if we consider exchanging the jobs with smaller occupied areas in the back set with the jobs with larger occupied areas in the front set, we can draw the following conclusions:

[0085] Corollary 1: If there is a job queue {job i , i>0}, and job i The execution time is t jobi , the number of GPU cards occupied is n jobi, Note Ajobi =n jobi ·t jobi The jobs are scheduled on the cluster according to the optimal priority algorithm, and the time from scheduling the first job to completing all jobs is t all , and for each job in the queue i , which is in b i The time is scheduled to start execution. Then let

[0086]

[0087] In

[0088]

[0089] Under the conditions, there should be the following size relationship:

[0090]

[0091] Proof: Assume I1={i k ,k=1,2,...,|I1|},I2={j k , k=1,2,...,|I2|}, then define the priority representation of "exchanging" these two sets. For each k greater than 0 and less than |I1|, the priorities of the ikth and jkth jobs are exchanged, and at the same time, the ith k Move the jth job from I1 to I2. k The job is moved from I2 to I1. Then the inference means that two of the above sets are selected, namely, the k1th and k2th ones. and Select two subsets I1 and I2 with the same number of elements, try to exchange the priorities of the two subsets, and check whether the average turnaround time is improved.

[0092] First, define the change amount as

[0093]

[0094] Then according to the definition in the lemma,

[0095]

[0096] Now consider only the case where the area occupied by each job in I1 is not larger than that of any job in I2. If this condition is not met, then we can delete a number of elements from small to large in I1, and delete an equal number of elements from large to small in I2 until the condition is met. At this time, the two sets become I1′ and I2′ respectively. In this process, the sum of the areas occupied by the jobs deleted in I1 is smaller than that of the jobs deleted in I2, so according to the previous definition of the change amount, we should have Therefore, if it can be proved As stated in the lemma, there is an upper bound, then Also has the same upper bound.

[0097] Since the number of jobs in the two subsets is the same, after adjusting the priorities, and In all other sets except the two sets, the priority of the job must not change. Then, as in Lemma 3, all sets are divided into five parts for consideration, namely Analyze the changes in their turnaround times.

[0098] right For example, jobs with larger occupied areas are replaced by jobs with the same number of smaller occupied areas, so the execution start time of each job must be advanced. Then the turnaround time of these jobs is shortened by at least a non-negative value.

[0099] right For all jobs, the priority of these jobs is higher than that of the adjusted jobs, so the priority adjustment has no effect on the time when they start to execute. Therefore, the change in the turnaround time of these jobs is 0.

[0100] right The only factors that will affect the start time of this part of the job are and in The start and end times of all jobs in the same process do not change, so we need to examine The moment when all jobs in are completed. It can be noticed that as smaller jobs move in and larger jobs move out, The sum of the areas occupied by all operations in Before adjustment, The sum of the areas occupied by all jobs in the cluster does not exceed the capacity n that the cluster can bear. cluster ·(t max +t all / N), then according to Lemma 2, after adjustment, The time from when the first job starts to when the last job is completed

[0101]

[0102] By definition, before the priority adjustment, The minimum time between the start of the first job and the end of the last job is tall / N. Therefore, after the priority adjustment, this time period is at most extended.

[0103]

[0104] In addition, due to The execution start and end times of the jobs in the The start time of the first job in the max Such a long execution time, therefore, its end time is at most delayed

[0105]

[0106] So now if we By delaying the start time of each job by Δe, we can ensure that these jobs are There will be no conflict between the jobs in the task list. That is, each job can be scheduled at an earlier time. That is, the start time of each of these jobs is delayed by at most Δe, which is equivalent to at least -Δe in advance, and the turnaround time is also shortened by at least -Δe.

[0107] right For example, consider the end time of the last job, and record this time as Also note The first job starts at Then we can know that All jobs in these collections are in arrive Then according to Lemma 2, we have

[0108]

[0109] And there are

[0110]

[0111] Therefore, the time when the last job ended can be obtained

[0112]

[0113] Then because exist All the operations have been completed before, so the start time of the left and right jobs in this set must be no later than And by According to the definition, before the priority is adjusted, the start time of the jobs in this set must not be earlier than (k2-1)t all / N, then it can be deduced that the start time of each job in the set is adjusted by priority, and the length of time delayed

[0114]

[0115] Then the turnaround time is increased by at most this value, i.e. it is decreased by the opposite of this value.

[0116] right For example, the first job must start executing no later than Before the job priority is adjusted, the first job must start executing no earlier than k2tall / N. Therefore, after the job priority is adjusted, each job is delayed by at most

[0117]

[0118] Then, if the average turnaround time of all jobs before the priority adjustment is l0, and the average turnaround time of all jobs after the adjustment is l2, according to the changes in the average turnaround time of jobs in the above five parts, we should have

[0119]

[0120] in

[0121]

[0122] Since the original algorithm is the optimal algorithm, it can be seen that after adjusting the priority, the average turnaround time of the new algorithm must be longer than that of the original algorithm. Therefore, the left side of the inequality sign in the above expression (2) must not be less than 0. According to (3), it can be seen that

[0123]

[0124] Arrange the formula, which is

[0125]

[0126] According to Lemma 3, for every i between k1+1 and k2-1, Should meet the conditions

[0127]

[0128] thereby

[0129]

[0130] Substituting this value back into expression (4), we can get

[0131]

[0132] Next, using the two constraints proposed above, we can reorganize expression (5) in a small amount. According to the definition of δ' in expression (3), we know that

[0133]

[0134] Combined with the above expression (5), we should have

[0135]

[0136] For any |I1|=|I2|, so according to expression (1), we know

[0137]

[0138] And this is exactly the result that the inference wants to prove.

[0139] To show that this conclusion is more meaningful than Lemma 3, we can still choose the example mentioned after proving Lemma 3, that is, for k2>k1, It consists of a small number of operations that occupy a large area and a few operations that occupy a small area. In this case, since most of the area is occupied by the large operations, The number of jobs in may not be large, so it cannot be restricted by Lemma 3. However, in Corollary 1, I2 is chosen to be The subset of the smallest operations that accounts for half of the total number of The subset of equal tasks with the largest area in the . Then due to There are a lot of operations that occupy very small areas, so the areas of the operations in I2 are all very small, and the sum of these areas is also very small; while the areas of the operations in I1 are more evenly distributed, so the sum of the areas of these operations is larger. This will directly violate the requirements of Corollary 1. Therefore, Corollary 1 is a stronger constraint than Lemma 3.

[0140] Of particular note is n cluster ·t all / N is approximately equal to each The sum of the areas of all operations in is equal. This means that the above inference cannot directly explain It is a small amount relative to the total area, and even in the case of k2-k1=1, this corollary cannot give any limit. However, if there is a long enough work queue, since N can be very large under the constraints of this chapter, the difference in the set rank of two specific jobs, that is, k2-k1, can also become very large. In this case, the right-hand side of the inequality is relative to a single The sum of the areas of all operations in is a small quantity, and the inferred constraints become very strong.

[0141] If there are a limited number of types of jobs, the area occupied by a single job must also be limited. Therefore, the absolute value of the difference between the areas occupied by two jobs must have a minimum value, let this minimum value be ΔA. The area occupied by a single job must also have a maximum value, let this maximum value be A. max Then there are two moments t1 and t2, satisfying t2>t1 and t2-t1=O(t all ), we can conclude that the area occupied by almost all jobs near time t1 is smaller than that near time t2. Consider dividing t1 to t2 into three large time periods: [t1, (2t1+t2) / 3), [(2t1+t2) / 3, (t1+2t2) / 3), [(t1+2t2) / 3, t2), and divide each time period into small time periods. The set of jobs whose start execution time falls within each small time period is Then let the set numbers of the two jobs in the first and third large time periods be k1 and k2, and the difference between the two numbers is at least N0. According to Corollary 1

[0142]

[0143] Assume that There are n jobs all than The n jobs in are large, and There are m jobs in total, so there should be

[0144]

[0145] According to Lemma 2

[0146]

[0147] Substituting back the above expression, we have

[0148]

[0149] When the above two constraints are met, N0 can be large enough, so n / m is very small. This means that the number of jobs that are in the reverse order of the small area first algorithm is very small compared to the total number of jobs, which also shows the optimality of the small area first algorithm.

[0150] The time estimation-based deep learning job priority scheduling method of the present invention can be used for single queue scheduling or multi-queue scheduling. Figure 1 As shown, when a single queue is scheduled, the specific implementation method of priority acquisition is as follows:

[0151] S1: Get the average failure probability p of all GPUs or all available GPUs in the cluster, satisfying p = 1-e -λ , and the average recovery time δ after GPU failure.

[0152] S2: Get information about all jobs (job numbers are 0, 1, 2, ... jobs) in the cluster waiting queue, including the theoretical remaining running time T, the number of GPU resources required n, and the checkpoint saving interval τ of each job.

[0153] S3: For each job in the cluster, estimate its remaining execution time T0.

[0154] S4: According to the calculation result of step S3, the remaining execution time of all jobs in the cluster waiting queue is updated.

[0155] S5: Calculate the area of ​​all jobs in the cluster waiting queue. The area of ​​the i-th job is recorded as S i , and added to the job information data.

[0156] S6: Find the jth job such that S j <S k Always holds, where k∈0,1,2,...,jobs. The jth job has the highest priority and is the next scheduled job.

[0157] The step S3 specifically includes:

[0158] S31: When each contains only one epoch, such as Figure 2 As shown in the figure, an error restart may occur, but the checkpoint is not saved, and the estimated execution time of the job on the cluster

[0159] S32: Based on τ0 obtained in S31, estimate Figure 3 The execution time when the job saves checkpoints every τ is shown in

[0160] The step S31 specifically includes:

[0161] S311: Define the average probability of GPU errors per second in the cluster as p, then the average probability of no errors is 1-p. Suppose job J is assigned to n GPU cards, then the probability of no errors on these n cards per second is (1-p) n , then the probability that all machines remain error-free after t seconds is (1-p) nt , from p = 1-e -λ Get e -λ =1-p, then the probability that the job is executed on n GPU cards without error is e -nλt .

[0162] S312: Define P(success) as the probability that the job is completed without error from the beginning to the end, P(success) = e -nλt ; Define the probability of a job going wrong from time t to time t+dt as f(t)·dt.

[0163] S313: Using the data obtained in step S1, step S2, and step S312, the expected total time to complete the job is calculated as

[0164] S314: According to the definition of step S312, P(success)=e -nλT and f(t) = nλe -nλt Substituting the total time expectation of step S313 into Replace T with τ and rearrange to get

[0165] The present invention also provides a computer-readable storage medium and a data processing device, such as Figure 4 As shown, it includes a GPU cluster, a processor and a computer-readable storage medium. The computer-readable storage medium of the present invention stores computer-executable instructions. When the computer-executable instructions are executed by the processor, the priority of the deep learning job executed on the GPU cluster is set to schedule the deep learning job. A person of ordinary skill in the art can understand that all or part of the steps in the above method can be completed by a program to instruct related hardware (such as a processor, FPGA, ASIC, etc.), and the program can be stored in a readable storage medium, such as a read-only memory, a disk or an optical disk. All or part of the steps of the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, such as implementing its corresponding functions through an integrated circuit, or in the form of a software function module, such as implementing its corresponding functions by executing a program / instruction stored in a memory by a processor. The embodiment of the present invention is not limited to any particular form of combination of hardware and software.

[0166] The present invention uses a heuristic algorithm to estimate the remaining execution time of deep learning jobs, and then uses the product of the estimated time and the number of resources required for the job as the area, and finally uses a small area job priority strategy that has been mathematically proven to optimize the average waiting time for priority assessment. Compared with the prior art, the priority strategy of the present invention has the following advantages: 1) Considering that when the number of nodes and GPU devices in the cluster reaches a certain level, the probability of cluster failure increases, and the impact of its failure cannot be ignored; 2) A shorter average response time can be obtained; 3) A pure heuristic algorithm is used, which has complete interpretability.

[0167] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The scope of protection of the present invention should be defined by the claims.

Claims

1. A deep learning job priority scheduling method, It is characterized in that include: In any job scheduling cycle, the predicted working parameters of all available GPUs in the GPU cluster and the predicted working parameters of all jobs in the waiting queue of the GPU cluster are obtained; the predicted working parameters include: a failure parameter λ reflecting the average failure probability p of all available GPUs, λ satisfies p = 1-e -λ , and the average recovery time after GPU failure δ; the predicted job parameters include: the theoretical remaining running time T of each job without failure, the estimated number of resources n, and the job scheduling period τ; The remaining execution time of each job is estimated based on the predicted work parameters and the predicted job parameters; the remaining execution time of any job Among them, τ 0 is the estimated execution time of the job within a job scheduling cycle τ, p' is the failure-free probability of the job within a job scheduling cycle τ, p' = e -nλτ ; The product of the remaining execution time of any job and the estimated number of resources for the job is taken as the job area of ​​the job; The job with the smallest job area among all jobs is selected and set to have the highest priority in the current job cycle.

2. The deep learning job priority scheduling method according to claim 1, It is characterized in that Execute all deep learning jobs with at least one job scheduling cycle.

3. A deep learning operation system, It is characterized in that include: The parameter acquisition module is used to obtain the predicted working parameters of all available GPUs in the GPU cluster and the predicted job parameters of all jobs in the waiting queue of the GPU cluster within any job scheduling cycle; the predicted working parameters include: a failure parameter λ reflecting the average failure probability p of all available GPUs, λ satisfies p=1-e -λ , and the average recovery time after GPU failure δ; the predicted job parameters include: the theoretical remaining running time T of each job without failure, the estimated number of resources n, and the job scheduling cycle τ; and the remaining execution time of each job is estimated using the predicted work parameters and the predicted job parameters; the remaining execution time of any job Among them, τ 0 is the estimated execution time of the job within a job scheduling cycle τ, p' is the failure-free probability of the job within a job scheduling cycle τ, p' = e -nλτ ; The priority scheduling module is used to set the priority of the current job cycle according to the job area of ​​the job; wherein the product of the remaining execution time of any job and the estimated number of resources of the job is used as the job area of ​​the job, and the job with the smallest job area among all jobs is selected to set the highest priority in the current job cycle.

4. The deep learning operation system according to claim 3, It is characterized in that Execute all deep learning jobs with at least one job scheduling cycle.

5. A computer-readable storage medium storing computer-executable instructions, It is characterized in that When the computer executable instruction is executed by the processor, the deep learning job priority scheduling method as described in claim 1 or 2 is implemented.

6. A data processing device, include: GPU clusters; processor; The computer-readable storage medium as described in claim 5, when the processor calls and executes the computer-executable instructions in the computer-readable storage medium, the deep learning job executed on the GPU cluster is scheduled.