Multi-DAG scheduling method and device, computer equipment and readable storage medium

Abstract

The invention relates to a multi-DAG scheduling method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring the latest execution starting time of a plurality of DAGs corresponding to a job; the latest execution starting time is the latest time at which the DAG is allowed to start execution on the premise that the deadline constraint of the DAG is met; the method comprises the following steps: firstly, sequentially scheduling a plurality of DAGs from small to large according to the latest execution starting time so as to guide a sequential scheduling process, sequentially scheduling the plurality of DAGs according to the sequence on the basis of a strategy of discarding the DAGs failed in scheduling and a backfilling algorithm in the sequential scheduling process, and scheduling tasks by utilizing gaps by other ready DAGs on the premise of not influencing the scheduling of the previous DAGs, so that the scheduling efficiency of the DAGs is improved. The resource utilization rate is further improved by discarding the DAG failed in scheduling to make a gap for the subsequent DAG for scheduling, and the resource utilization rate and the DAG scheduling success rate are further improved.

Description

Technical Field

[0001] The present application relates to the technical field of DAG scheduling in a distributed system, and in particular to a multi-DAG scheduling method, apparatus, computer equipment, and readable storage medium. Background Art

[0002] With the development of cloud computing, the scheduling problem of multiple DAGs (Directed Acyclic Graphs) from different users sharing a set of distributed resources has attracted widespread attention.

[0003] Since there are execution order constraints and data dependencies between single DAG tasks, resource gaps will inevitably occur during the scheduling process. Other DAGs can use these resource gaps, which is the key to improving resource utilization.

[0004] Currently, the resource utilization of multi-DAG scheduling in cloud environments still needs to be further improved. Summary of the invention

[0005] Based on this, it is necessary to provide a multi-DAG scheduling method, device, computer equipment, and readable storage medium that can optimize resource utilization and scheduling success rate in response to the above technical problems.

[0006] In a first aspect, the present application provides a multi-DAG scheduling method, which is applied to a distributed system, and the method includes:

[0007] Obtain the latest execution start time of multiple DAGs corresponding to the job; the latest execution start time is the latest time that the DAG is allowed to start executing under the premise of satisfying the deadline constraint of the DAG;

[0008] Sort the DAGs in ascending order according to the latest execution start time;

[0009] Based on the strategy of discarding the DAG that fails to be scheduled and the backfill algorithm, multiple DAGs are scheduled sequentially according to the sorting.

[0010] In one embodiment, obtaining the latest start execution time of each DAG includes:

[0011] According to the sub-deadline time of each DAG and the execution time of the entry task, as well as the following expression, the latest start execution time LST of each DAG is determined:

[0012]

[0013] in, is the sub-deadline time, is the execution time of the entry task.

[0014] In one embodiment, before the step of determining the latest start execution time LST of each DAG according to the sub-deadline time of each DAG and the execution time of the entry task and the following expression, the step further includes:

[0015] Relax the sub-deadlines of all tasks in the task set.

[0016] In one embodiment, the multiple DAGs are stored in a DAG list in a list form, and the strategy of discarding the DAG that fails to be scheduled and the backfill algorithm are used to sequentially schedule the multiple DAGs according to the sorting, including:

[0017] Iterate the following steps until the DAG list is empty:

[0018] Get the head DAG in the DAG list;

[0019] Save the current processor load as a copy, and calculate the load on the [ST, DL] interval; where ST is the submission time of the task set, and DL is the deadline of the task set;

[0020] Using the ICSTS algorithm to schedule the task set corresponding to the head DAG in the interval, so that the tasks in the task set are scheduled using the gaps in the interval;

[0021] If the task set is successfully scheduled, updating the processor load according to the scheduling result of the task set;

[0022] The head DAG is removed from the DAG list to update the DAG list.

[0023] In one embodiment, the iterative execution of the following steps, until the DAG list is empty, further includes:

[0024] If the task set scheduling fails, the head DAG is discarded, and the load condition of the processor is restored to the load condition before the task set scheduling using the copy.

[0025] In a second aspect, a multi-DAG scheduling device is provided, which is applied to a distributed system, and the device includes:

[0026] The latest execution start time acquisition module is used to acquire the latest execution start time of multiple DAGs corresponding to the job; the latest execution start time is the latest time that the DAG is allowed to start executing under the premise of satisfying the deadline constraint of the DAG;

[0027] A sorting module, used to sort the DAGs in ascending order according to the latest execution start time;

[0028] The scheduling execution module is used to schedule multiple DAGs in order according to the sorting based on the strategy of discarding the DAG that fails to be scheduled and the backfill algorithm.

[0029] In one embodiment, the latest execution start time acquisition module includes:

[0030] The LST determination unit is used to determine the latest start execution time LST of each DAG according to the sub-deadline time of each DAG and the execution time of the entry task, as well as the following expression:

[0031]

[0032] in, is the sub-deadline time, is the execution time of the entry task.

[0033] In one embodiment, multi-DAG scheduling further includes:

[0034] The sub-deadline relaxation module is used to relax the sub-deadline of all tasks in the task set.

[0035] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0037] The above-mentioned multi-DAG scheduling method, device, computer equipment, and readable storage medium obtain the latest start execution time of multiple DAGs corresponding to the job; the latest start execution time is the latest time allowed for the DAG to start execution under the premise of satisfying the deadline constraint of the DAG; then the DAGs are sorted from small to large according to the latest start execution time to guide the sequential scheduling process. In the sequential scheduling process, based on the strategy of discarding the DAG that failed to be scheduled and the backfill algorithm, multiple DAGs are scheduled sequentially according to the sorting. Other ready DAGs can use the gap to schedule tasks without affecting the previous DAG scheduling to improve resource utilization. By discarding the DAG that failed to be scheduled, a gap is made for the subsequent DAG to be scheduled, thereby further improving resource utilization and improving the DAG scheduling success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 One of the flowcharts of a multi-DAG scheduling method in one embodiment;

[0040] Figure 2 FIG2 is a second flow chart of a multi-DAG scheduling method in an embodiment;

[0041] Figure 3 is a structural block diagram of a multi-DAG scheduling device in an embodiment;

[0042] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] DAG (Directed Acyclic Graph): refers to a directed graph without loops. In graph theory, if a directed graph cannot start from a node, pass through several nodes and then return to the initial node, it is called a directed acyclic graph.

[0045] In a directed acyclic graph, each node represents a task or operation, and the directed edges between nodes represent the dependencies between tasks. This representation method enables the DAG model to clearly display the execution order and dependency hierarchy between tasks, providing strong support for task scheduling and execution. For example, if task B depends on the output of task A, then task B cannot start executing before task A is completed. This data flow ensures the correct execution order and data transfer between tasks.

[0046] During the DAG scheduling process, due to data dependencies between tasks, subsequent tasks must wait for certain predecessor tasks to complete before they can start executing. This waiting time causes resources (such as CPU, memory, etc.) to be idle for a period of time, resulting in resource gaps. When a large amount of data needs to be transferred between tasks, the data transmission time may be very long, causing subsequent tasks to be unable to start execution for a long time. In addition, network delays and data transmission overhead during data transmission will also affect the effective use of resources.

[0047] There are currently several methods for DAG scheduling:

[0048] (1) Sequential method: Schedule DAG in a certain order. Commonly used sequential methods include the DAG number-based algorithm, the EDF (Earliest Deadline First) algorithm based on the DAG deadline, and the LLF (Least Laxity First) algorithm based on the DAG slack.

[0049] (2) Merge method: Multiple DAGs are combined into a large DAG for scheduling. The scheduling goal of the MCSW (Merge-based Strategy for Workload Scheduling) algorithm based on DAG merging is to optimize resource utilization and scheduling success rate while ensuring fairness.

[0050] (3) Round-robin method: Select a task with the highest priority from each DAG in turn and assign it to the processor according to a certain strategy.

[0051] (4) Fair Scheduling Method: The Fairness (algorithmic fairness) algorithm schedules tasks based on fairness and task priority, giving priority to the tasks with the highest priority among all DAGs that are relatively lagging behind.

[0052] In a distributed cloud environment, resource gaps will inevitably occur during DAG scheduling due to data transmission. When multiple DAGs share the same set of resources, these gaps can be used for scheduling, improving resource utilization and scheduling success rate, and bringing economic benefits to cloud service providers and users. In this process, based on the above introduction, the execution of one DAG will inevitably affect the execution of other DAGs, and different DAG scheduling orders will produce different scheduling results. In applications such as e-commerce websites, it is common for multiple DAGs with deadline constraints to be submitted simultaneously. Therefore, it is of great significance and urgent to study how to schedule multiple DAGs with deadline constraints and submitted simultaneously, and how to optimize resource utilization and scheduling success rate.

[0053] Existing research on the scheduling of multiple DAGs with the same submission time mainly focuses on multi-DAGs without deadline constraints. The scheduling objectives are to minimize the overall completion time, maximize fairness, optimize resource allocation, etc. Regarding the scheduling of multiple DAGs with deadline constraints, most studies use the merging method and the sequential method. The merging method pays more attention to the fairness of scheduling, and the sequential method needs to measure the priority order of multiple DAGs. Different scheduling orders will affect the distribution of resource gaps on the processor, thereby affecting the scheduling of subsequent DAGs.

[0054] In view of the above problems, in an exemplary embodiment, Figure 1 As shown, a multi-DAG scheduling method is provided, which is applied to a distributed system. The method includes:

[0055] S102, obtaining the latest execution start time of multiple DAGs corresponding to the job; the latest execution start time is the latest time when the DAG is allowed to start execution under the premise of satisfying the deadline constraint of the DAG.

[0056] S104, sorting the DAGs according to the latest execution start time from small to large. For a given DAG, the latest execution start time (Latest Start Time, LST) refers to the latest time point at which the DAG can start execution without violating the deadline constraint of the DAG, so as to ensure that the entire DAG can be completed before its deadline. Sort all DAGs according to their latest execution start time so that they can be scheduled in chronological order. The sorted DAG list will guide the subsequent scheduling process to ensure that those DAGs that need to start earlier are scheduled first. Specifically, the sorting values ​​of all DAGs can be calculated to sort multiple DAGs, and an ordered DAG list can be constructed based on it, and then the single DAG scheduling algorithm can be used to schedule them in sequence according to the ordered list.

[0057] S106 , based on the strategy of discarding the DAG that fails to be scheduled and the backfill algorithm, multiple DAGs are scheduled sequentially according to the sorting.

[0058] Among them, the backfill algorithm can be used to find and insert additional DAGs between already scheduled DAGs to optimize resource usage. The distributed system attempts to schedule each DAG one by one according to the sorted DAG list. If the current resources are insufficient to schedule a DAG, etc., causing the DAG scheduling to fail, the DAG scheduling is abandoned according to the strategy of discarding the failed DAG to make room. If the resources are sufficient, the subsequent DAG is found and inserted between the scheduled DAGs based on the backfill algorithm to optimize resource usage.

[0059] Specifically, the latest start execution time of multiple DAGs corresponding to the job is obtained; the latest start execution time is the latest time that the DAG is allowed to start execution under the premise of satisfying the deadline constraint of the DAG; then the DAGs are sorted from small to large according to the latest start execution time to guide the sequential scheduling process. In the sequential scheduling process, based on the strategy of discarding the DAG that fails to be scheduled and the backfill algorithm, multiple DAGs are scheduled sequentially according to the sorting. Other ready DAGs can use the gaps to schedule tasks without affecting the scheduling of the previous DAG to improve resource utilization. By discarding the DAG that fails to be scheduled, gaps are made for the subsequent DAG to be scheduled, further improving resource utilization and improving the success rate of DAG scheduling.

[0060] The multi-DAG scheduling method provided in the embodiment of the present application proposes a multi-DAG scheduling algorithm based on the latest start execution time at the job level, uses the latest start execution time of DAG to measure the scheduling order of DAG, and optimizes resource utilization and scheduling success rate through backfilling and restoration methods.

[0061] In one embodiment, obtaining the latest start execution time of each DAG includes:

[0062] According to the sub-deadline time of each DAG and the execution time of the entry task, as well as the following expression, the latest start execution time LST of each DAG is determined:

[0063]

[0064] in, is the sub-deadline time, is the execution time of the entry task.

[0065] In the multi-DAG scheduling method provided by the embodiment of the present application, the smaller the LST, the earlier the DAG needs to be scheduled. The LST of a DAG is the lst of its entry task, that is, , where tdl is the sub-deadline of each task set G, and the sub-deadline of G can be based on the tdl of the DAG entry task, that is, .

[0066] based on and The LST determined by the time difference can measure the scheduling order of DAG and will not depend on the algorithm execution effect. For example, if the critical path of the task set G of DAG is long and slightly less than its deadline, then the LST of G is very small and will be scheduled first. The result of its mixed scheduling is closer to the result of single scheduling. Regardless of the effect of single scheduling, the deadline constraint of G can be met as much as possible. A DAG with good single scheduling effect will not exceed the deadline due to the influence of other DAGs due to the later scheduling order. A DAG with a larger LST can use the processor gap to execute, try to meet the deadline constraint, optimize resource utilization and take into account the scheduling success rate.

[0067] In one embodiment, before the step of determining the latest start execution time LST of each DAG according to the sub-deadline time of each DAG and the execution time of the entry task and the following expression, the step further includes:

[0068] Relax the sub-deadlines of all tasks in the task set.

[0069] When a critical subtask and its parent task are scheduled on the same processor, the critical path of the DAG may change. The relaxed sub-deadline tdl will be more suitable for single DAG scheduling and improve the scheduling success rate.

[0070] In one embodiment, Figure 2 As shown, the multiple DAGs are stored in a DAG list in a list form, and the step S106 of sequentially scheduling the multiple DAGs according to the sorting based on the strategy of discarding the DAG that failed to be scheduled and the backfill algorithm includes:

[0071] Iterate the following steps until the DAG list is empty:

[0072] S201, obtaining the head DAG in the DAG list;

[0073] S202, saving the load condition of the current processor as a copy, and calculating the load condition on the [ST, DL] interval; wherein ST is the submission time of the task set, and DL is the deadline of the task set;

[0074] S203, using the ICSTS algorithm to schedule the task set corresponding to the head DAG in the interval, so that the tasks in the task set are scheduled using the gaps in the interval;

[0075] S204, if the task set is successfully scheduled, updating the processor load according to the scheduling result of the task set;

[0076] S205, the head DAG is removed from the DAG list to update the DAG list. It should be understood that here the head DAG that has completed scheduling is removed, and after the DAG list is updated, the next DAG in the list is used as the new head DAG to perform scheduling of the next DAG.

[0077] Among them, the ICSTS (Interval-based Critical Section Task Scheduling) algorithm is an algorithm for scheduling tasks within a given time interval, and is applicable to task scheduling problems with critical sections. When scheduling a set of tasks G within the interval [ST, DL] (start time ST, deadline DL), it can ensure that the tasks are completed as efficiently as possible without affecting the execution of each other's critical sections.

[0078] When a distributed system uses DAG to schedule jobs, it will split the jobs into different stages (Stage) according to the dependencies in the DAG. Each stage contains a set of tasks with dependencies (i.e., task sets). The scheduler (such as DAGScheduler and TaskScheduler in Spark) will determine the execution order and allocation of tasks based on the dependencies and resource conditions in the DAG. Taking the scheduling of a task set G corresponding to the DAG as an example, the specific steps of the backfill method are:

[0079] First, save the current processor load as a copy and calculate the load on the [ST, DL] interval. Then, use the ICSTS algorithm to schedule G within the above interval. Tasks in G can be scheduled using the gaps in the interval. Finally, update the processor load according to the scheduling result of G to prepare for the scheduling of the next DAG.

[0080] The specific implementation can be based on the following code:

[0081]

[0082] The time complexity analysis of the above code execution process is as follows:

[0083] Line 1: The complexity of the initialization operation is , where z is the number of DAGs;

[0084] Lines 2 to 6: The for loop is executed a total of z times;

[0085] Line 3: Calculate the complexity of tdl for all tasks in each DAG , the total complexity is , The number of tasks for the DAG with the largest number of tasks;

[0086] Lines 4-5: Calculate the complexity of the LST of each DAG as , the total complexity is O(z); Line 7: The complexity of sorting the DAG list is O(z×logz);

[0087] Lines 8 to 17: The while loop is executed z times, because each loop moves a DAG to DAGList;

[0088] Line 9: The complexity of getting the first element of the DAGList is , the total complexity is O(z); Line 10: The complexity of saving the processor load is O(m), and the total complexity is O(m×z);

[0089] Line 11: The complexity of scheduling each DAG is , the complexity of scheduling all DAGs is O(z× ×m);

[0090] Lines 11 to 15: The complexity of determining the scheduling result is , the total complexity is O(z);

[0091] Line 12: The complexity of updating the schedule is , the total complexity is O(z);

[0092] Line 14: The complexity of restoring the processor load is O(m), and the total complexity is O(m×z); Line 16: The complexity of removing an element from DAGList is , the total complexity is O(z);

[0093] Line 18: The complexity of the returned result is .

[0094] According to the above analysis, the time complexity of the multi-DAG scheduling method provided in the embodiment of the present application is O(z× ×m), while the time complexity of the number-based algorithm is , the time complexity of the merge scheduling algorithm is O(z 2× ×m), the time complexity of the round-robin algorithm is , the time complexity of the fair scheduling algorithm is Compared with these algorithms, the time complexity of the multi-DAG scheduling method provided in the embodiment of the present application does not increase, that is, a solution to maximize resource utilization and scheduling success rate can be provided for the multi-DAG shared resource scheduling problem with deadline constraints within a power polynomial time without increasing the time complexity.

[0095] In one embodiment, Figure 2 As shown, the iteration executes the following steps until the DAG list is empty and the loop step further includes:

[0096] S206: If the task set scheduling fails, discard the head DAG and use the copy to restore the load condition of the processor to the load condition before the task set scheduling.

[0097] This method adopts a discard strategy when DAG scheduling fails. This is because once a task fails to schedule in DAG, the entire DAG fails to schedule, and the DAG that fails to schedule does not participate in the calculation of resource utilization and scheduling success rate. When DAG scheduling fails, the copy is used to restore the load situation before DAG scheduling to make room for subsequent DAG scheduling, which can improve resource utilization.

[0098] The multi-DAG scheduling method provided in the embodiment of the present application uses the backfill method as a discarding strategy for DAG scheduling failures. This is because once a task fails to be scheduled in a DAG, the entire DAG scheduling fails, and the DAG that fails to be scheduled does not participate in the calculation of resource utilization and scheduling success rate. By discarding the DAG that fails to be scheduled, a gap is made for the subsequent DAG to be scheduled.

[0099] Aiming at the problem of multi-DAG shared resource scheduling with deadline constraints, a multi-DAG scheduling algorithm based on the latest start execution time is proposed, taking resource utilization and scheduling success rate as scheduling optimization objectives. A multi-DAG scheduling algorithm based on the latest start execution time is proposed at the job level. The algorithm fully considers the deadline constraints and structural differences of DAG, measures the DAG scheduling order with the latest start execution time, and schedules multiple DAGs through backfilling and restoration methods. Theoretical analysis shows that the time complexity of the algorithm is That is, the solution to maximize UR and SR can be given for the multi-DAG shared resource scheduling problem with deadline constraints within a power polynomial time, which can be applied to the system design and maintenance related strategies of cloud platforms in cloud computing.

[0100] When multiple DAGs share the same set of resources, these gaps can be used for scheduling, improving resource utilization and scheduling success rate, bringing economic benefits to cloud service providers and users.

[0101] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0102] Based on the same inventive concept, the embodiment of the present application also provides a multi-DAG scheduling device for implementing the multi-DAG scheduling method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more multi-DAG scheduling device embodiments provided below can refer to the limitations of the multi-DAG scheduling method above, and will not be repeated here.

[0103] In an exemplary embodiment, Figure 3 As shown, a multi-DAG scheduling device is provided, which is applied to a distributed system. The device includes: a latest start execution time acquisition module 301, a sorting module 302, and a scheduling execution module 303. Among them:

[0104] The latest execution start time acquisition module 301 is used to acquire the latest execution start time of multiple DAGs corresponding to the job; the latest execution start time is the latest time that the DAG is allowed to start executing under the premise of satisfying the deadline constraint of the DAG;

[0105] A sorting module 302, configured to sort the DAGs in ascending order according to the latest execution start time;

[0106] The scheduling execution module 303 is used to schedule multiple DAGs in order according to the sorting based on the strategy of discarding the DAG that fails to be scheduled and the backfill algorithm.

[0107] In one embodiment, the latest execution start time acquisition module 301 includes:

[0108] The LST determination unit is used to determine the latest start execution time LST of each DAG according to the sub-deadline time of each DAG and the execution time of the entry task, as well as the following expression:

[0109]

[0110] in, is the sub-deadline time, is the execution time of the entry task.

[0111] In one embodiment, the multi-DAG scheduling module further includes:

[0112] The sub-deadline relaxation module is used to relax the sub-deadline of all tasks in the task set.

[0113] Each module in the above multi-DAG scheduling device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0114] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as DAG lists. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-DAG scheduling method is implemented.

[0115] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0116] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the above method are implemented and corresponding beneficial effects are achieved.

[0117] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented and corresponding beneficial effects are achieved.

[0118] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above method when executed by a processor and achieves corresponding beneficial effects.

[0119] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0121] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-DAG scheduling method, characterized in that: Applied to a distributed system, the method comprises: Obtain the latest start execution time of multiple DAGs corresponding to the job; the latest start execution time is the latest time that the DAG is allowed to start executing under the premise of satisfying the deadline constraint of the DAG; Sort the DAGs in ascending order according to the latest execution start time; Based on the strategy of discarding the DAG that fails to be scheduled and the backfill algorithm, multiple DAGs are scheduled sequentially according to the sorting.

2. The method according to claim 1, characterized in that: The obtaining of the latest start execution time of each DAG includes: According to the sub-deadline time of each DAG and the execution time of the entry task, as well as the following expression, the latest start execution time LST of each DAG is determined: in, is the sub-deadline time, is the execution time of the entry task.

3. The method according to claim 2, characterized in that Before the step of determining the latest start execution time LST of each DAG according to the sub-deadline time of each DAG and the execution time of the entry task and the following expression, the method further includes: Relax the sub-deadlines of all tasks in the task set.

4. The method according to claim 3, characterized in that: The multiple DAGs are stored in a DAG list in a list form, and the strategy of discarding the DAG that fails to be scheduled and the backfill algorithm are used to sequentially schedule the multiple DAGs according to the sorting, including: Iterate the following steps until the DAG list is empty: Get the head DAG in the DAG list; Save the current processor load as a copy, and calculate the load on the [ST, DL] interval; where ST is the submission time of the task set, and DL is the deadline of the task set; Using the ICSTS algorithm to schedule the task set corresponding to the head DAG in the interval, so that the tasks in the task set are scheduled using the gaps in the interval; If the task set is successfully scheduled, updating the processor load according to the scheduling result of the task set; The head DAG is removed from the DAG list to update the DAG list.

5. The method according to claim 4, characterized in that The iterative execution of the following steps, until the DAG list is empty, further includes: If the task set scheduling fails, the head DAG is discarded, and the load condition of the processor is restored to the load condition before the task set scheduling using the copy.

6. A multi-DAG scheduling device, characterized in that: Applied to a distributed system, the device comprises: The latest execution start time acquisition module is used to acquire the latest execution start time of multiple DAGs corresponding to the job; the latest execution start time is the latest time that the DAG is allowed to start executing under the premise of satisfying the deadline constraint of the DAG; A sorting module, used to sort the DAGs in ascending order according to the latest execution start time; The scheduling execution module is used to schedule multiple DAGs in order according to the sorting based on the strategy of discarding the DAG that fails to be scheduled and the backfill algorithm.

7. The device according to claim 6, characterized in that The latest execution start time acquisition module includes: The LST determination unit is used to determine the latest start execution time LST of each DAG according to the sub-deadline time of each DAG and the execution time of the entry task, as well as the following expression: in, is the sub-deadline time, is the execution time of the entry task.

8. The device according to claim 7, characterized in that The device further comprises: The sub-deadline relaxation module is used to relax the sub-deadline of all tasks in the task set.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images