A workflow scheduling method, system and device

Abstract

The present invention provides a workflow scheduling method, system, and device. The method includes: constructing a workflow directed acyclic graph (DAG); using a center-point-based DAG clustering method based on the workflow directed acyclic graph (DAG) to obtain a set of clusters; based on the set of clusters and a pre-allocation model, obtaining a CPU allocation plan for each cluster; and based on the workflow directed acyclic graph (DAG) and the CPU allocation plan for each cluster, obtaining a final workflow scheduling plan based on a virtual earliest end time scheduling model. The present invention uses a center-point-based DAG clustering method to obtain a set of clusters, and uses the set of clusters to guide scheduling, thereby obtaining a more optimal workflow scheduling plan, thereby fundamentally solving the communication congestion problem in existing scheduling for shared bus heterogeneous computing systems.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of workflow scheduling, and particularly relates to a workflow scheduling method, system and device. BACKGROUND

[0002] Heterogeneous computing system integrates multiple different types of computing units such as CPU, GPU, DSP, ASIC and FPGA, and achieves effective balance between performance and thermal efficiency. In order to meet the performance requirements of specific functions, higher thermal efficiency and stronger reliability, workflow applications tend to run on heterogeneous computing systems, and shared bus is a commonly used communication architecture in heterogeneous computing systems. Shared bus provides a unified communication platform, but also brings the problem of resource contention: when multiple processors transmit data at the same time, communication contention may occur, resulting in performance degradation. In addition, networking makes the amount of communication data generated by the system increase dramatically and the structure diversify, making the scheduling easily produce a large amount of communication overhead, and then affecting the overall performance of the system. There are some existing workflow scheduling methods combined with communication contention awareness, but most of them are limited to idealized models or specific application scenarios, and have some improvements in dealing with communication contention, but in the face of the explosive growth of communication data, the performance of the scheduling scheme given by this method is not optimal.

[0003] However, in the bus-based communication model, list scheduling determines the scheduling order according to the priority of the node, and since the priorities of sibling nodes are usually very close, they will be scheduled at adjacent time points. The traditional method usually allocates resources based on the greedy strategy, which further increases the possibility of them competing for the same resources at the same time. This situation leads to the proximity of the edges connected to the parent node (or child node) in the transmission time. When scheduling to the sink point, the sudden burst of communication demand accumulated due to the greedy strategy significantly increases the possibility of congestion. Communication congestion can cause processors to be idle.

[0004] Therefore, for a shared bus heterogeneous computing system, how to avoid communication congestion in scheduling is a problem to be solved in the field. SUMMARY

[0005] The purpose of the present invention is to provide a workflow scheduling method, system and device in response to the defects of the prior art. The workflow scheduling method of the present invention constructs a workflow directed acyclic graph (DAG); based on the workflow directed acyclic graph (DAG), a DAG clustering method based on a center point is adopted to obtain a cluster set; based on the cluster set, a CPU allocation scheme for each cluster is obtained based on a pre-allocation model; based on the workflow directed acyclic graph (DAG) and the CPU allocation scheme for each cluster, a final workflow scheduling scheme is obtained based on a scheduling model of a virtual earliest end time; the present invention adopts a DAG clustering method based on a center point to obtain a cluster set, and uses the cluster set to guide scheduling, thereby obtaining a better workflow scheduling scheme, thereby fundamentally solving the communication congestion problem in the existing workflow scheduling for shared bus heterogeneous computing systems.

[0006] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0007] The present invention proposes a workflow scheduling method, which is characterized by comprising the steps of:

[0008] S1. Construct a directed acyclic graph (DAG) of the workflow; where DAG = (V, E), V = {v1, v2, ..., v i} represents the set of i computing tasks in the workflow, E={e1,e2,…,e k} represents the set of k communication tasks in the workflow, e k =e(i,j)∈E represents a communication task, reflecting the dependency between computing tasks;

[0009] S2. According to the workflow directed acyclic graph (DAG), a DAG clustering method based on a center point is adopted to obtain a cluster set; specifically, the method includes:

[0010] S21, constructing a center point set C;

[0011] S22. Initialize the cluster corresponding to each node in the DAG

[0012] S23, traverse the central point set C, and in each traversal, calculate the nodes to be added to The threshold Θ i , and the center point corresponding to Assign the center point c i , then call the clustering function to recursively add nodes, and finally get the cluster set;

[0013] S3. Based on the cluster set and the pre-allocation model, a CPU allocation plan for each cluster is obtained; specifically, the plan includes:

[0014] S31. Calculate each cluster Priority Based on the priority, all clusters are sorted in descending order;

[0015] S32, each processor P m The occupied time Ω m Initialized to 0;

[0016] S33. For each cluster Calculate its r Estimated earliest finish time on Assign it to minimize The processor and update the corresponding processor P accordingly m Ω m ;

[0017] S4. Obtain a final workflow scheduling solution based on the workflow directed acyclic graph (DAG) and the CPU allocation solution for each cluster and the scheduling model based on the virtual earliest end time. Specifically, the solution includes:

[0018] S41. Generate a task list sorted in non-increasing order by task priority rank

[0019] S42. Initialize the available time of all resources to 0;

[0020] S43. In each iteration, the first task node is selected from the task list for scheduling, and its predecessor communication task is simultaneously scheduled. When the scheduling of the task node is completed, it is deleted from the task list;

[0021] The steps of each iteration are: i Generate a priority list of all its immediate predecessor tasks And sort in non-increasing order according to rank; if the current node has been pre-allocated, update the reservation time τ m Otherwise, v i Temporarily assigned to each processor P r ∈P on;

[0022] For each such temporary assignment, each predecessor communication task e k Temporarily assigned to bus B according to priority k ∈B, so that the communication task e k Completion time t · (e k ) minimum;

[0023] Based on the distribution results of these communication tasks, calculate the Pm Virtual earliest end time on v i is actually allocated to make it On the smallest processor;

[0024] In v i After the processor is determined, its predecessor communication task is assigned to the bus that makes the communication task end earliest, and the corresponding bus is updated to minimize their end time.

[0025] Furthermore, in step S21, the center point set C is defined as follows:

[0026]

[0027] in, Indicates v i The parameter α is used to adjust the strictness of the selection condition, B represents the bus, P represents the processor, and CCR represents the computation-communication ratio.

[0028] Furthermore, in step S23, the calculation adds the node to The threshold Θ i , specifically including:

[0029] By calculation The average weight on all buses determines the node to be added to The threshold Θ i ,Θ i is calculated as follows:

[0030]

[0031] in, Indicates v i The set of adjacent edges of Indicates e k The average weight on the bus.

[0032] Furthermore, in step S23, the clustering function is called to recursively add nodes, and finally a cluster set is obtained, specifically:

[0033] In the clustering function, all predecessor nodes of node node are traversed If the predecessor node has been visited, skip the node; calculate the average edge weight of the node and the corresponding predecessor If it is greater than the threshold Θ, the predecessor node pred corresponding to Assigned to Then continue to call recursively through this node; otherwise, the node corresponding to the predecessor node Assigning a value of -1 means that the node will not be clustered into any cluster;

[0034] Traverse all successor nodes of node node If the successor node has been visited, skip the node; calculate the average edge weight of the node node and the corresponding successor, if it is greater than the threshold Θ, then the node corresponding to the successor node succ is accessed. Assigned to Then continue the recursive call through this node; otherwise, the node corresponding to the successor node Assigning a value of -1 means that the node will not be clustered into any cluster;

[0035] Each node v i of Attributes are assigned values; have the same The values ​​belong to the same cluster; if A value of -1 indicates that the node is not assigned to any cluster.

[0036] Furthermore, in step S31, each cluster is calculated Priority The specific steps are:

[0037] First, OCT (v i ,P m ,B n ) is calculated as follows:

[0038]

[0039] in, Represents node v i The set of successor nodes, OCT(v i ,P m ,B n )express, Represents node v j Assigned to processor P x , and communicate with the successor node through B y When executing, the maximum optimistic execution time of its child nodes, Represents the communication task e i,j On Bus B n The communication overhead on v i and v j When mapped to the same processor, P m =P x ,but OCT(v i ,P m ,Bn the maximum optimistic execution time of the child nodes of node v i ;

[0040] The priority of each node is defined by averaging the priority of the OCT reaction tasks as follows:

[0041]

[0042] The priority of the cluster is defined as follows:

[0043]

[0044] Further, in step S33, the earliest completion time is calculated as follows:

[0045]

[0046] wherein represents the average weight of node v j on the processor.

[0047] Further, in step S43, specifically comprising: recording the set of nodes in the cluster which have not been actually allocated as W, assuming that the pre-allocated processor is P m , then the reserved time τ m,i of processor P m is represented by the following formula:

[0048]

[0049] wherein represents the time required for node v i to execute on processor P m ;

[0050] For processor P m , the total reserved time τ m is:

[0051]

[0052] By introducing τ m , the virtual earliest end time is calculated as follows:

[0053]

[0054] wherein t · (v i , P m ) represents the time required for node v​i The actual end time is executed on the processor P m The actual end time is executed on the processor P

[0055] The application further provides a workflow scheduling system, characterized in that the workflow scheduling system executes the workflow scheduling method, and comprises a workflow DAG construction module, a cluster set generation module, a cluster CPU pre-allocation scheme generation module and a workflow scheduling scheme generation module.

[0056] The workflow DAG construction module constructs a workflow directed acyclic graph (DAG).

[0057] The cluster set generation module obtains a cluster set according to the workflow DAG by using a center-point-based DAG clustering method.

[0058] The cluster CPU pre-allocation scheme generation module obtains a CPU allocation scheme of each cluster based on the cluster set and a pre-allocation model.

[0059] The workflow scheduling scheme generation module obtains a final workflow scheduling scheme based on a virtual earliest end time scheduling model according to the workflow DAG and the CPU allocation scheme of each cluster.

[0060] The application further provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the method.

[0061] Compared with the prior art, the application has the following beneficial effects:

[0062] The workflow scheduling method of the application constructs a workflow DAG, obtains a cluster set according to the workflow DAG by using a center-point-based DAG clustering method, obtains a CPU allocation scheme of each cluster based on the cluster set and a pre-allocation model, and obtains a final workflow scheduling scheme based on a virtual earliest end time scheduling model according to the workflow DAG and the CPU allocation scheme of each cluster. The application obtains a cluster set by using a center-point-based DAG clustering method, and guides scheduling by using the cluster set, thereby effectively relieving communication congestion and obtaining a more optimal workflow scheduling scheme. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0064] Figure 1 The working flow scheduling method provided by the embodiment of the present application is shown in the schematic diagram.

[0065] Figure 2 The working flow scheduling system provided by the embodiment of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present application.

[0067] It should be understood that when used in the specification and the appended claims, the terms "comprise" and "include" indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0068] The present application will be further described below in conjunction with the drawings and specific embodiments, but not as a limitation of the present application.

[0069] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0070] It should be further understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0071] The character meaning is explained as shown in the following table:

[0072]

[0073]

[0074] The present application proposes a working flow scheduling method. As shown inFigure 1 As shown, the workflow scheduling method includes the following steps S1 to S4.

[0075] S1, a directed acyclic graph DAG of a workflow is constructed; wherein, DAG=(V, E), V={v1, v2, …, v i} represents a set of i computing tasks in the workflow, E={e1, e2, …, e k} represents a set of k communication tasks in the workflow, e k =e(i, j) E represents a communication task, reflecting the dependency relationship between computing tasks. Wherein, v i is the direct predecessor of v j .

[0076] Since a computing task can have multiple predecessor nodes and successor nodes, we define and represent the predecessor node set and successor node set of v i , respectively.

[0077]

[0078] Correspondingly, the adjacent edge set of v i is represented as contains all incoming edges and outgoing edges connecting v i :

[0079]

[0080] DAG has only one entry node v entry , which satisfies indicates that there is no predecessor node;

[0081] DAG also has only one exit node v exit , which satisfies indicates that there is no successor node.

[0082] Suppose that computing tasks v i and v j are executed on processors P l and P m , respectively, and there is a communication task e k =e i,j between them. If P l =P m , then e k can start directly after the execution of v i ends. If P l ≠P m , then e k in addition to the bus B n can complete the communication task ek The feasible time slice is B n You can execute e k The earliest time is t ◇ (e k ,B n ). From this we can get e k Earliest start time:

[0083]

[0084] When the task v i and v j When assigned to the same processor (i.e. P l =P m ), the communication between them is carried out in a shared memory manner without involving an external network. This communication method can significantly reduce communication delay and overhead. Compared with communication through an external bus, the time consumed by this communication is negligible. k The earliest end time can be calculated as follows:

[0085]

[0086] For the computation task v i , he needs to wait until all input data reaches the processor P where it is located m , and P m Ready to execute v i Then you can start. m You can execute v i The earliest time is t ◇ (v i ,P m ). Then v i In P m The earliest start time on is:

[0087]

[0088] S2. According to the workflow directed acyclic graph DAG, a center-based DAG clustering method is adopted to obtain a cluster set.

[0089] Traditional methods typically allocate resources based on a greedy strategy, further increasing the likelihood that they will compete for the same resources at the same time. This situation results in the edges connecting them to their parent nodes (or child nodes) being close in transmission time. When dispatched to the convergence point, the accumulated communication demand due to the greedy strategy suddenly bursts, significantly increasing the possibility of congestion. Communication congestion can lead to processor idleness. However, this processor idleness is precisely what the clustering method of the present invention aims to optimize. The clustering method of the present invention is also proposed to address this type of problem that is widely present in real workflows.

[0090] On the other hand, as mentioned before, the communication demand usually increases dramatically before and after the schedule of the sink, which means that the demand for communication resources varies significantly at different time periods. If the communication resource shortage at a specific time period is alleviated by increasing the number of buses, it will inevitably lead to the idle of communication resources at most of the time. In order to balance the economy and resource utilization, and avoid the idle of computing units and communication units due to mutual waiting, it is the most reasonable choice to avoid communication congestion by optimization method.

[0091] The communication congestion usually occurs at the sink, and the essential reason is that the communication resources represented by the buses are difficult to handle the simultaneous communication demand at the time period before and after the schedule of the sink. Inspired by this observation, the present application proposes a natural and direct solution: scheduling the nodes that need to transmit more data to each other to the same processor to eliminate the occupation of the corresponding high-occupied communication task to the bus. The center-based DAG clustering method proposed by the present application mainly aims to allocate the nodes in the same cluster to the same processor so that the communication between them does not need to pass through the bus, thereby alleviating the shortage of communication resources.

[0092] The center-based DAG clustering method of the present application needs to traverse the DAG in advance to obtain the structure information of the whole graph, retain this information in some way, and utilize these information in the actual scheduling. In the present application, the cluster is obtained by the clustering strategy, and then the cluster is used to guide the scheduling, so as to obtain a more optimal scheduling scheme.

[0093] According to the workflow directed acyclic graph DAG, the core idea of the center-based DAG clustering method to obtain the cluster set is to select a part of center nodes c i according to the standard, and expand from these nodes to the parent node and child node direction respectively, so as to increase the number of nodes in the cluster c i corresponding to the cluster ; Specifically, it includes:

[0094] S21, constructing a center point set C;

[0095] Further, in step S21, the center point set C is defined as follows:

[0096]

[0097] Wherein, represents the adjacent edge set of v i , and the parameter a is used to adjust the strictness of the selection condition, B represents the bus, P represents the processor, and CCR represents the computing communication ratio.

[0098] Specifically, the center node c iA specific in-degree criterion should be met. We use the graph properties of G = (V, E) and the computational environment ρ to determine this criterion and define the set of central points based on this criterion.

[0099] Furthermore, by adjusting the value of α, we can control the lower limit of the number of neighbors of the selected node. Increasing α will increase the lower limit of the number of neighbors, making the selection of set C more stringent; conversely, decreasing α will relax the selection criteria. By default, the value of α is 1.

[0100] S22. Initialize the cluster corresponding to each node in the DAG

[0101] S23, traverse the central point set C, and in each traversal, calculate the nodes to be added to The threshold Θ i , and the center point corresponding to Assign the center point c i , then call the clustering function to recursively add nodes, and finally get the cluster set;

[0102] For each central node c in C i , there is a corresponding cluster By calculation The average weight on all buses determines the node to be added to The threshold Θ i .

[0103] The calculation adds the node to The threshold Θ i , specifically including: by calculating The average weight on all buses determines the node to be added to The threshold Θ i ,Θ i is calculated as follows:

[0104]

[0105] in, Indicates v i The set of adjacent edges of Indicates e k The average weight on the bus.

[0106] Furthermore, in step S23, the calling function clustering is used to recursively add nodes.

[0107] Finally, the cluster set is obtained, specifically:

[0108] In the clustering function, all predecessor nodes of node node are traversed If the predecessor node has been visited, skip the node; calculate the average edge weight of the node and the corresponding predecessor If it is greater than the threshold Θ, the predecessor node pred corresponding to Assigned to Then continue to call recursively through this node; otherwise, the node corresponding to the predecessor node Assigning a value of -1 means that the node will not be clustered into any cluster;

[0109] Traverse all successor nodes of node node If the successor node has been visited, skip the node; calculate the average edge weight of the node node and the corresponding successor, if it is greater than the threshold Θ, then the node corresponding to the successor node succ is accessed. Assigned to Then continue the recursive call through this node; otherwise, the node corresponding to the successor node Assigning a value of -1 means that the node will not be clustered into any cluster;

[0110] Each node v i of Attributes are assigned values; have the same The values ​​belong to the same cluster; if A value of -1 indicates that the node is not assigned to any cluster.

[0111] S3. Based on the cluster set and the pre-allocation model, a CPU allocation plan for each cluster is obtained; specifically, the plan includes:

[0112] S31. Calculate each cluster Priority Based on the priority, all clusters are sorted in descending order;

[0113] Furthermore, in step S31, each cluster is calculated Priority The specific steps are:

[0114] In a heterogeneous environment, the Optimistic Cost Table (OCT) represents the maximum value among all the shortest paths from each node to the exit node across all node subtasks. The calculation of the OCT depends not only on the node itself but also on the processor selected. The OCT of a node in G on a specific processor must be recursively traversed from the exit node to the entry node.

[0115] First, OCT(v i , P m , B n ) is calculated as follows:

[0116]

[0117] where, denotes the successor node set of node v i , OCT(v j , P x , B y ) denotes, denotes the maximum optimistic execution time of the child nodes of node v j when it is assigned to processor P x and the communication with the successor nodes is performed through B y ; OCT(v i,j , v n ) denotes the communication overhead of communication task e i on bus B j ; when v m and v x are mapped to the same processor, i.e., P i = P m , then OCT(v n , P i , B j ) represents the maximum optimistic execution time of the child nodes of node v m ; it considers the processing unit and the communication unit that can make the execution time minimum;

[0118] The priority of each node is defined as follows by averaging the priority of the OCT reaction tasks:

[0119]

[0120] The priority of the clustering cluster is defined as follows:

[0121] After the clustering phase, we obtain the set of clustering clusters

[0122] In this phase, we will assign a processor to each clustering cluster in based on the idle situation of the processors and the time required to execute the current clustering cluster to calculate the estimated earliest completion time and assign the processor in the direction of minimizing . Further, in step S31, the earliest completion time The calculation formula is as follows:

[0124]

[0125] wherein, denotes the node v j The average weight on the processor.

[0126] S32, each processor P m The occupied time Ω m is initialized to 0;

[0127] S33, for each cluster Calculate its estimated earliest completion time r on any processor P Assign it to the processor that minimizes Update the Ω m of the corresponding processor P m .

[0128] S4, according to the workflow directed acyclic graph DAG and the CPU allocation scheme of each cluster, based on the virtual earliest end time scheduling model, obtain the final workflow scheduling scheme;

[0129] In order to quantify the impact of the remaining nodes to be allocated to the current processor in the future, we introduce the variable τ m , which represents the reserving time of the processor P m When any node in a cluster is actually scheduled (that is, the start time is determined), the corresponding CPU will reserve time slices for all nodes in . And every time a node in a cluster is actually scheduled, the corresponding processor will no longer reserve time slices for the node.

[0130] Step S4 specifically includes:

[0131] S41, generate a task list sorted in non-decreasing order according to the task priority rank

[0132] S42, initialize the available time of all resources to 0;

[0133] S43, in each iteration, select the first task node from the task list for scheduling, and schedule its predecessor communication tasks synchronously, and when the task node is scheduled, delete it from the task list;

[0134] The steps of each iteration are: for task v iGenerate a priority list of all its immediate predecessor tasks And sort in non-increasing order according to rank; if the current node has been pre-allocated, update the reservation time τ m Otherwise, v i Temporarily assigned to each processor P r ∈P on;

[0135] For each such temporary assignment, each predecessor communication task e k Temporarily assigned to bus B according to priority k ∈B, so that the communication task e k Completion time t · (e k ) minimum;

[0136] Based on the distribution results of these communication tasks, calculate the P m Virtual earliest end time on v i is actually allocated to make it On the smallest processor;

[0137] In v i After the processor is determined, its predecessor communication task is assigned to the bus that makes the communication task end earliest, and the corresponding bus is updated to minimize their end time.

[0138] Furthermore, in step S43, the following steps are specifically included: clustering the nodes that have been actually assigned The set of nodes that have not been actually allocated is recorded as W, assuming that the pre-allocated processor is P m , then processor P m Reserved time τ m,i It is expressed by the following formula:

[0139]

[0140] in, Represents node v i In the processor P m The time required for execution;

[0141] For processor P m , its total reservation time τ m for:

[0142]

[0143] By introducing τ m , virtual earliest end time The calculation formula is as follows:

[0144]

[0145] where t · (v i , P m ) denotes the node v i The actual end time on the processor P m .

[0146] Figure 2 The application provides a workflow scheduling system, which executes the workflow scheduling method. As shown in the figure, the workflow scheduling system comprises a workflow DAG construction module, a cluster set generation module, a cluster CPU pre-allocation scheme generation module and a workflow scheduling scheme generation module. Figure 2

[0147] The workflow DAG construction module constructs a workflow directed acyclic graph (DAG).

[0148] The cluster set generation module adopts a center point-based DAG clustering method to obtain a cluster set according to the workflow DAG.

[0149] The cluster CPU pre-allocation scheme generation module obtains a CPU allocation scheme of each cluster based on a pre-allocation model based on the cluster set.

[0150] The workflow scheduling scheme generation module obtains a final workflow scheduling scheme based on a virtual earliest end time scheduling model according to the workflow DAG and the CPU allocation scheme of each cluster.

[0151] The above workflow scheduling system can be realized in the form of a computer program, which can run on a computer device.

[0152] The computer device comprises a processor, a memory and a network interface connected through a system bus, wherein the memory can comprise a non-volatile storage medium and an internal memory.

[0153] The non-volatile storage medium can store an operating system and a computer program. The computer program comprises program instructions, which, when executed, can enable the processor to execute a workflow scheduling method.

[0154] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0155] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium, which, when executed by the processor, can enable the processor to execute a workflow scheduling method.

[0156] ​The network interface is used for network communication with other devices. Those skilled in the art can understand that the above computer device structure is only part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0157] The processor is configured to run a computer program stored in the memory, and the computer program implements the workflow scheduling method of the first embodiment.

[0158] It should be understood that, in the embodiments of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0159] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments of the method can be completed by a computer program instructing related hardware. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments of the method.

[0160] The present application also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program is executed by a processor to make the processor execute the workflow scheduling method of the first embodiment.

[0161] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer-readable storage media that can store program codes.

[0162] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0163] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and actual implementation can have another division. For example, multiple units or components can be combined or integrated into another system, or some features can be omitted or not implemented.

[0164] The steps in the method embodiments of the present application can be sequentially adjusted, combined and deleted according to actual needs. The units in the device embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0165] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or say the parts that make contributions to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a terminal or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0166] Note that the above is only the preferred embodiment of the present application and the technical principle applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A workflow scheduling method, characterized in that: Including steps: S1. Construct a directed acyclic graph (DAG) of the workflow; wherein DAG = (V, E), V = {v1, v2, ..., v i } represents the set of i computing tasks in the workflow, E = {e1, e2, ..., e k } represents the set of k communication tasks in the workflow, e k =e(i, j)∈E represents a communication task, reflecting the dependency between computing tasks; S2. According to the workflow directed acyclic graph (DAG), a DAG clustering method based on a center point is adopted to obtain a cluster set; specifically, the method includes: S21, constructing a center point set C; S22. Initialize the cluster corresponding to each node in the DAG S23, traverse the central point set C, and in each traversal, calculate the nodes to be added to The threshold Θ i , and the center point corresponding to Assign the center point c i , then call the clustering function to recursively add nodes, and finally get the cluster set; S3. Based on the cluster set and the pre-allocation model, a CPU allocation plan for each cluster is obtained; specifically, the plan includes: S31. Calculate each cluster Priority Based on the priority, all clusters are sorted in descending order; S32, each processor P m The occupied time Ω m Initialized to 0; S33. For each cluster Calculate its r Estimated earliest finish time on Assign it to minimize The processor and update the corresponding processor P accordingly m Ω m ; S4. Obtain a final workflow scheduling solution based on the workflow directed acyclic graph (DAG) and the CPU allocation solution for each cluster and the scheduling model based on the virtual earliest end time. Specifically, the solution includes: S41. Generate a task list sorted in non-increasing order by task priority rank S42. Initialize the available time of all resources to 0; S43. In each iteration, the first task node is selected from the task list for scheduling, and its predecessor communication task is simultaneously scheduled. When the scheduling of the task node is completed, it is deleted from the task list; The steps of each iteration are: i Generate a priority list of all its immediate predecessor tasks And sort in non-increasing order according to rank; if the current node has been pre-allocated, update the reservation time τ m Otherwise, v i Temporarily assigned to each processor P r ∈P on; For each such temporary assignment, each predecessor communication task e k Temporarily assigned to bus B according to priority k ∈B, so that the communication task e k The completion time t.(e k ) minimum; Based on the distribution results of these communication tasks, calculate the P m Virtual earliest end time on v i is actually allocated to make it On the smallest processor; In v i After the processor is determined, its predecessor communication task is assigned to the bus that makes the communication task end earliest, and the corresponding bus is updated to minimize their end time.

2. The method according to claim 1, characterized in that In step S21, the center point set C is defined as follows: in, Indicates v i The parameter α is used to adjust the strictness of the selection condition, B represents the bus, P represents the processor, and CCR represents the computation-communication ratio.

3. The method according to claim 1, characterized in that In step S23, the calculation adds the node to The threshold Θ i , specifically including: By calculation The average weight on all buses determines the node to be added to The threshold Θ i ,Θ i is calculated as follows: in, Indicates v i The set of adjacent edges of Indicates e k The average weight on the bus.

4. The method according to claim 1, wherein In step S23, the clustering function is called to recursively add nodes, and finally a cluster set is obtained, specifically: In the clustering function, all predecessor nodes of node node are traversed If the predecessor node has been visited, skip the node; calculate node nod e and the average edge weight of the corresponding predecessor If it is greater than the threshold Θ, the predecessor node pred corresponding to Assigned to Then continue to call recursively through this node; otherwise, the node corresponding to the predecessor node Assigning a value of -1 means that the node will not be clustered into any cluster; Traverse all successor nodes of node node If the successor node has been visited, skip the node; calculate node nod e And the average edge weight of the corresponding successor, if it is greater than the threshold Θ, then the successor node succ corresponding to Assigned to Then continue the recursive call through this node; otherwise, the node corresponding to the successor node Assigning a value of -1 means that the node will not be clustered into any cluster; Each node v i of Attributes are assigned values; have the same The values ​​belong to the same cluster; if A value of -1 indicates that the node is not assigned to any cluster.

5. The method according to claim 1, wherein In step S31, each cluster is calculated Priority The specific steps are: First, OCT (v i , P m , B n ) is calculated as follows: in, Represents node v i The successor node set of OCT(v i , P m , B n )express, Represents node v j Assigned to processor P x , and communicate with the successor node through B y When executing, the maximum optimistic execution time of its child nodes, Represents the communication task e i，j On Bus B n The communication overhead on v i and v j When mapped to the same processor, P m =P x ,but OCT(v i , P m , B n ) represents node v i The maximum optimistic execution time of the child nodes; By averaging the priority of the OCT response tasks, the priority of each node is defined as follows: Cluster The priority is defined as follows:

6. The method according to claim 1, characterized in that In step S33, the earliest completion time The calculation formula is as follows: in, Represents node v j Average weight across processors.

7. The method according to claim 1, characterized in that Step S43 specifically includes: clustering the nodes that have been actually assigned The set of nodes that have not been actually allocated is recorded as W, assuming that the pre-allocated processor is P m , then processor P m Reserved time τ m，i It is expressed by the following formula: in, Represents node v i In the processor P m The time required for execution; For processor P m , its total reservation time τ m for: By introducing τ m , virtual earliest end time The calculation formula is as follows: Among them, t.(v i , P m ) represents node v i In the processor P m The actual end time of the execution.

8. A workflow scheduling system, characterized in that: The workflow scheduling system executes the workflow scheduling method according to claim 1, comprising: a workflow DAG construction module, a cluster set generation module, a cluster CPU pre-allocation plan generation module, and a workflow scheduling plan generation module; The workflow DAG construction module constructs a workflow directed acyclic graph DAG; The clustering cluster set generation module obtains a clustering cluster set according to the workflow directed acyclic graph DAG by adopting a center point-based DAG clustering method; The cluster CPU pre-allocation scheme generating module obtains a CPU allocation scheme for each cluster based on the cluster set and the pre-allocation model; The workflow scheduling solution generation module obtains a final workflow scheduling solution according to the workflow directed acyclic graph (DAG) and the CPU allocation solution of each cluster, based on a scheduling model of a virtual earliest end time.

9. A computer device, characterized in that: The device includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images