Workflow scheduling method, system and equipment

Abstract

The invention provides a workflow scheduling method, system and equipment. The method comprises the following steps: constructing a workflow directed acyclic graph (DAG); according to the workflow directed acyclic graph (DAG), adopting a DAG clustering method based on a central point to obtain a cluster set; based on the cluster set and a pre-distribution model, obtaining a CPU distribution scheme of each cluster; and according to the workflow directed acyclic graph (DAG) and the CPU allocation scheme of each cluster, obtaining a final workflow scheduling scheme based on a scheduling model of virtual earliest end time. According to the method, the clustering cluster set is obtained by adopting the DAG clustering method based on the central point, and the scheduling is guided by utilizing the clustering cluster set, so that a better workflow scheduling scheme is obtained, and the problem of communication congestion in the existing scheduling oriented to the shared bus heterogeneous computing system is radically solved.

Description

Technical Field

[0001] The present invention belongs to the field of workflow scheduling, and in particular relates to a workflow scheduling method, system and equipment. Background Art

[0002] Heterogeneous computing systems achieve an effective balance between performance and thermal efficiency by integrating multiple different types of computing units, such as CPU, GPU, DSP, ASIC and FPGA. In order to meet the performance requirements of specific functions, higher thermal efficiency and stronger reliability, workflow applications tend to run on heterogeneous computing systems, and shared buses are a commonly used communication architecture in heterogeneous computing systems. The shared bus provides a unified communication platform, but it also brings the problem of resource contention: when multiple processors transmit data at the same time, communication contention may occur, resulting in performance degradation. In addition, networking has caused a surge in the amount of communication data generated by the system and diversified structures, making it very easy for scheduling to generate a large amount of communication overhead, which in turn affects the overall performance of the system. There are some existing workflow scheduling methods that combine communication contention awareness, but most of them are limited to idealized models or specific application scenarios. They have made some improvements in dealing with communication contention, but in the face of the explosive growth of communication data, the scheduling scheme given by this method does not perform well.

[0003] However, in the bus-based communication model, list scheduling determines the scheduling order based on the priority of the node. Since the priorities of sibling nodes are usually very close, they will be scheduled at close time points. Traditional methods usually allocate resources based on greedy strategies, which further increases the possibility that they will compete for the same resources at the same time. This situation causes the edges connecting them to the parent node (or child node) to be close in transmission time. When scheduling to the convergence point, the accumulated communication demand due to the greedy strategy suddenly explodes, significantly increasing the possibility of congestion. Communication congestion can cause the processor to be idle.

[0004] Therefore, for a shared bus heterogeneous computing system, how to avoid communication congestion during scheduling is an urgent problem to be solved in the art. Summary of the invention

[0005] The purpose of the present invention is to provide a workflow scheduling method, system and device in view of the defects of the prior art. The workflow scheduling method of the present invention constructs a workflow directed acyclic graph DAG; according to the workflow directed acyclic graph DAG, a DAG clustering method based on a center point is adopted to obtain a cluster cluster set; based on the cluster cluster set, a CPU allocation scheme for each cluster cluster is obtained based on a pre-allocation model; according to the workflow directed acyclic graph DAG and the CPU allocation scheme for each cluster cluster, a final workflow scheduling scheme is obtained based on a scheduling model of a virtual earliest end time; the present invention adopts a DAG clustering method based on a center point to obtain a cluster cluster set, and uses the cluster cluster set to guide scheduling, so as to obtain a better workflow scheduling scheme, thereby fundamentally solving the problem of communication congestion in the existing workflow scheduling for shared bus heterogeneous computing systems.

[0006] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] The present invention proposes a workflow scheduling method, which is characterized by comprising the steps of:

[0008] S1. Construct a directed acyclic graph (DAG) of the workflow; where DAG = (V, E), V = {v 1 ,v 2 ,…,v i} represents the set of i computing tasks in the workflow, E = {e 1 ,e 2 ,…,e k} represents the set of k communication tasks in the workflow, e k =e(i,j)∈E represents a communication task, reflecting the dependency between computing tasks;

[0009] S2. According to the workflow directed acyclic graph DAG, a DAG clustering method based on a center point is adopted to obtain a cluster set; specifically including:

[0010] S21, construct a center point set C;

[0011] S22, initialize the cluster corresponding to each node in the DAG

[0012] S23, traverse the central point set C, and in each traversal, calculate the nodes to be added to The threshold Θ i , and the center point corresponding to Assign the value to the center point c i , then call the clustering function to recursively add nodes, and finally get the cluster set;

[0013] S3, based on the cluster set and the pre-allocation model, obtaining a CPU allocation scheme for each cluster; specifically including:

[0014] S31. Calculate each cluster Priority Based on the priority, all clusters are sorted in descending order;

[0015] S32, each processor P m The occupied time Ω m Initialized to 0;

[0016] S33. For each cluster Calculate its r Estimated earliest finish time on Assign it to minimize The processor and updates the corresponding processor P accordingly m Ω m ;

[0017] S4, according to the workflow directed acyclic graph DAG and the CPU allocation plan of each cluster, based on the scheduling model of the virtual earliest end time, obtain the final workflow scheduling plan; specifically including:

[0018] S41. Generate a task list sorted in non-increasing order according to task priority rank

[0019] S42, initializing the available time of all resources to 0;

[0020] S43, in each iteration, selecting the first task node from the task list for scheduling, and synchronously scheduling its predecessor communication task, and when the scheduling of the task node is completed, deleting it from the task list;

[0021] The steps of each iteration are: i Generate a priority list of all its immediate predecessor tasks And sort in non-increasing order according to rank; if the current node has been pre-allocated, update the reservation time τ m Otherwise, v i Temporarily assigned to each processor P r ∈P;

[0022] For each such temporary assignment, each predecessor communication task e k Temporarily assigned to bus B according to priority k ∈B, so that the communication task e k The completion time t · (e k) minimum;

[0023] Based on the distribution results of these communication tasks, calculate all processors P m Virtual earliest end time on v i is actually allocated to make it On the smallest processor;

[0024] In v i After the processor is determined, its predecessor communication task is assigned to the bus that makes the communication task end earliest, and the corresponding bus is updated to minimize their end time.

[0025] Furthermore, in step S21, the center point set C is defined as follows:

[0026]

[0027] in, Indicates v i The parameter α is used to adjust the strictness of the selection condition. B represents the bus, P represents the processor, and CCR represents the computation-to-communication ratio.

[0028] Further, in step S23, the calculation adds the node to The threshold Θ i , including:

[0029] By calculation The average weight on all buses determines the node to be added to The threshold Θ i , Θ i The calculation method is as follows:

[0030]

[0031] in, Indicates v i The set of adjacent edges of Indicates e k The average weight on the bus.

[0032] Furthermore, in step S23, the calling function clustering is used to recursively add nodes, and finally a cluster set is obtained, specifically:

[0033] In the clustering function, all predecessor nodes of node node are traversed If the predecessor node has been visited, skip the node; calculate the average edge weight of the node node and the corresponding predecessor If it is greater than the threshold Θ, the predecessor node pred corresponding to Assigned to Then continue to recursively call through this node; otherwise, the node corresponding to the predecessor node The value is -1, which means that the node will not be clustered into any cluster;

[0034] Traverse all the successor nodes of node node If the successor node has been visited, skip the node; calculate the average edge weight of the node node and the corresponding successor. If it is greater than the threshold Θ, the node corresponding to the successor node succ is accessed. Assigned to Then continue the recursive call through this node; otherwise, the node corresponding to the successor node The value is -1, which means that the node will not be clustered into any cluster;

[0035] Each node v i of The properties have been assigned values; have the same The values ​​belong to the same cluster; if A value of -1 indicates that the node is not assigned to any cluster.

[0036] Furthermore, in step S31, each cluster is calculated Priority The specific steps are:

[0037] First, OCT (v i ,P m ,B n ) is calculated as follows:

[0038]

[0039] in, Represents node v i The set of successor nodes, OCT(v i ,P m ,B n )express, Represents node v j Assigned to processor P x , and communicate with the successor node through B y When executing, the maximum optimistic execution time of its child nodes, Represents the communication task e i,j On Bus B n The communication overhead on v i and v j When mapped to the same processor, P m =P x ,but OCT(v i ,P m ,B n ) represents node v i The maximum optimistic execution time of the child nodes;

[0040] By averaging the priority of the OCT response tasks, the priority of each node is defined as follows:

[0041]

[0042] Cluster The priority is defined as follows:

[0043]

[0044] Further, in step S33, the earliest completion time The calculation formula is as follows:

[0045]

[0046] in, Represents node v j Average weight across processors.

[0047] Furthermore, in step S43, the following is specifically included: clustering the nodes that have been actually assigned The set of nodes that have not been actually allocated is recorded as W, assuming that the pre-allocated processor is P m , then processor P m Reserved time τ m,i It is expressed by the following formula:

[0048]

[0049] in, Represents node v i In the processor P m The time required for execution;

[0050] For processor P m , its total reservation time τ m for:

[0051]

[0052] By introducing τ m , virtual earliest end time The calculation formula is as follows:

[0053]

[0054] Among them, t· (v i ,P m ) represents node v i In the processor P m The actual end time of the execution.

[0055] The present invention also proposes a workflow scheduling system, characterized in that the workflow scheduling system executes the workflow scheduling method, including: a workflow DAG construction module, a clustering cluster set generation module, a clustering cluster CPU pre-allocation plan generation module and a workflow scheduling plan generation module;

[0056] The workflow DAG construction module constructs a workflow directed acyclic graph DAG;

[0057] The clustering cluster set generation module obtains a clustering cluster set according to the workflow directed acyclic graph DAG by using a center point-based DAG clustering method;

[0058] The cluster CPU pre-allocation scheme generating module obtains a CPU allocation scheme for each cluster based on the cluster set and the pre-allocation model;

[0059] The workflow scheduling scheme generating module obtains a final workflow scheduling scheme according to the workflow directed acyclic graph DAG and the CPU allocation scheme of each cluster cluster and based on the scheduling model of the virtual earliest end time.

[0060] The present invention further provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0061] Compared with the prior art, it has the following beneficial effects:

[0062] The workflow scheduling method of the present invention constructs a workflow directed acyclic graph DAG; according to the workflow directed acyclic graph DAG, a DAG clustering method based on a center point is adopted to obtain a cluster cluster set; based on the cluster cluster set, a CPU allocation scheme for each cluster cluster is obtained based on a pre-allocation model; according to the workflow directed acyclic graph DAG and the CPU allocation scheme for each cluster cluster, a final workflow scheduling scheme is obtained based on a scheduling model of a virtual earliest end time. The present invention adopts a DAG clustering method based on a center point to obtain a cluster cluster set, and uses the cluster cluster set to guide scheduling, effectively alleviating communication congestion, and obtaining a workflow scheduling scheme with better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0064] Figure 1 A schematic diagram of a workflow scheduling method provided by an embodiment of the present invention.

[0065] Figure 2 A schematic diagram of a workflow scheduling system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0068] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0069] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0070] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0071] The meaning of the characters is explained in the following table:

[0072]

[0073]

[0074] The present invention proposes a workflow scheduling method. Figure 1 As shown, the workflow scheduling method includes the following steps S1 to S4.

[0075] S1. Construct a directed acyclic graph (DAG) of the workflow; where DAG = (V, E), V = {v 1 ,v 2 ,…,v i} represents the set of i computing tasks in the workflow, E = {e 1 ,e 2 ,…,e k} represents the set of k communication tasks in the workflow, e k =e(i,j)∈E represents a communication task, reflecting the dependency between computing tasks. i Yes j Direct predecessor.

[0076] Since a computing task can have multiple predecessor nodes and successor nodes, we define and Respectively represent v i The predecessor node set and the successor node set.

[0077]

[0078] Accordingly, v i The set of adjacent edges of Contains connection v i All incoming and outgoing edges of :

[0079]

[0080] DAG has only one entry node v entry , the node satisfies Indicates that there is no predecessor node;

[0081] DAG also has only one exit node v exit , the node satisfies Indicates that there is no successor node.

[0082] Assume that the computation task v i and v j In processor P l and P m and there is a communication task between them k =e i,j If P l =P m , then e k Can be in v i Start directly after the execution ends. If P l ≠P m , then ek In addition, bus B is required n Able to fully perform communication tasks k The feasible time slice is denoted by B n You can execute k The earliest time is t ◇ (e k ,B n ). Thus we can get e k Earliest start time:

[0083]

[0084] When the task v i and v j When they are assigned to the same processor (i.e. P l =P m ), the communication between them is carried out in a shared memory manner without involving an external network. This communication method can significantly reduce communication delay and overhead, and the time consumed by this communication is negligible compared to communication through an external bus. k The earliest end time can be calculated as follows:

[0085]

[0086] For the computation task v i , he needs to wait until all input data arrives at its processor P m , and P m Ready to execute v i After that, you can start. m You can execute v i The earliest time is t ◇ (v i ,P m ). Then v i In P m The earliest start time on is:

[0087]

[0088] S2. According to the workflow directed acyclic graph DAG, a center point-based DAG clustering method is adopted to obtain a cluster set.

[0089] Traditional methods usually allocate resources based on greedy strategies, which further increases the possibility that they compete for the same resources at the same time. This situation causes the edges connected to their parent nodes (or child nodes) to be close in transmission time. When dispatched to the convergence point, the communication demand accumulated due to the greedy strategy suddenly explodes, significantly increasing the possibility of congestion. Communication congestion will cause processors to be idle. However, this processor idleness is exactly what the clustering method of the present invention is intended to optimize. The clustering method of the present invention is also proposed to address this type of problem that is widely present in real workflows.

[0090] On the other hand, as mentioned above, the communication demand usually increases sharply before and after the aggregation point scheduling, which means that the demand for communication resources varies significantly in different time periods. If the communication resource shortage in a specific period is alleviated by increasing the number of buses, it will inevitably lead to the situation that the communication resources are idle most of the time. In order to strike a balance between economy and resource utilization, and to avoid the idleness of computing units and communication units due to waiting for each other, it is the most reasonable choice to avoid communication congestion through optimization methods.

[0091] Communication congestion usually occurs at the convergence point. The essential reason is that the communication resources represented by the bus are difficult to handle the simultaneous communication needs in the time period before and after the scheduling convergence point. Inspired by this observation, the present invention proposes a natural and direct solution: scheduling nodes that need to transmit more data to each other to the same processor to eliminate the occupation of the bus by the corresponding high-occupancy communication tasks. The main purpose of the center-point-based DAG clustering method proposed in the present invention is to assign nodes in the same cluster to the same processor so that the communication between them does not need to be carried out through the bus, thereby alleviating the tension of communication resources.

[0092] The center point based DAG clustering method of the present invention needs to traverse the DAG in advance to obtain the whole graph structure information, retain this information in some way, and use this information in actual scheduling. In the present invention, clustering clusters are obtained through clustering strategies, and then the clustering clusters are used to guide scheduling, thereby obtaining a better scheduling solution.

[0093] According to the workflow directed acyclic graph DAG, the core idea of ​​obtaining the cluster set by using the center point based DAG clustering method is to select a part of the center nodes c according to the standard. i , and starting from these nodes, expand towards the parent node and child node respectively, so as to increase c i Corresponding cluster The number of nodes in; specifically including:

[0094] S21, construct a center point set C;

[0095] Furthermore, in step S21, the center point set C is defined as follows:

[0096]

[0097] in, Indicates v i The parameter α is used to adjust the strictness of the selection condition. B represents the bus, P represents the processor, and CCR represents the computation-to-communication ratio.

[0098] Specifically, the central node c i A specific in-degree criterion should be met. We use the graph properties of G = (V, E) and the computing environment ρ to determine this criterion and define the set of central points based on this criterion.

[0099] Furthermore, by adjusting the value of α, the lower limit of the number of neighbors of the selected node can be controlled. Increasing α will increase the lower limit of the number of neighbors, making the selection of set C more stringent; conversely, decreasing α will relax the selection criteria. By default, the value of α is 1.

[0100] S22, initialize the cluster corresponding to each node in the DAG

[0101] S23, traverse the central point set C, and in each traversal, calculate the nodes to be added to The threshold Θ i , and the center point corresponding to Assign the value to the center point c i , then call the clustering function to recursively add nodes, and finally get the cluster set;

[0102] For each central node c in C i , there is a corresponding cluster By calculation The average weight on all buses determines the node to be added to The threshold Θ i .

[0103] The calculation adds the node to The threshold Θ i , including: by calculating The average weight on all buses determines the node to be added to The threshold Θ i , Θ i The calculation method is as follows:

[0104]

[0105] in, Indicates vi The set of adjacent edges of Indicates e k The average weight on the bus.

[0106] Furthermore, in step S23, the calling function clustering is used to recursively add nodes.

[0107] Finally, we get a set of clusters, specifically:

[0108] In the clustering function, all predecessor nodes of node node are traversed If the predecessor node has been visited, skip the node; calculate the average edge weight of the node node and the corresponding predecessor If it is greater than the threshold Θ, the predecessor node pred corresponding to Assigned to Then continue to recursively call through this node; otherwise, the node corresponding to the predecessor node The value is -1, which means that the node will not be clustered into any cluster;

[0109] Traverse all the successor nodes of node node If the successor node has been visited, skip the node; calculate the average edge weight of the node node and the corresponding successor. If it is greater than the threshold Θ, the node corresponding to the successor node succ is accessed. Assigned to Then continue the recursive call through this node; otherwise, the node corresponding to the successor node The value is -1, which means that the node will not be clustered into any cluster;

[0110] Each node v i of The properties have been assigned values; have the same The values ​​belong to the same cluster; if A value of -1 indicates that the node is not assigned to any cluster.

[0111] S3, based on the cluster set and the pre-allocation model, obtaining a CPU allocation scheme for each cluster; specifically including:

[0112] S31. Calculate each cluster Priority Based on the priority, all clusters are sorted in descending order;

[0113] Furthermore, in step S31, each cluster is calculated Priority The specific steps are:

[0114] In a heterogeneous environment, OCT (Optimistic Cost Table) predicts the maximum value of the shortest path from each node to the exit node among all node subtasks. The calculation of OCT depends not only on the node itself, but also on the choice of processor. The OCT of a node in G on a specific processor needs to be traversed recursively from the exit node to the entry node.

[0115] First, OCT (v i ,P m ,B n ) is calculated as follows:

[0116]

[0117] in, Represents node v i The set of successor nodes, OCT(v j ,P x ,B y )express, Represents node v j Assigned to processor P x , and communicate with the successor node through B y When executing, the maximum optimistic execution time of its child nodes, Represents the communication task e i,j On Bus B n The communication overhead on v i and v j When mapped to the same processor, P m =P x ,but OCT(v i ,P m ,B n ) represents node v i The maximum optimistic execution time of the child nodes of ; it takes into account the processing units and communication units that can minimize the execution time;

[0118] By averaging the priority of the OCT response tasks, the priority of each node is defined as follows:

[0119]

[0120] Cluster The priority is defined as follows:

[0121]

[0122] After the clustering phase, we get a set of clusters In this phase, we will Each cluster in Allocate processors based on processor availability and execution of the current cluster The time required to calculate the estimated earliest completion time and minimize Assigns a processor to a guide.

[0123] Further, in step S31, the earliest completion time The calculation formula is as follows:

[0124]

[0125] in, Represents node v j Average weight across processors.

[0126] S32, each processor P m The occupied time Ω m Initialized to 0;

[0127] S33. For each cluster Calculate its r Estimated earliest finish time on Assign it to minimize The processor and updates the corresponding processor P accordingly m Ω m .

[0128] S4, according to the workflow directed acyclic graph DAG and the CPU allocation plan of each cluster, based on the scheduling model of the virtual earliest end time, obtain the final workflow scheduling plan;

[0129] To quantify the impact of the remaining nodes that will be assigned to the current processor in the future, we introduce the variable τ m , indicating the processor P m Reservation Time. After any node in is actually scheduled (i.e. the start time is determined), the corresponding CPU will be All nodes in the cluster reserve time slices. If one of the nodes is actually scheduled, the corresponding processor will no longer reserve a time slice for the node.

[0130] Step S4 specifically includes:

[0131] S41. Generate a task list sorted in non-increasing order according to task priority rank

[0132] S42, initializing the available time of all resources to 0;

[0133] S43, in each iteration, selecting the first task node from the task list for scheduling, and synchronously scheduling its predecessor communication task, and when the scheduling of the task node is completed, deleting it from the task list;

[0134] The steps of each iteration are: i Generate a priority list of all its immediate predecessor tasks And sort in non-increasing order according to rank; if the current node has been pre-allocated, update the reservation time τ m Otherwise, v i Temporarily assigned to each processor P r ∈P;

[0135] For each such temporary assignment, each predecessor communication task e k Temporarily assigned to bus B according to priority k ∈B, so that the communication task e k The completion time t · (e k ) minimum;

[0136] Based on the distribution results of these communication tasks, calculate all processors P m Virtual earliest end time on v i is actually allocated to make it On the smallest processor;

[0137] In v i After the processor is determined, its predecessor communication task is assigned to the bus that makes the communication task end earliest, and the corresponding bus is updated to minimize their end time.

[0138] Furthermore, in step S43, the following is specifically included: clustering the nodes that have been actually assigned The set of nodes that have not been actually allocated is recorded as W, assuming that the pre-allocated processor is P m , then processor P m Reserved time τ m,i It is expressed by the following formula:

[0139]

[0140] in, Represents node v i In the processor P m The time required for execution;

[0141] For processor Pm , its total reservation time τ m for:

[0142]

[0143] By introducing τ m , virtual earliest end time The calculation formula is as follows:

[0144]

[0145] Among them, t · (v i ,P m ) represents node v i In the processor P m The actual end time of the execution.

[0146] Figure 2 A workflow scheduling system is provided in an embodiment of the present invention, and the system executes the workflow scheduling method. Figure 2 As shown, the workflow scheduling system includes: a workflow DAG building module, a clustering cluster set generation module, a clustering cluster CPU pre-allocation plan generation module and a workflow scheduling plan generation module;

[0147] The workflow DAG construction module constructs a workflow directed acyclic graph DAG;

[0148] The clustering cluster set generation module obtains a clustering cluster set according to the workflow directed acyclic graph DAG by using a center point-based DAG clustering method;

[0149] The cluster CPU pre-allocation scheme generating module obtains a CPU allocation scheme for each cluster based on the cluster set and the pre-allocation model;

[0150] The workflow scheduling scheme generating module obtains a final workflow scheduling scheme according to the workflow directed acyclic graph DAG and the CPU allocation scheme of each cluster cluster and based on the scheduling model of the virtual earliest end time.

[0151] The above workflow scheduling system can be implemented in the form of a computer program, which can be run on a computer device.

[0152] The computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0153] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, a processor can execute a workflow scheduling method.

[0154] The processor is used to provide computing and control capabilities to support the operation of the entire computer equipment.

[0155] The internal memory provides an environment for the operation of a computer program in a non-volatile storage medium. When the computer program is executed by a processor, the processor can execute a workflow scheduling method.

[0156] The network interface is used to communicate with other devices over the network. Those skilled in the art will appreciate that the above-mentioned computer device structure is only a partial structure related to the present application solution, and does not constitute a limitation on the computer device to which the present application solution is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] The processor is used to run a computer program stored in the memory, and the program implements the workflow scheduling method described in the first embodiment.

[0158] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0159] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0160] The present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes a workflow scheduling method described in the first embodiment.

[0161] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, or any other computer-readable storage medium that can store program codes.

[0162] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0164] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0165] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0166] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A workflow scheduling method, characterized in that: Includes steps: S1. Construct a directed acyclic graph (DAG) of the workflow; where DAG = (V, E), V = {v1, v2, ..., v i } represents the set of i computing tasks in the workflow, E = {e1, e2, ..., e k } represents the set of k communication tasks in the workflow, e k =e(i, j)∈E represents a communication task, reflecting the dependency between computing tasks; S2. According to the workflow directed acyclic graph DAG, a DAG clustering method based on a center point is adopted to obtain a cluster set; specifically including: S21, construct a center point set C; S22, initialize the cluster corresponding to each node in the DAG S23, traverse the central point set C, and in each traversal, calculate the nodes to be added to The threshold Θ i , and the center point corresponding to Assign the value to the center point c i , then call the clustering function to recursively add nodes, and finally get the cluster set; S3, based on the cluster set and the pre-allocation model, obtaining a CPU allocation scheme for each cluster; specifically including: S31. Calculate each cluster Priority Based on the priority, all clusters are sorted in descending order; S32, each processor P m The occupied time Ω m Initialized to 0; S33. For each cluster Calculate its r Estimated earliest finish time on Assign it to minimize The processor and updates the corresponding processor P accordingly m Ω m ; S4, according to the workflow directed acyclic graph DAG and the CPU allocation plan of each cluster, based on the scheduling model of the virtual earliest end time, obtain the final workflow scheduling plan; specifically including: S41. Generate a task list sorted in non-increasing order according to task priority rank S42, initializing the available time of all resources to 0; S43, in each iteration, selecting the first task node from the task list for scheduling, and synchronously scheduling its predecessor communication task, and when the scheduling of the task node is completed, deleting it from the task list; The steps of each iteration are: i Generate a priority list of all its immediate predecessor tasks And sort in non-increasing order according to rank; if the current node has been pre-allocated, update the reservation time τ m Otherwise, v i Temporarily assigned to each processor P r ∈P; For each such temporary assignment, each predecessor communication task e k Temporarily assigned to bus B according to priority k ∈B, so that the communication task e k The completion time t.(e k ) minimum; Based on the distribution results of these communication tasks, calculate all processors P m Virtual earliest end time on v i is actually allocated to make it On the smallest processor; In v i After the processor is determined, its predecessor communication task is assigned to the bus that makes the communication task end earliest, and the corresponding bus is updated to minimize their end time.

2. The method according to claim 1, characterized in that In step S21, the center point set C is defined as follows: in, Indicates v i The parameter α is used to adjust the strictness of the selection condition. B represents the bus, P represents the processor, and CCR represents the computation-to-communication ratio.

3. The method according to claim 1, characterized in that In step S23, the calculation adds the node to The threshold Θ i , specifically including: By calculation The average weight on all buses determines the node to be added to The threshold Θ i , Θ i The calculation method is as follows: in, Indicates v i The set of adjacent edges of Indicates e k The average weight on the bus.

4. The method according to claim 1, characterized in that: In step S23, the calling function clustering is used to recursively add nodes, and finally a cluster set is obtained, specifically: In the clustering function, all predecessor nodes of node node are traversed If the predecessor node has been visited, skip the node; calculate node nod e and the average edge weight of the corresponding predecessor If it is greater than the threshold Θ, the predecessor node pred corresponding to Assigned to Then continue to recursively call through this node; otherwise, the node corresponding to the predecessor node The value is -1, which means that the node will not be clustered into any cluster; Traverse all the successor nodes of node node If the successor node has been visited, skip the node; calculate node nod e and the average edge weight of the corresponding successor. If it is greater than the threshold Θ, the successor node succ corresponding to Assigned to Then continue the recursive call through this node; otherwise, the node corresponding to the successor node The value is -1, which means that the node will not be clustered into any cluster; Each node v i of The properties have been assigned values; have the same The values ​​belong to the same cluster; if A value of -1 indicates that the node is not assigned to any cluster.

5. The method according to claim 1, characterized in that In step S31, each cluster is calculated Priority The specific steps are: First, OCT (v i , P m , B n ) is calculated as follows: in, Represents node v i The set of successor nodes, OCT(v i , P m , B n )express, Represents node v j Assigned to processor P x , and communicate with the successor node through B y When executing, the maximum optimistic execution time of its child nodes, Represents the communication task e i，j On Bus B n The communication overhead on v i and v j When mapped to the same processor, P m =P x ,but OCT(v i , P m , B n ) represents node v i The maximum optimistic execution time of the child nodes; By averaging the priority of the OCT response tasks, the priority of each node is defined as follows: Cluster The priority is defined as follows:

6. The method according to claim 1, characterized in that In step S33, the earliest completion time The calculation formula is as follows: in, Represents node v j Average weight across processors.

7. The method according to claim 1, characterized in that Step S43 specifically includes: clustering the nodes that have been actually assigned The set of nodes that have not been actually allocated is recorded as W, assuming that the pre-allocated processor is P m , then processor P m Reserved time τ m，i It is expressed by the following formula: in, Represents node v i In the processor P m The time required for execution; For processor P m , its total reservation time τ m for: By introducing τ m , virtual earliest end time The calculation formula is as follows: Among them, t.(v i , P m ) represents node v i In the processor P m The actual end time of the execution.

8. A workflow scheduling system, characterized in that: The workflow scheduling system executes the workflow scheduling method according to claim 1, comprising: a workflow DAG construction module, a clustering cluster set generation module, a clustering cluster CPU pre-allocation scheme generation module and a workflow scheduling scheme generation module; The workflow DAG construction module constructs a workflow directed acyclic graph DAG; The clustering cluster set generation module obtains a clustering cluster set according to the workflow directed acyclic graph DAG by using a center point-based DAG clustering method; The cluster CPU pre-allocation scheme generating module obtains a CPU allocation scheme for each cluster based on the cluster set and the pre-allocation model; The workflow scheduling scheme generating module obtains a final workflow scheduling scheme according to the workflow directed acyclic graph DAG and the CPU allocation scheme of each cluster cluster and based on the scheduling model of the virtual earliest end time.

9. A computer device, characterized in that: The device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images