Power-aware multi-objective online scheduling method for dynamic DAG workflows

Abstract

The application discloses a power-aware multi-objective online scheduling method for dynamic DAG workflow, comprising the following steps: obtaining a workflow set and a resource set; generating a ready task set dynamically according to a workflow arrival time, a task release time and a task interdependence completion condition; calculating a criticality index of each task of a DAG of each workflow, and clustering the tasks in the DAG based on a task feature to obtain a plurality of task clusters; in the ready task set, selecting a target cluster according to a cluster score, and selecting a target task in the target cluster according to the criticality index; and calculating a completion time, an energy consumption, a resource unavailability and a brownout energy consumption of the target task on each heterogeneous machine, and selecting a target machine to execute the target task according to a multi-objective cost function. The application can guarantee or improve the completion time, reduce the brownout energy consumption and carbon emission, and improve the resource utilization rate.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and distributed system task scheduling technology, specifically to a power sensing multi-objective online scheduling method oriented towards dynamic DAG workflow. Background Technology

[0002] In cloud data centers, edge computing, and heterogeneous cluster environments, scientific workflows and industrial data processing often use Directed Acyclic Graphs (DAGs) to describe the sequential dependencies between tasks. As business scales up, workflows exhibit a dynamic nature of "online arrival and online visibility": firstly, tasks may have a release time, meaning they can only be scheduled after dependencies are satisfied and the release time is reached; secondly, multiple workflows can enter the system successively at different arrival times. Simultaneously, computing resources are often deployed across sites, and the proportion of renewable energy supply at each site varies over time, resulting in significant differences in the "green electricity ratio, brown electricity consumption, and carbon emissions" of the same task when executed at different time periods and sites.

[0003] Existing classic DAG scheduling algorithms (such as HEFT and LIST) primarily optimize completion time (makespan) or critical path, making it difficult to simultaneously address multiple objectives such as energy consumption / carbon emissions and resource utilization. Even energy-aware scheduling strategies often employ static energy consumption or static carbon intensity models, failing to capture the time-varying characteristics of the site-level green electricity proportion g(t). Furthermore, frequent rescheduling in dynamic environments can lead to scheduling jitter, which in turn reduces system performance and stability.

[0004] Therefore, there is a need for an online scheduling method that can comprehensively optimize completion time, energy consumption, resource utilization, and brown electricity consumption (or carbon emissions) on heterogeneous multi-site resources for dynamic DAG workflows, and has a robust dynamic rescheduling mechanism. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a power sensing multi-objective online scheduling method for dynamic DAG workflows. This method enables the reduction of palm oil energy consumption and carbon emissions, as well as the improvement of resource utilization, in scenarios involving dynamic DAG workflows and multiple workflows arriving online, while ensuring or improving completion time and avoiding jitter caused by frequent rescheduling.

[0006] To achieve the aforementioned objectives, the present invention employs the following technical solution: a power sensing multi-objective online scheduling method oriented towards dynamic DAG workflow, comprising the following steps:

[0007] S1: Obtain the workflow set and resource set. The workflow is represented by a DAG, where nodes are tasks and edges are dependencies between tasks. The resource set contains multiple heterogeneous machines bound to a site. Each site is configured with a green electricity percentage curve, green electricity carbon intensity, and brown electricity carbon intensity.

[0008] S2: During event-driven online scheduling, a set of ready tasks is dynamically generated based on workflow arrival time, task release time, and task dependency completion status.

[0009] S3: Calculate the key metrics for each task in the DAG of each workflow, and cluster the tasks in the DAG based on task characteristics to obtain multiple task clusters;

[0010] S4: In the set of ready tasks, select a target cluster based on the cluster score, and select a target task within the target cluster based on key indicators;

[0011] S5: Calculate the completion time, energy consumption, resource unavailability and brown electricity consumption of the target task on each heterogeneous machine, and select the target machine to execute the target task according to the multi-objective cost function to realize power-aware multi-objective online scheduling for dynamic DAG workflow.

[0012] Furthermore, the task includes a runtime and a release time;

[0013] The heterogeneous machine is also equipped with processing speed, DVFS frequency, and power consumption model parameters.

[0014] Furthermore, in step S3, a reverse topological traversal is performed on the DAG of each workflow to calculate the key performance indicators (KPIs) for each task. If a task has no successor, then the KPIs are... If the task has follow-up tasks, then the key performance indicators (KPIs) should be... for:

[0015]

[0016] in, The computation time for the task itself, Let be the set of successor tasks of a given task in a DAG. This is a critical value for subsequent tasks.

[0017] Furthermore, the task characteristics include task runtime, in-degree, out-degree, and topology depth.

[0018] Furthermore, the cluster score The calculation formula is:

[0019]

[0020] in, For the different clusters after clustering, The three weighting coefficients for cluster scoring, It is the largest key indicator within the cluster. This is the sum of the execution times of tasks within the cluster. This is the sum of the estimated energy consumption of tasks within the cluster.

[0021] Furthermore, the multi-objective cost function for:

[0022]

[0023]

[0024] in, For multi-objective weighting coefficients, For normalization function, This represents the completion time of the task on the corresponding machine. For energy consumption, For resource unavailability, For palm oil energy consumption, This represents the average percentage of green electricity generated during the task execution period.

[0025] Furthermore, the average percentage of green electricity within the task execution interval The average value is obtained by integrating or discretely sampling the piecewise constant curve of the station within the task execution interval.

[0026] Furthermore, the resource unavailability The future availability of the machine is estimated through a short time window, and the calculation formula is as follows:

[0027]

[0028]

[0029] in, The estimated busy or unavailable duration within the window. The time window length for utilizing the agent. This is a truncation function. The next available time for a particular machine. This represents the current simulation moment.

[0030] Furthermore, the method also includes:

[0031] The system monitors the rate of change of the number of tasks in the ready task set of each workflow in real time. When the rate of change exceeds a preset threshold and the time since the last clustering exceeds a preset cooldown time, it triggers the re-clustering of the workflow and updates the cluster score cache to complete dynamic rescheduling.

[0032] The beneficial effects of this invention are:

[0033] (1) Green electricity perception: Explicitly model the time-varying characteristics of the green electricity ratio g(t) of the site, and use brown electricity consumption as one of the optimization targets, so as to reduce brown electricity consumption and carbon emissions under the same or similar completion time.

[0034] (2) Multi-objective trade-offs: A unified multi-objective cost function enables adjustable trade-offs between completion time, energy consumption, resource unavailability and brown electricity consumption, adapting to different business preferences.

[0035] (3) Cluster-level hierarchical decision-making: The decision space is reduced by using a hierarchical strategy of "cluster priority + intra-cluster criticality priority", which improves online scheduling efficiency and enhances the focus on critical paths.

[0036] (4) Robust dynamic rescheduling: The threshold triggering + cooldown time mechanism suppresses the jitter caused by frequent rescheduling, thereby improving scheduling stability and repeatability.

[0037] (5) Scalability and feasibility: This method can be implemented on an event-driven scheduler and can reduce redundant calculations by caching criticality and cluster scores, making it suitable for large-scale DAG instances. Attached Figure Description

[0038] Figure 1 This is a flowchart of the power sensing multi-objective online scheduling method based on dynamic DAG workflow of the present invention.

[0039] Figure 2 This is a schematic diagram of the green electricity ratio g(t) and effective carbon intensity CI(t) at the site level.

[0040] Figure 3 Pareto plots for (energy consumption, completion time) under three different scale tests.

[0041] Figure 4 Energy consumption vs. carbon emissions: a comparison chart of the improvement in performance under different W4 (green electricity percentage) weightings. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, a power sensing multi-objective online scheduling method for dynamic DAG workflow includes the following steps:

[0044] S1: Obtain the workflow set and resource set. The workflow is represented by a DAG, where nodes are tasks and edges are dependencies between tasks. The resource set contains multiple heterogeneous machines bound to a site. Each site is configured with a green electricity percentage curve, green electricity carbon intensity, and brown electricity carbon intensity.

[0045] The task includes a runtime and a release time;

[0046] The heterogeneous machine is also equipped with processing speed, DVFS frequency, and power consumption model parameters.

[0047] In this embodiment, the DVFS power consumption model is used:

[0048] Dynamic power ;

[0049] Idle power ;

[0050] Task execution time ;

[0051] Task energy consumption ;

[0052] in, and For power consumption model parameters, For DVFS frequency, For runtime, For processing speed, The "time slot / duration" assigned to the task. Desirable For simplification, the average green electricity ratio of the task within the interval is... If the mean is obtained by piecewise constant exact integration or discrete sampling, then:

[0053] ,

[0054]

[0055] in, For green electricity consumption, For palm oil energy consumption, This is the sum of the estimated energy consumption of tasks within the cluster. For total carbon emissions, For green carbon intensity, The strength of the brown carbon electrode.

[0056] S2: During event-driven online scheduling, a set of ready tasks is dynamically generated based on workflow arrival time, task release time, and task dependency completion status.

[0057] For any workflow Task Its earliest schedulable time is defined as:

[0058]

[0059] in, For workflow arrival time, Release time for the task. For the precursor mission set, This refers to the completion time of the precursor mission.

[0060] At any moment If the task Simultaneously satisfying: 1) The mission has not yet started and has not been completed; 2) All precursor missions have been completed; 3) (Numerical tolerance may be considered);

[0061] Then the task Add to ready task set and record it As a scheduling input.

[0062] when If the time is empty and there are still unfinished tasks, the system advances the time to the minimum time when a ready task may appear. (For example, due to release time or dependency completion), then regenerate. This is to avoid simulation stagnation.

[0063] S3: Calculate the key metrics for each task in the DAG of each workflow, and cluster the tasks in the DAG based on task characteristics to obtain multiple task clusters;

[0064] Perform a reverse topological traversal of the DAG for each workflow, calculate the key performance indicators (KPIs) for each task, and if a task has no successor, then the key performance indicators are... If the task has follow-up tasks, then the key performance indicators (KPIs) should be... for:

[0065]

[0066] in, The computation time for the task itself, Let be the set of successor tasks of a given task in a DAG. This is a critical value for subsequent tasks.

[0067] Key indicators specifically include:

[0068] Makespan: Makespan refers to the total time it takes for all tasks to complete, and it is a core metric for scheduling performance. The formula is:

[0069]

[0070] Total Energy: The total energy consumed by the system during task execution. It can be calculated using weighted energy consumption, as shown in the formula:

[0071]

[0072] in, For the first The energy consumption of each task, for each machine, is calculated using the following formula based on its dynamic power factor and static power factor:

[0073]

[0074] Green Ratio: This measures the proportion of green electricity used by the system during task execution. The use of green electricity helps reduce carbon emissions. The calculation formula is:

[0075]

[0076] Carbon Emission: Calculate the energy consumption and carbon emissions of green electricity and gray electricity based on the ratio of green electricity to gray electricity and the carbon emission factor.

[0077]

[0078]

[0079] in, This represents the average percentage of green electricity used by the task within the execution interval [start, finish].

[0080] Resource utilization: This measures the usage of each resource (such as a machine). High or low resource utilization directly impacts scheduling efficiency. It can be calculated using the following formula:

[0081]

[0082] in, The effective runtime of a machine within a certain period of time. This represents the total running time.

[0083] The task characteristics include task execution time, in-degree, out-degree, and topology depth.

[0084] S4: In the set of ready tasks, select a target cluster based on the cluster score, and select a target task within the target cluster based on key indicators;

[0085] Cluster score The calculation formula is:

[0086]

[0087] in, For the different clusters after clustering, The three weighting coefficients for cluster scoring, It is the largest key indicator within the cluster. This is the sum of the execution times of tasks within the cluster. This is the sum of the estimated energy consumption of tasks within the cluster.

[0088] During scheduling, the cluster with the highest cluster score is selected from the clusters to which the ready tasks belong, and the ready tasks with the highest critical indicators are selected from that cluster.

[0089] S5: Calculate the completion time, energy consumption, resource unavailability and brown electricity consumption of the target task on each heterogeneous machine, and select the target machine to execute the target task according to the multi-objective cost function to realize power-aware multi-objective online scheduling for dynamic DAG workflow.

[0090] The multi-objective cost function for:

[0091]

[0092]

[0093] in, For multi-objective weighting coefficients, For normalization function, This represents the completion time of the task on the corresponding machine. For energy consumption, For resource unavailability, For palm oil energy consumption, This represents the average percentage of green electricity generated during the task execution period.

[0094] The average percentage of green electricity within the task execution period The average value is obtained by integrating or discretely sampling the piecewise constant curve of the station within the task execution interval.

[0095] The resource unavailability The future availability of the machine is estimated through a short time window, and the calculation formula is as follows:

[0096]

[0097]

[0098] in, The estimated busy or unavailable duration within the window. The time window length for utilizing the agent. This is a truncation function. The next available time for a particular machine. This represents the current simulation moment.

[0099] Choose to make ( , The machine with the smallest lexicographical order is used as the mapping result.

[0100] The method further includes:

[0101] The system monitors the rate of change of the number of tasks in the ready task set of each workflow in real time. When the rate of change exceeds a preset threshold and the time since the last clustering exceeds a preset cooldown time, it triggers the re-clustering of the workflow and updates the cluster score cache to complete dynamic rescheduling, so as to adapt to changes in online visibility and suppress frequent rescheduling.

[0102] In one embodiment of the present invention, Figure 2 This diagram illustrates the relationship between the green electricity share g(t) and effective carbon intensity CI(t) at the site level. The results show that site qh has a higher green electricity share and lower carbon intensity for most of the time period, while site grid exhibits the opposite. Therefore, the scheduling algorithm needs to strike a balance between performance and green / low-carbon performance.

[0103] Figure 3 Pareto graphs (energy consumption, completion time) under three different scale experiments are presented. By comparing the algorithm of this invention (ECDS), First-Come, First-Served (FCFS) scheduling, Heterogeneous Earliest Finish Time (HEFT) scheduling algorithm, and List scheduling algorithm (LIST), under three different scale DAG experiments, the ECDS algorithm designed in this invention can achieve lower carbon emissions and energy consumption at the cost of sacrificing the smallest possible completion time. This improvement is particularly prominent in small and medium scale experiments.

[0104] Figure 4 The chart compares the performance improvement under different w4 (green electricity percentage) weights for energy consumption and carbon emissions. It can be seen that the ECDS algorithm proposed in this invention outperforms existing classic algorithms in terms of both total energy consumption and carbon emissions.

[0105] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the invention.

Claims

1. A power sensing multi-objective online scheduling method oriented towards dynamic DAG workflow, characterized in that, Includes the following steps: S1: Obtain the workflow set and resource set. The workflow is represented by a DAG, where nodes are tasks and edges are dependencies between tasks. The resource set contains multiple heterogeneous machines bound to a site. Each site is configured with a green electricity percentage curve, green electricity carbon intensity, and brown electricity carbon intensity. S2: During event-driven online scheduling, a set of ready tasks is dynamically generated based on workflow arrival time, task release time, and task dependency completion status. S3: Calculate the key metrics for each task in the DAG of each workflow, and cluster the tasks in the DAG based on task characteristics to obtain multiple task clusters; S4: In the set of ready tasks, select a target cluster based on the cluster score, and select a target task within the target cluster based on key indicators; S5: Calculate the completion time, energy consumption, resource unavailability and brown electricity consumption of the target task on each heterogeneous machine, and select the target machine to execute the target task according to the multi-objective cost function to realize power-aware multi-objective online scheduling for dynamic DAG workflow.

2. The power sensing multi-objective online scheduling method for dynamic DAG workflow as described in claim 1, characterized in that, The task includes a runtime and a release time; The heterogeneous machine is also equipped with processing speed, DVFS frequency, and power consumption model parameters.

3. The power sensing multi-objective online scheduling method for dynamic DAG workflow as described in claim 1, characterized in that, In step S3, a reverse topological traversal is performed on the DAG of each workflow, and the key performance indicators (KPIs) of each task are calculated. If a task has no successor, then the key performance indicators are... ; If the task has follow-up tasks, then the key performance indicators (KPIs) for: in, The computation time for the task itself, Let be the set of successor tasks of a given task in a DAG. This is a critical value for subsequent tasks.

4. The power sensing multi-objective online scheduling method for dynamic DAG workflow as described in claim 1, characterized in that, The task characteristics include task execution time, in-degree, out-degree, and topology depth.

5. The power sensing multi-objective online scheduling method for dynamic DAG workflow as described in claim 1, characterized in that, Cluster score The calculation formula is: in, For the different clusters after clustering, The three weighting coefficients for cluster scoring, It is the largest key indicator within the cluster. This is the sum of the execution times of tasks within the cluster. This is the sum of the estimated energy consumption of tasks within the cluster.

6. The power sensing multi-objective online scheduling method for dynamic DAG workflow according to claim 1, characterized in that, The multi-objective cost function for: in, For multi-objective weighting coefficients, For normalization function, This represents the completion time of the task on the corresponding machine. For energy consumption, For resource unavailability, For palm oil energy consumption, This represents the average percentage of green electricity generated during the task execution period.

7. The power sensing multi-objective online scheduling method for dynamic DAG workflow as described in claim 6, characterized in that, The average percentage of green electricity within the task execution period The average value is obtained by integrating or discretely sampling the piecewise constant curve of the station within the task execution interval.

8. The power sensing multi-objective online scheduling method for dynamic DAG workflow according to claim 6, characterized in that, The resource unavailability The future availability of the machine is estimated through a short time window, and the calculation formula is as follows: in, The estimated busy or unavailable duration within the window. The time window length for utilizing the agent. This is a truncation function. The next available time for a particular machine. This represents the current simulation moment.

9. The power sensing multi-objective online scheduling method for dynamic DAG workflow according to any one of claims 1-8, characterized in that, The method further includes: The system monitors the rate of change of the number of tasks in the ready task set of each workflow in real time. When the rate of change exceeds a preset threshold and the time since the last clustering exceeds a preset cooldown time, it triggers the re-clustering of the workflow and updates the cluster score cache to complete dynamic rescheduling.

Images