Task scheduling method and device
Through the improved whale algorithm, based on energy consumption cost optimization task scheduling, the problem of high energy consumption in cluster nodes during peak time is solved, and the effect of resource balance and energy consumption reduction is achieved.
Patent Information
- Application Number
- CN202411340156.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-12-27
AI Technical Summary
In the era of big data, cluster nodes may experience task blockage during peak hours, resulting in a surge in node utilization and energy consumption increase. The existing task scheduling algorithm is difficult to effectively optimize, resulting in a higher overall energy consumption.
A task scheduling method is adopted to randomly allocate tasks to cluster nodes and calculate energy consumption costs based on energy consumption costs. The optimal execution node of each task is determined using the improved whale algorithm (WOA) and prioritizes the energy consumption. On low nodes, to achieve resource scheduling balance and energy consumption reduction.
By using energy consumption costs as the evaluation standard, we optimize task scheduling, significantly reduce the energy consumption costs of the cluster, improve resource utilization, and avoid the problems of unbalanced node resources and excessive energy consumption.
Smart Images
Figure CN119201396A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a task scheduling method and device. Background Art
[0002] With the surge in data volume and the advent of the big data era, the components of big data and cloud computing are becoming more and more abundant, such as batch processing components (Hive, Spark) and stream processing components (Storm, Spark Streaming, Flink Stream). More and more people are beginning to pay attention to cluster task scheduling. Since nodes may be blocked by tasks during peak hours, resulting in a surge in node utilization, the energy consumption of the entire cluster will increase significantly, so task scheduling is a crucial link.
[0003] In view of this, how to optimize task scheduling and reduce energy consumption is a technical problem that needs to be solved urgently. Summary of the invention
[0004] The present application provides a task scheduling method and device for reducing energy consumption.
[0005] In a first aspect, an embodiment of the present application provides a task scheduling method, which is applied to any electronic device with processing capability, and the method includes:
[0006] Randomly obtain N nodes in the cluster and randomly assign N tasks to the N nodes; each of the N tasks corresponds to a node on the N nodes;
[0007] Repeat the following steps until the number of repetitions reaches the threshold:
[0008] Calculate the energy consumption cost of each task executed on a corresponding node of the task according to preset rules;
[0009] Generate a first parameter, select a target formula from a preset formula for updating a corresponding node of each task using the first parameter, and determine a new corresponding node for each task from all nodes in the cluster based on the target formula; the new corresponding node for each task is different from a previous corresponding node for each task;
[0010] Calculate the energy consumption cost of each task executed on a new corresponding node of the task. If the energy consumption cost of each task executed on a new corresponding node of the task is less than the energy consumption cost of each task executed on the previous corresponding node of the task, save the new corresponding node of each task; otherwise, save the previous corresponding node of each task.
[0011] Output the latest corresponding node of each task when the number of repetitions reaches the threshold, and assign each task to the latest corresponding node for execution.
[0012] In this method, the energy consumption cost of each task executed on a corresponding node of the task is used as a criterion for evaluating the task scheduling scheme, which can maximize the utilization of cluster resources; and the whale algorithm is improved based on the energy consumption cost, so that more tasks can be placed preferentially on nodes with lower energy consumption costs, making the cluster resource scheduling more balanced, thereby reducing the energy consumption cost of the cluster.
[0013] Optionally, the energy consumption is used to indicate the CPU consumption and memory consumption of each task executed on a corresponding node of the task; the energy consumption cost of each task executed on a corresponding node of the task is calculated according to a preset rule, including:
[0014] Get the CPU usage, memory usage, and time required to complete each task when it is executed on a corresponding node of the task;
[0015] Based on the CPU usage rate and the memory usage rate, the energy consumption per unit time when each task is executed on a corresponding node of the task is calculated according to the first rule;
[0016] Based on the energy consumption per unit time when each task is executed on a corresponding node of the task and the time required to complete the task execution, the energy consumption cost of each task executed on a corresponding node of the task is calculated according to the second rule.
[0017] In this method, since the energy consumption when a task is executed on a node is mainly the energy consumption of the CPU and memory, the energy consumption cost of each task on a corresponding node of the task is calculated based on the CPU utilization rate and the memory utilization rate, so as to facilitate the subsequent task scheduling process to determine the optimal corresponding node for each task with the purpose of reducing the energy consumption cost.
[0018] Optionally, the first rule is as follows:
[0019] E=C0+C1*U cpu +C2*U mem
[0020] Where E represents the energy consumption per unit time when each task is executed on a corresponding node of the task, C0, C1 and C2 are constants, and U cpu Indicates the CPU usage of each task when it is executed on a corresponding node of the task, U mem Indicates the memory usage of each task when it is executed on a corresponding node of the task.
[0021] In this method, the energy consumption per unit time when each task is executed on a corresponding node of the task can be counted as the CPU and memory consumed per unit time when each task is executed on a corresponding node of the task. The energy consumption of each task on a corresponding node of the task is calculated, which facilitates the subsequent use of the energy consumption cost as a basis for selecting the corresponding node, thereby reducing the energy consumption cost of the cluster.
[0022] Optionally, the second rule is as follows:
[0023] Cost energy =E×P×t
[0024] Among them, Cost energy It represents the energy consumption cost of each task executed on a corresponding node of the task, P represents the energy consumption billing standard per unit time, and t represents the time required for each task to complete the task execution on a corresponding node of the task.
[0025] In this solution, based on the energy consumption per unit time when each task is executed on a corresponding node of the task, the energy consumption cost of each task executed on a corresponding node of the task is calculated according to the energy consumption billing standard of the cluster and the time required to complete the task execution, so as to facilitate the subsequent use of the energy consumption cost as the basis for selecting the corresponding node, thereby reducing the energy consumption cost of the cluster.
[0026] Optionally, obtain the CPU usage, memory usage, and time required to complete the task execution when each task is executed on a corresponding node of the task, including:
[0027] Check whether there is a historical record of each task being executed on a corresponding node of the task;
[0028] If it exists, get the historical CPU usage, memory usage, and time required to complete the execution of each task when it is executed on a corresponding node of the task;
[0029] If it does not exist, the CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task are predicted based on the historical records of the execution of each task on non-corresponding nodes and the information of each corresponding node.
[0030] In this method, based on the historical records of each task's execution, the historical CPU usage, memory usage, and time required to complete the execution of each task when it is executed on a corresponding node of the task can be directly obtained or predicted. There is no need to put each task on its corresponding node for execution. Obtaining specific data during execution can improve the efficiency of obtaining data, and thus improve the efficiency of subsequent calculation of energy consumption costs.
[0031] Optionally, the first parameter includes a random number p and a coefficient vector A; the value of the random number p is not less than 0 and less than 1, and the coefficient vector A=2a×r1-a, wherein a is a vector that linearly decreases from 2 to 0, and r1 is a random vector that is not less than 0 and not greater than 1;
[0032] Before generating the first parameter, the method further includes: determining a corresponding node with the lowest energy consumption cost according to the calculated energy consumption cost of each task executed on a corresponding node of the task;
[0033] The target formula for updating a corresponding node of each task by the user is selected from a preset formula using the first parameter, and a new corresponding node of each task is determined from all nodes of the cluster based on the target formula, including:
[0034] When the absolute value of A is less than 1,
[0035]
[0036] D′ energy =|C×X(energy)-X(tsk)|;
[0037] When the absolute value of A is greater than 1, get any node X in the cluster rand (energy),
[0038]
[0039] D″ energy =|C×X rand (energy)-X(tsk)|;
[0040] Among them, X(tsk+1) represents a new corresponding node for each task, X(tsk) represents the previous corresponding node for each task, X(energy) represents a corresponding node with the lowest energy cost, r2 is a random vector not less than 0 and not greater than 1, b is the spiral shape parameter, l is a uniformly distributed random number not less than -1 and not greater than 1, C is a coefficient vector, and D' energy Denotes the distance between the previous corresponding node of each task and a corresponding node of the task with the lowest energy cost, D” energy Represents the distance between any node in the cluster and the previous corresponding node of each task.
[0041] In this method, the coefficient vector A can be used to control each task to search for a new corresponding node in the direction close to a corresponding node with the lowest energy cost, that is, to narrow the scope, or to search for a new corresponding node in a random position away from a corresponding node with the lowest energy cost, that is, to expand the scope; p can be used to select different formulas to calculate the new corresponding node, thereby expanding the possibility and scope of determining the new corresponding node and avoiding the new corresponding node being selected only in a local range; in addition, based on the energy cost as the heuristic function of the whale algorithm, tasks can be placed on nodes with lower energy costs first, so that the resource scheduling of the cluster can be more balanced, and energy consumption can be minimized, thereby reducing energy costs.
[0042] Optionally, the available CPU resources and memory resources of a new corresponding node for each task are greater than the CPU resources and memory resources required for executing each task on a new corresponding node for the task.
[0043] In this method, it is ensured that the available CPU resources and memory resources of a new corresponding node are greater than the CPU resources and memory resources required to execute the task on the new corresponding node, so as to avoid too many tasks being deployed on the new corresponding node, which may cause the node to crash, thereby improving the reliability of this application.
[0044] In a second aspect, an embodiment of the present application provides a task scheduling device, including:
[0045] The acquisition module is used to: randomly acquire N nodes in the cluster and randomly assign N tasks to the N nodes; each task in the N tasks corresponds to a node on the N nodes;
[0046] A processing module, used to: repeatedly perform the following steps until the number of repetitions reaches a threshold: calculate the energy consumption cost of each task executed on a corresponding node of the task according to a preset rule; generate a first parameter, use the first parameter to select a target formula for updating a corresponding node of each task from a preset formula, and determine a new corresponding node for each task from all nodes in the cluster based on the target formula; a new corresponding node for each task is different from a previous corresponding node for each task; calculate the energy consumption cost of each task executed on a new corresponding node for the task, if the energy consumption cost of each task executed on a new corresponding node for the task is less than the energy consumption cost of each task executed on a previous corresponding node for the task, save a new corresponding node for each task; otherwise, save the previous corresponding node for each task;
[0047] The processing module is also used to: output the latest corresponding node of each task when the number of repetitions reaches a threshold;
[0048] The allocation module is used to allocate each task to the latest corresponding node of the task for execution.
[0049] Optionally, energy consumption is used to indicate the CPU consumption and memory consumption of each task executed on a corresponding node of the task; when calculating the energy consumption cost of each task executed on a corresponding node of the task according to preset rules, the acquisition module is used to: obtain the CPU usage rate, memory usage rate and the time required to complete the execution of each task when it is executed on a corresponding node of the task; the processing module is used to: calculate the energy consumption per unit time of each task when it is executed on a corresponding node of the task according to a first rule based on the CPU usage rate and the memory usage rate; calculate the energy consumption cost of each task executed on a corresponding node of the task according to a second rule based on the energy consumption per unit time of each task when it is executed on a corresponding node of the task.
[0050] Optionally, the first rule is as follows:
[0051] E=C0+C1*U cpu +C2*U mem
[0052] Where E represents the energy consumption per unit time when each task is executed on a corresponding node of the task, C0, C1 and C2 are constants, and U cpu Indicates the CPU usage of each task when it is executed on a corresponding node of the task, U mem Indicates the memory usage of each task when it is executed on a corresponding node of the task.
[0053] Optionally, the second rule is as follows:
[0054] Cost energy =E×P×t
[0055] Among them, Cost energy It represents the energy consumption cost of each task executed on a corresponding node of the task, P represents the energy consumption billing standard per unit time, and t represents the time required for each task to complete the task execution on a corresponding node of the task.
[0056] Optionally, when obtaining the CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task, the acquisition module is specifically used to: query whether there is a historical record of each task being executed on a corresponding node of the task; if so, obtain the historical CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task; if not, predict the CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task based on the historical record of each task being executed on non-corresponding nodes and the information of each corresponding node.
[0057] Optionally, the first parameter includes a random number p and a coefficient vector A; the value of the random number p is not less than 0 and less than 1, and the coefficient vector A=2a×r1-a, wherein a is a vector that linearly decreases from 2 to 0, and r1 is a random vector that is not less than 0 and not greater than 1;
[0058] Before generating the first parameter, the processing module is further used to: determine a corresponding node with the lowest energy consumption cost according to the calculated energy consumption cost of each task executed on a corresponding node of the task;
[0059] The processing module selects a target formula for updating a corresponding node of each task from a preset formula by the user using the first parameter, and determines a new corresponding node of each task from all nodes of the cluster based on the target formula, specifically for:
[0060] When the absolute value of A is less than 1,
[0061]
[0062] D′ energy =|C×X(energy)-X(tsk)|;
[0063] When the absolute value of A is greater than 1, get any node X in the cluster rand (energy),
[0064]
[0065] D″ energy =|C×X rand (energy)-X(tsk)|;
[0066] Among them, X(tsk+1) represents a new corresponding node for each task, X(tsk) represents the previous corresponding node for each task, X(energy) represents a corresponding node with the lowest energy cost, r2 is a random vector not less than 0 and not greater than 1, b is the spiral shape parameter, l is a uniformly distributed random number not less than -1 and not greater than 1, C is a coefficient vector, and D' energy Denotes the distance between the previous corresponding node of each task and a corresponding node of the task with the lowest energy cost, D” energy Represents the distance between any node in the cluster and the previous corresponding node of each task.
[0067] Optionally, the available CPU resources and memory resources of a new corresponding node for each task are greater than the CPU resources and memory resources required for executing each task on a new corresponding node for the task.
[0068] In a third aspect, an embodiment of the present application provides an electronic device, comprising at least one processor, wherein the at least one processor is used to execute a computer program stored in a memory so that a method as in the first aspect or any optional implementation of the first aspect is implemented.
[0069] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the method in the first aspect or any optional implementation of the first aspect is implemented.
[0070] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program code, which, when executed on a computer, enables the method in the first aspect or any optional implementation of the first aspect to be implemented.
[0071] The technical effects or advantages of one or more technical solutions provided in the second, third, fourth and fifth aspects of the embodiments of the present application can be correspondingly explained by the technical effects or advantages of one or more corresponding technical solutions provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] Figure 1 A flowchart of a task scheduling method provided in an embodiment of the present application;
[0073] Figure 2 A structural diagram of a task scheduling device provided in an embodiment of the present application;
[0074] Figure 3 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0075] In the technical solution of this application, the collection, dissemination, and use of data are in compliance with the requirements of relevant national laws and regulations.
[0076] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.
[0077] The technical solution of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.
[0078] It should be understood that "multiple" in the description of the embodiment of the present application refers to two or more. "First", "second" etc. in the embodiment of the present application are used to distinguish different objects, rather than to describe a specific order. The term "and / or" in the embodiment of the present application is merely a kind of association relationship describing associated objects, indicating that three relationships may exist, for example, A and / or B, which may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "includes" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. The module in the embodiment of the present application refers to a part with independent functions in a software system.
[0079] Task scheduling refers to the reasonable arrangement of tasks on computing nodes according to specific rules of tasks, such as priority and execution time, in order to achieve efficient operation. It is specifically reflected in specific optimization goals, such as the goal of optimizing energy consumption costs. The physical characteristics of resources can be combined with some attributes of task execution, and tasks can be scheduled in a specific way. At present, cluster energy consumption costs and task scheduling are both research hotspots that have received much attention in recent years.
[0080] At present, the research on task scheduling is mainly focused on the task scheduling algorithms proposed for cloud computing components (such as Kubernetes, dockers) and big data components (such as Storm, Spark). The main optimization goals are to improve the execution time of the cluster, resource utilization, and reduce node communication. For example, the relevant modules of the cloud computing platform are optimized and transformed to achieve the purpose of reducing costs, or based on the greedy algorithm (that is, each step adopts the optimal solution) to minimize costs, etc., as well as Dolphin task scheduling based on energy consumption costs.
[0081] The current Dolphin scheduler provides a load balancing algorithm by Worker, which specifically refers to balancing the load (i.e., work tasks) and distributing them to multiple operation units (i.e., nodes) for execution, so as to achieve the purpose of reasonable pressure distribution. The main process of the Dolphin scheduler to implement task scheduling includes: after the local node resource Master-server obtains the task, it first obtains the list of all machine information of the work group to which the current task belongs from the distributed application coordination software Zookeeper, and then places the task on a node according to the scheduling strategy. In the Dolphin cluster, there are three default scheduling strategy algorithms for allocating the same task group to different nodes. The three scheduling strategy algorithms are random scheduling, polling algorithm, and node resource weight algorithm. From the perspective of resource cost, these three scheduling strategy algorithms all have some drawbacks and shortcomings. Specifically: 1) The principle of the random scheduling algorithm is relatively simple. After the Zookeeper cluster obtains all the nodes on the task work group, it will first determine the total number of nodes. If the total number of nodes is 1, the work group is directly assigned to the node. If the total number of nodes is not 1, a node is randomly selected as the execution node of the task. Random scheduling algorithms will lead to uneven task placement and may place more tasks on nodes with poor performance, which will not only increase the execution time of tasks, but also cause serious imbalance of resources among cluster nodes. 2) Polling algorithm, which is implemented with the help of the atomic class AtomicInteger, counting through AtomicInteger, and then obtaining the location of the data of each node by taking the modulus of the number of nodes. Compared with the random algorithm, the polling algorithm will make the task distribution more even and make better use of each node in the cluster. However, the polling algorithm does not take energy consumption into consideration and will place more tasks on nodes with higher costs, resulting in higher overall energy consumption of the cluster. 3) Node resource weight algorithm, its implementation method mainly focuses on resources. The Master-server obtains the node resource information through the heartbeat detection of the node through Zookeeper (that is, the Master-server and each node periodically send information to determine the health status of each node), and then selects the node with the smallest weight according to the weight in the resource information of each node. The node resource weight algorithm is designed based on the perspective of resources and load. Therefore, tasks may not be placed on nodes with lower energy consumption first, which may also lead to higher energy consumption of the overall cluster.
[0082] It can be seen that the scheduling strategy algorithm based on Dolphin itself cannot ensure that tasks are evenly distributed under low energy consumption, so that the node resources of the cluster can be maximized.
[0083] In order to solve the above problems, the embodiment of the present application adopts the Whale Optimization Algorithm (WOA) as the scheduling strategy algorithm. The whale algorithm is a meta-heuristic optimization algorithm that simulates the hunting behavior of humpback whales. Humpback whales have a special hunting method, which is called bubble net foraging; the standard WOA simulates the humpback whale's unique search method and hunting mechanism, mainly including: searching for prey, hunting prey, and bubble net hunting. Three important stages. The position of each humpback whale in WOA represents a potential solution. By continuously updating the position of the whale in the solution space, the global optimal solution is finally obtained.
[0084] The embodiment of the present application is improved on the basis of the traditional whale algorithm, and the energy consumption cost of each task executed on a corresponding node of the task is calculated, and the energy consumption cost is used as the heuristic function in the whale algorithm. The target formula is determined from the preset formula using the first parameter, and a new corresponding node for each task is determined from all nodes of the cluster based on the target formula. The energy consumption cost of each task executed on a new corresponding node of the task is compared with the energy consumption cost on the previous corresponding node of the task, and the corresponding node with lower energy consumption cost is saved. Repeat the above method until the number of repetitions reaches the threshold, and output the latest corresponding node of each task when the number of repetitions reaches the threshold, and assign each task to the latest corresponding node of the task for execution. Use the energy consumption cost as the judgment condition for selecting the corresponding node, so that more tasks can be placed on the node with lower energy consumption cost first, so that the resource scheduling of the cluster can be more balanced, thereby reducing the energy consumption cost of the cluster.
[0085] The task scheduling method provided in the embodiment of the present application can be applied to any task scheduling scenario, including but not limited to the task scheduling in the above-mentioned Dophin cluster.
[0086] See also Figure 1 , a task scheduling method provided in an embodiment of the present application, can be applied to any electronic device with processing capabilities, such as: a computer, a virtual machine, a tablet, etc. The method includes steps S101 to S106:
[0087] S101. Randomly obtain N nodes in the cluster, and randomly assign N tasks to the N nodes.
[0088] Each of the N tasks corresponds to a node on the N nodes, where N is a positive integer.
[0089] Exemplarily, a corresponding node of each current task is recorded.
[0090] It can be understood that, initially, after N tasks are randomly assigned to N nodes of the cluster, it is necessary to continuously update the corresponding node of each task by repeatedly executing the node updating method (ie, the following steps S103 to S105) to find the optimal node for executing each task.
[0091] S102: Determine whether the number of repetitions of executing the method for updating the node is less than a threshold.
[0092] Exemplarily, before the method for updating nodes (ie, the following steps S103 to S105 ) is executed for the first time, the initial number of repetitions is 0, and the number of repetitions increases by 1 each time steps S103 to S105 are executed.
[0093] If the number of repetitions is less than the threshold, steps S103 to S105 are executed; if the number of repetitions is not less than the threshold, step S106 is executed.
[0094] It is understandable that the threshold of the number of repetitions can be determined based on the balance accuracy and computing efficiency required by the cluster resources, as well as the total resources of the cluster, the number of tasks, and other actual conditions. For example, if the number of tasks is small (e.g., a single task), a smaller threshold can be set; if the number of tasks is large (i.e., a more complex task scheduling scenario), a larger threshold can be set to ensure that the optimal task scheduling solution can be found.
[0095] S103: Calculate the energy consumption cost of each task executed on a corresponding node of the task according to a preset rule.
[0096] In the embodiment of the present application, energy consumption is used to indicate the CPU consumption and memory consumption of each task executed on a corresponding node of the task. However, in actual situations, the energy consumption of a node is not limited to CPU consumption and memory consumption, but also includes the consumption of input and output devices, the consumption of boards, etc., and the proportion of various types of consumption of different nodes is also different, but the main energy consumption still comes from CPU consumption and memory consumption. This application takes energy consumption including CPU consumption and memory consumption as an example, and energy consumption can actually be calculated according to specific circumstances.
[0097] In a possible example, the method for calculating the energy consumption cost of each task executed on a corresponding node of the task may include steps S201 to S203:
[0098] S201, obtaining the CPU usage rate, memory usage rate and time required to complete the task execution when each task is executed on a corresponding node of the task;
[0099] S202, based on the CPU usage rate and the memory usage rate, calculating the energy consumption per unit time when each task is executed on a corresponding node of the task according to the first rule;
[0100] In a possible embodiment, the first rule is as follows:
[0101] E=C0+C1*U cpu +C2*U mem
[0102] Where E represents the energy consumption per unit time when each task is executed on a corresponding node of the task, C0, C1 and C2 are constants, and U cpu Indicates the CPU usage of each task when it is executed on a corresponding node of the task, U mem Indicates the memory usage of each task when it is executed on a corresponding node of the task.
[0103] C0, C1 and C2 can be determined according to actual conditions.
[0104] S203 , calculating the energy consumption cost of each task executed on a corresponding node of the task according to a second rule based on the energy consumption per unit time when each task is executed on a corresponding node of the task and the time required to complete the task execution.
[0105] The second rule is as follows:
[0106] Cost energy =E×P×t
[0107] Among them, Cost energy represents the energy consumption cost of each task executed on a corresponding node of the task, P represents the energy consumption billing standard per unit time (such as Amazon EC2 billing standard), and t represents the time required for each task to complete the task execution on a corresponding node of the task.
[0108] In a possible embodiment, obtaining the CPU usage rate, memory usage rate, and time required to complete the task execution when each task is executed on a corresponding node of the task includes:
[0109] Check whether there is a historical record of each task being executed on a corresponding node of the task;
[0110] If it exists, get the historical CPU usage, memory usage, and time required to complete the execution of each task when it is executed on a corresponding node of the task;
[0111] If it does not exist, the CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task are predicted based on the historical records of the execution of each task on non-corresponding nodes and the information of each corresponding node.
[0112] It can be understood that the execution status of each task on the historical node will be recorded, or the historical task execution status on each node will be recorded; based on the historical records of each task execution, the historical CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task can be directly obtained or predicted. There is no need to put each task on its corresponding node for execution. Obtaining specific data during execution can improve the efficiency of data acquisition, and thus improve the efficiency of subsequent calculation of energy consumption costs.
[0113] It can be understood that the above preset rule for calculating the energy consumption cost is only a possible example, and other rules can actually be used to calculate the energy consumption cost.
[0114] After calculating the energy consumption cost of each task executed on a corresponding node of the task, a corresponding node with the lowest energy consumption cost is determined from all corresponding nodes and saved.
[0115] S104, generating a first parameter, selecting a target formula for updating a corresponding node of each task from a preset formula using the first parameter, and determining a new corresponding node for each task from all nodes in the cluster based on the target formula.
[0116] It can be understood that when a whale (equivalent to the task in the embodiment of the present application) hunts, the search range is the global solution space, and the position of the prey (equivalent to the node in the embodiment of the present application) needs to be determined first in order to surround it. The WOA algorithm assumes that the current best candidate solution is the target prey or prey close to the target, and other whales try to update the position of the prey to the position of the best candidate solution.
[0117] Corresponding to the embodiment of the present application, the nodes where the task can be placed are all idle nodes in the cluster and nodes whose available resources are greater than the resources required for task execution. The best candidate solution is the corresponding node with the lowest energy consumption cost calculated in S103. When other tasks determine the new corresponding node, they can move closer to the corresponding node with the lowest energy consumption cost.
[0118] Since there are two hunting methods for whales to approach their prey, one is contraction and the other is spiral update, and the probability of the two hunting methods is the same, a random variable p is introduced, and p is used to determine which of the two hunting methods is selected.
[0119] And because there are two ways for whales to hunt, one is bubble net hunting, that is, approaching the target prey, and the other is random hunting, that is, moving away from the target prey and exploring new random prey, therefore, the vector coefficient A is introduced, and A is used to determine whether the current task is to move away from the corresponding node with the lowest energy consumption cost or to move close to the corresponding node with the lowest energy consumption cost.
[0120] Exemplarily, the first parameter includes a random number p and a coefficient vector A; the value of the random number p is not less than 0 and less than 1, and the coefficient vector A=2a×r1-a, wherein a is a vector that linearly decreases from 2 to 0, and r1 is a random vector that is not less than 0 and not greater than 1;
[0121] The target formula for updating a corresponding node of each task by the user is selected from a preset formula using the first parameter, and a new corresponding node of each task is determined from all nodes of the cluster based on the target formula, specifically including:
[0122] When the absolute value of A is less than 1,
[0123]
[0124] D′ energy =|C×X(energy)-X(tsk)|;
[0125] When the absolute value of A is greater than 1, get any node X in the cluster rand (energy),
[0126]
[0127] D″ energy =|C×X rand (energy)-X(tsk)|;
[0128] Among them, X(tsk+1) represents a new corresponding node for each task, X(tsk) represents the previous corresponding node for each task, X(energy) represents a corresponding node with the lowest energy cost, r2 is a random vector not less than 0 and not greater than 1, b is the spiral shape parameter, l is a uniformly distributed random number not less than -1 and not greater than 1, C is a coefficient vector, and D' energy Denotes the distance between the previous corresponding node of each task and a corresponding node of the task with the lowest energy cost, D” energy Represents the distance between any node in the cluster and the previous corresponding node of each task.
[0129] According to the values of the random number p and the coefficient vector A, the target formula can be selected from the four calculation formulas. By substituting the target formula into the parameter calculation, a new corresponding node for each task can be obtained, avoiding the new corresponding nodes that are only selected in a local range. In addition, based on the energy consumption cost as the heuristic function of the whale algorithm, tasks can be placed on nodes with lower energy consumption first, so that the resource scheduling of the cluster can be more balanced, and energy consumption can be minimized, thereby reducing energy consumption costs.
[0130] It should be noted that the available CPU resources and memory resources of a new corresponding node of each task are greater than the CPU resources and memory resources required for each task to be executed on a new corresponding node of the task.
[0131] In this way, it is avoided that too many tasks are deployed on the new corresponding node, resulting in insufficient resources of the corresponding node and causing the corresponding node to crash.
[0132] S105. Calculate the energy consumption cost of executing each task on a new corresponding node of the task. If the energy consumption cost of executing each task on a new corresponding node of the task is less than the energy consumption cost of executing each task on the previous corresponding node of the task, save a new corresponding node of each task; otherwise, save the previous corresponding node of each task.
[0133] It can be understood that by comparing the energy consumption cost of each task on the new corresponding node with the energy consumption cost of the task on the previous corresponding node, the corresponding node with lower energy consumption cost is saved, and the task is preferentially placed on the node with lower energy consumption cost to minimize the energy consumption cost.
[0134] S106 , output the latest corresponding node of each task when the number of repetitions reaches a threshold, and assign each task to the latest corresponding node of the task for execution.
[0135] In this method, the energy consumption cost of each task executed on a corresponding node of the task is used as a criterion for evaluating the task scheduling scheme, which can maximize the utilization of cluster resources; and the whale algorithm is improved based on the energy consumption cost, so that more tasks can be placed preferentially on nodes with lower energy consumption costs, making the cluster resource scheduling more balanced, thereby reducing the energy consumption cost of the cluster.
[0136] In a possible design, before executing the above task scheduling method provided in the embodiment of the present application, it is also necessary to ensure that the memory requirements and CPU requirements of the N tasks are not greater than the total memory resources and CPU resources of the cluster. Specifically, the method for calculating the CPU linear requirements and memory linear requirements of the N tasks can be as follows:
[0137]
[0138] in, Represents the task flow t i to j The set of task flows, β cpu and β mem They represent the CPU requirements and memory requirements of a single task respectively.
[0139] In this way, cluster crash caused by the memory and CPU requirements of N tasks exceeding the total memory and CPU resources of the cluster can be avoided.
[0140] The method provided by the embodiment of the present application is introduced above, and the device provided by the embodiment of the present application is introduced below.
[0141] Based on the same technical concept, the embodiment of the present application provides a task scheduling device, which includes a module / unit / means for executing the method executed by the electronic device in the above method embodiment. The module / unit / means can be implemented by software, or by hardware, or the corresponding software can be implemented by hardware.
[0142] For example, see Figure 2 , the device 200 comprises:
[0143] The acquisition module 201 is used to: randomly acquire N nodes in the cluster, and randomly assign N tasks to the N nodes; each task in the N tasks corresponds to a node on the N nodes;
[0144] The processing module 202 is used to: repeatedly perform the following steps until the number of repetitions reaches a threshold value: calculate the energy consumption cost of each task executed on a corresponding node of the task according to a preset rule; generate a first parameter, use the first parameter to select a target formula for updating a corresponding node of each task from a preset formula, and determine a new corresponding node for each task from all nodes in the cluster based on the target formula; a new corresponding node for each task is different from a previous corresponding node for each task; calculate the energy consumption cost of each task executed on a new corresponding node for the task, if the energy consumption cost of each task executed on a new corresponding node for the task is less than the energy consumption cost of each task executed on a previous corresponding node for the task, save a new corresponding node for each task; otherwise, save the previous corresponding node for each task;
[0145] The processing module 202 is also used to: output the latest corresponding node of each task when the number of repetitions reaches a threshold;
[0146] The allocation module 203 is used to allocate each task to the latest corresponding node of the task for execution.
[0147] Optionally, energy consumption is used to indicate the CPU consumption and memory consumption of each task executed on a corresponding node of the task; when calculating the energy consumption cost of each task executed on a corresponding node of the task according to preset rules, the acquisition module 201 is used to: obtain the CPU usage rate, memory usage rate and the time required to complete the execution of each task when it is executed on a corresponding node of the task; the processing module 202 is used to: calculate the energy consumption per unit time of each task executed on a corresponding node of the task according to a first rule based on the CPU usage rate and the memory usage rate; calculate the energy consumption cost of each task executed on a corresponding node of the task according to a second rule based on the energy consumption per unit time of each task executed on a corresponding node of the task and the time required to complete the execution of the task.
[0148] Optionally, the first rule is as follows:
[0149] E=C0+C1*U cpu +C2*U mem
[0150] Where E represents the energy consumption per unit time when each task is executed on a corresponding node of the task, C0, C1 and C2 are constants, and U cpu Indicates the CPU usage of each task when it is executed on a corresponding node of the task, U mem Indicates the memory usage of each task when it is executed on a corresponding node of the task.
[0151] Optionally, the second rule is as follows:
[0152] Cost energy =E×P×t
[0153] Among them, Cost energy It represents the energy consumption cost of each task executed on a corresponding node of the task, P represents the energy consumption billing standard per unit time, and t represents the time required for each task to complete the task execution on a corresponding node of the task.
[0154] Optionally, when acquiring the CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task, the acquisition module 201 is specifically used to: query whether there is a historical record of each task being executed on a corresponding node of the task; if so, acquire the historical CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task; if not, predict the CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task based on the historical record of each task being executed on non-corresponding nodes and the information of each corresponding node.
[0155] Optionally, the first parameter includes a random number p and a coefficient vector A; the value of the random number p is not less than 0 and less than 1, and the coefficient vector A=2a×r1-a, wherein a is a vector that linearly decreases from 2 to 0, and r1 is a random vector that is not less than 0 and not greater than 1;
[0156] Before generating the first parameter, the processing module 202 is further used to: determine a corresponding node with the lowest energy consumption cost according to the calculated energy consumption cost of each task executed on a corresponding node of the task;
[0157] The processing module 202 selects a target formula for updating a corresponding node of each task from a preset formula by a user using the first parameter, and determines a new corresponding node of each task from all nodes of the cluster based on the target formula, specifically for:
[0158] When the absolute value of A is less than 1,
[0159]
[0160] D′ energy =|C×X(energy)-X(tsk)|;
[0161] When the absolute value of A is greater than 1, get any node X in the cluster rand (energy),
[0162]
[0163] D″ energy =|C×X rand (energy)-X(tsk)|;
[0164] Among them, X(tsk+1) represents a new corresponding node for each task, X(tsk) represents the previous corresponding node for each task, X(energy) represents a corresponding node with the lowest energy cost, r2 is a random vector not less than 0 and not greater than 1, b is the spiral shape parameter, l is a uniformly distributed random number not less than -1 and not greater than 1, C is a coefficient vector, and D' energy Denotes the distance between the previous corresponding node of each task and a corresponding node of the task with the lowest energy cost, D” energy Represents the distance between any node in the cluster and the previous corresponding node of each task.
[0165] Optionally, the available CPU resources and memory resources of a new corresponding node for each task are greater than the CPU resources and memory resources required for executing each task on a new corresponding node for the task.
[0166] It should be understood that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.
[0167] Based on the same technical concept, see Figure 3 The embodiment of the present application further provides an electronic device 300, including:
[0168] At least one processor 301; and a communication interface 303 that is communicatively connected to the at least one processor 301; the at least one processor 301 executes instructions stored in the memory 302, so that the electronic device 300 executes the method steps performed by the billboard in the above method embodiment through the communication interface 303.
[0169] Optionally, the memory 302 is located outside the electronic device 300 .
[0170] Optionally, the electronic device 300 includes the memory 302, the memory 302 is connected to the at least one processor 301, and the memory 302 has instructions that can be executed by the at least one processor 301. Figure 3 The dashed lines indicate that the memory 302 is optional for the electronic device 300 .
[0171] The at least one processor 301 and the memory 302 may be coupled via an interface circuit or may be integrated together, which is not limited here.
[0172] The specific connection medium between the at least one processor 301, the memory 302 and the communication interface 303 is not limited in the embodiment of the present application. Figure 3 In the embodiment, at least one processor 301, a memory 302 and a communication interface 303 are connected via a bus 304. Figure 3 The connection between other components is only for schematic illustration and is not intended to be limiting. The bus part may be an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 It is represented by only one thick line, but it does not mean that there is only one bus or one type of bus.
[0173] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor implemented by reading software code stored in a memory.
[0174] Exemplarily, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0175] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which acts as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DirectRambus RAM, DR RAM).
[0176] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.
[0177] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0178] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the computer executes the method steps executed by any device in the above method embodiments.
[0179] Based on the same technical concept, an embodiment of the present application also provides a computer program product, including computer program code. When the computer program code is executed on a computer, the method steps executed by any device in the above method embodiment are implemented.
[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0181] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0182] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0184] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A task scheduling method, characterized in that: include: Randomly obtain N nodes in the cluster, and randomly assign N tasks to the N nodes; each of the N tasks corresponds to one of the N nodes; Repeat the following steps until the number of repetitions reaches the threshold: Calculate the energy consumption cost of each task executed on a corresponding node of the task according to a preset rule; Generate a first parameter, select a target formula for updating a corresponding node of each task from a preset formula using the first parameter, and determine a new corresponding node of each task from all nodes of the cluster based on the target formula; A new corresponding node for each task is different from a previous corresponding node for each task; Calculating the energy consumption cost of executing each task on a new corresponding node of the task, and if the energy consumption cost of executing each task on the new corresponding node of the task is less than the energy consumption cost of executing each task on the previous corresponding node of the task, saving the new corresponding node of each task; Otherwise, the previous corresponding node of each task is saved; Output the latest corresponding node of each task when the number of repetitions reaches a threshold, and assign each task to the latest corresponding node of the task for execution.
2. The method according to claim 1, characterized in that The energy consumption is used to indicate the CPU consumption and memory consumption of each task executed on a corresponding node of the task; and the energy consumption cost of each task executed on a corresponding node of the task is calculated according to a preset rule, including: Obtain the CPU usage rate, memory usage rate and time required to complete the task execution when each task is executed on a corresponding node of the task; Based on the CPU usage rate and the memory usage rate, calculating the energy consumption per unit time when each task is executed on a corresponding node of the task according to a first rule; Based on the energy consumption per unit time when each task is executed on a corresponding node of the task and the time required to complete the task execution, the energy consumption cost of each task executed on a corresponding node of the task is calculated according to the second rule.
3. The method according to claim 2, characterized in that The first rule is as follows: E=C0+C1*U cpu +C2*U mem Where E represents the energy consumption per unit time when each task is executed on a corresponding node of the task, C0, C1 and C2 are constants, and U cpu It represents the CPU usage of each task when it is executed on a corresponding node of the task, U mem Indicates the memory usage of each task when it is executed on a corresponding node of the task.
4. The method according to claim 2 or 3, characterized in that The second rule is as follows: Cost energy =E×P×t Among them, Cost energy represents the energy consumption cost of executing each task on a corresponding node of the task, P represents the energy consumption billing standard per unit time, and t represents the time required for each task to complete the task execution on a corresponding node of the task.
5. The method according to claim 2, characterized in that The obtaining of the CPU usage rate, the memory usage rate and the time required to complete the execution of each task when the task is executed on a corresponding node of the task includes: Query whether there is a historical record of each task being executed on a corresponding node of the task; If so, obtain the historical CPU usage, memory usage, and time required to complete the execution of each task when it is executed on a corresponding node of the task; If not present, then based on the historical records of each task being executed on non-corresponding nodes and the information of each corresponding node, predict the CPU usage, memory usage and time required to complete the execution of each task when it is executed on a corresponding node of the task.
6. The method according to claim 1, characterized in that The first parameter includes a random number p and a coefficient vector A; the value of the random number p is not less than 0 and less than 1, and the coefficient vector A=2a×r1-a, wherein a is a vector that linearly decreases from 2 to 0, and r1 is a random vector that is not less than 0 and not greater than 1; Before generating the first parameter, the method further includes: determining a corresponding node with the lowest energy consumption cost according to the calculated energy consumption cost of each task executed on a corresponding node of the task; The step of selecting a target formula for updating a corresponding node of each task from a preset formula by using the first parameter, and determining a new corresponding node of each task from all nodes of the cluster based on the target formula includes: When the absolute value of A is less than 1, D′ energy =|C×X(energy)-X(tsk)|; When the absolute value of A is greater than 1, obtain any node X in the cluster rand (energy), D″ energy =|C×X rand (energy)-X(tsk)|; Wherein, X(tsk+1) represents a new corresponding node of each task, X(tsk) represents the previous corresponding node of each task, X(energy) represents a corresponding node with the lowest energy consumption cost, r2 is a random vector not less than 0 and not greater than 1, b is a spiral shape parameter, l is a uniformly distributed random number not less than -1 and not greater than 1, C is a coefficient vector, and D' energy Represents the distance between the previous corresponding node of each task and the corresponding node with the lowest energy consumption cost, D” energy Represents the distance between any node in the cluster and the previous corresponding node of each task.
7. The method according to claim 6, characterized in that The available CPU resources and memory resources of a new corresponding node of each task are greater than the CPU resources and memory resources required for executing each task on a new corresponding node of the task.
8. A task scheduling device, characterized in that: include: An acquisition module is used to: randomly acquire N nodes in the cluster, and randomly assign N tasks to the N nodes; each of the N tasks corresponds to one of the N nodes; The processing module is used to repeatedly perform the following steps until the number of repetitions reaches a threshold: Calculate the energy consumption cost of each task executed on a corresponding node of the task according to a preset rule; Generate a first parameter, select a target formula for updating a corresponding node of each task from a preset formula using the first parameter, and determine a new corresponding node of each task from all nodes of the cluster based on the target formula; A new corresponding node for each task is different from a previous corresponding node for each task; Calculate the energy consumption cost of executing each task on a new corresponding node of the task, and if the energy consumption cost of executing each task on the new corresponding node of the task is less than the energy consumption cost of executing each task on the previous corresponding node of the task, save the new corresponding node of each task; otherwise, save the previous corresponding node of each task; The processing module is further used to: output the latest corresponding node of each task when the number of repetitions reaches a threshold; The allocation module is used to allocate each task to the latest corresponding node of the task for execution.
9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the method according to any one of claims 1 to 7 according to the obtained program instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to have a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the method according to any one of claims 1 to 7 is implemented.
11. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 7.