Computing power scheduling method in distributed computing environment

Abstract

The invention belongs to the technical field of computing power scheduling, and particularly relates to a computing power scheduling method in a distributed computing environment. According to the task scheduling method and system, the task needing to be processed at present is obtained through the task management system, the task is divided into the real-time task set, the priority task set and the common task set, and the related tasks are scheduled by adopting the corresponding scheduling strategy based on the set corresponding to the current task and the priority in the set. Resource scheduling and allocation in a distributed computing environment are realized; according to the method, through the steps of task classification, real-time resource monitoring, resource demand estimation, priority ranking, reasonable scheduling strategy, load calculation, resource migration and the like, the technical problem of computing resource scheduling and allocation in a distributed environment is effectively solved, timely processing of high priority is ensured, the system load is balanced, and the system performance is improved. And efficient utilization of resources and stable operation of the system are realized, so that the performance and reliability of the whole distributed computing environment are improved.

Description

Technical Field

[0001] The present application belongs to the technical field of computing power scheduling, and more specifically, to a computing power scheduling method in a distributed computing environment. Background Art

[0002] With the rapid growth of global electricity demand and the continuous expansion of power grids, the management and dispatch of power systems have become increasingly complex. The changes in the scale and structure of modern power grid systems, as well as the increase in the proportion of new energy access, have brought new challenges to power grid dispatch.

[0003] With the rapid development of technologies such as cloud computing, big data, and artificial intelligence, distributed computing has become the mainstream method for processing massive data and complex computing tasks. Distributed computing can significantly improve computing efficiency and resource utilization by distributing computing tasks to multiple computing nodes for parallel processing.

[0004] However, due to the particularity of the power grid, in a distributed computing environment, with the continuous increase in computing tasks, how to effectively schedule and allocate computing resources has become an important technical issue. Summary of the invention

[0005] The present invention provides a computing power scheduling method in a distributed computing environment, which is intended to solve the technical problem of how to effectively schedule and allocate computing resources.

[0006] The computing power scheduling method in a distributed computing environment includes the following steps:

[0007] Step 1: The task management system periodically obtains the tasks that need to be processed from the task queue and classifies the tasks to be processed into real-time tasks, priority tasks and ordinary tasks. The priority of real-time tasks is higher than that of priority tasks, and the priority of priority tasks is higher than that of ordinary tasks.

[0008] Step 2: Collect resource usage of all computing nodes in real time based on the resource monitoring system;

[0009] Step 3: Estimate resource requirements for new tasks using the trained linear regression model. For existing tasks, directly obtain resource requirements from the task description.

[0010] Step 4: Sort the real-time tasks according to their priorities, adopt the least recently loaded node strategy, traverse all nodes to find the node with the least load and assign tasks;

[0011] Step 5: Sort the priority tasks from high to low, adopt a round-robin scheduling strategy, and poll and allocate tasks based on load balancing;

[0012] Step 6: Sort common tasks by priority from high to low, use the least task node strategy, traverse all nodes to find the node with the least number of tasks and assign tasks;

[0013] Step 7: Regularly calculate the current load of each node. If the load of a node exceeds the threshold, perform resource migration and migrate the task to the node with the lowest load.

[0014] The present invention obtains the tasks that need to be processed currently through the task management system, and divides the tasks into real-time tasks, priority tasks and ordinary task sets, and schedules the related tasks based on the set corresponding to the current task and the priority in the set using the corresponding scheduling strategy, thereby realizing resource scheduling and allocation in a distributed computing environment; the present invention effectively solves the technical problems of computing resource scheduling and allocation in a distributed environment through the steps of task classification, real-time resource monitoring, resource demand estimation, priority sorting, reasonable scheduling strategy, load calculation and resource migration, ensures timely processing of high priority, system load balancing, efficient use of resources and stable operation of the system, thereby improving the performance and reliability of the entire distributed computing environment.

[0015] Preferably, the step of classifying the tasks to be processed into a set of real-time tasks, priority tasks and common tasks in step 1 is as follows:

[0016] Task acquisition and initialization: The task management system obtains all tasks that need to be processed from the task queue at every fixed time interval;

[0017] Task classification: classify each task and classify it into corresponding task sets according to its importance and urgency;

[0018] Task priority determination: Within each task set, sort the tasks within each task set according to their importance and urgency.

[0019] Preferably, the resource usage in step 2 includes:

[0020] CPU Usage:

[0021]

[0022] Where: CPU usage Indicates the CPU usage of the computing node; CPU time,used,i Indicates the usage time of the i-th core; CPU total,time,i represents the total time of the i-th core; n represents the number of cores;

[0023] Memory usage:

[0024]

[0025] Where: Memory usage Indicates the memory usage of the computing node; Memory total Indicates the total memory of the computing node; Memory used Indicates the used memory of the computing node;

[0026] Storage usage:

[0027]

[0028] Where: Storage usage Indicates the storage usage of the computing node; Storage used Indicates the amount of storage used by the compute node; Storage total Indicates the total storage capacity of the computing node;

[0029] Network bandwidth:

[0030]

[0031] Where: Bandwidth usage Indicates the bandwidth usage of the computing node; Bandwidth used Indicates the bandwidth used by the computing node; Bandwidth total Indicates the total bandwidth of the computing node.

[0032] Preferably, the linear regression model in step 3 is as follows:

[0033] M usage =β0+β1×Task Type +β2×Input Size +β3×Task Complexity +∈;

[0034] Where: M usage represents the predicted resource requirements, including CPU requirements, memory requirements, storage requirements, and network bandwidth requirements; β0, β1, β2, and β3 represent regression coefficients; Task Type Indicates the task type; Input Size Indicates the amount of input data for the task; Task Complexity represents the task complexity; ∈ represents the error term.

[0035] Preferably, the task complexity is calculated as follows:

[0036] Calculate the logical complexity of the task logic logic :

[0037]

[0038] Where: Steps i Indicates the complexity of the i-th step; Weight i represents the weight of the i-th step;

[0039] Data structure complexity:

[0040]

[0041] Where: m represents the total number of data structure operations; Operations j Indicates the number of j-th operation; Cost j represents the complexity cost of the jth operation;

[0042] The task complexity is obtained based on the logic complexity and data structure complexity:

[0043] Complexity total =μ1×Complexity logic +μ2×Complexity data ;

[0044] Where: μ1 and μ2 both represent weights.

[0045] Preferably, step 4 comprises the following steps:

[0046] Step 4.1: Take the weighted sum of the node's CPU, memory, storage, and network bandwidth usage to get the current node load;

[0047] Step 4.2: Sort all computing nodes in ascending order according to load and find the node with the smallest load;

[0048] Step 4.3: Sort each real-time task according to its priority, assign tasks with higher priority first, traverse the sorted node list, and assign the real-time task to the node with the smallest load.

[0049] Preferably, the steps of allocating priority tasks to computing nodes using the cyclic scheduling strategy are as follows:

[0050] Take a weighted sum of the node's CPU, memory, storage, and network bandwidth usage to get the current node's load;

[0051] Sort the tasks in the priority task set from high to low priority;

[0052] Sort all calculation records according to their load from small to large;

[0053] Task allocation: Take the highest priority tasks in turn, calculate whether the available resources of the current node meet the requirements of the task, and allocate them if they do. If not, calculate whether the next node meets the requirements until the current task is allocated.

[0054] The task allocation step is repeated until all tasks are allocated to corresponding nodes.

[0055] The beneficial effects of the present invention include:

[0056] The present invention obtains the tasks that need to be processed currently through the task management system, and divides the tasks into real-time tasks, priority tasks and ordinary task sets, and schedules the related tasks based on the set corresponding to the current task and the priority in the set using the corresponding scheduling strategy, thereby realizing resource scheduling and allocation in a distributed computing environment; the present invention effectively solves the technical problems of computing resource scheduling and allocation in a distributed environment through the steps of task classification, real-time resource monitoring, resource demand estimation, priority sorting, reasonable scheduling strategy, load calculation and resource migration, ensures timely processing of high priority, system load balancing, efficient use of resources and stable operation of the system, thereby improving the performance and reliability of the entire distributed computing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 The overall system architecture diagram provided for the embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] See also Figure 1 As shown, the best embodiment of the present invention is further described;

[0061] The computing power scheduling method in a distributed computing environment includes the following steps:

[0062] Step 1: The task management system periodically obtains the tasks that need to be processed from the task queue and classifies the tasks to be processed into real-time tasks, priority tasks and ordinary tasks. The priority of real-time tasks is higher than that of priority tasks, and the priority of priority tasks is higher than that of ordinary tasks.

[0063] The steps of classifying the tasks to be processed into real-time tasks, priority tasks and common task sets in step 1 are as follows:

[0064] Task acquisition and initialization: The task management system obtains all tasks that need to be processed from the task queue at every fixed time interval;

[0065] Task classification: classify each task and classify it into corresponding task sets according to its importance and urgency;

[0066] Task priority determination: Within each task set, sort the tasks within each task set according to their importance and urgency.

[0067] Step 2: Collect resource usage of all computing nodes in real time based on the resource monitoring system;

[0068] The resource usage in step 2 includes:

[0069] CPU Usage:

[0070]

[0071] Where: CPU usage Indicates the CPU usage of the computing node; CPU time,used,i Indicates the usage time of the i-th core; CPU total,time,i represents the total time of the i-th core; n represents the number of cores;

[0072] Memory usage:

[0073]

[0074] Where: Memory usage Indicates the memory usage of the computing node; Memory total Indicates the total memory of the computing node; Memory used Indicates the used memory of the computing node;

[0075] Storage usage:

[0076]

[0077] Where: Storage usageIndicates the storage usage of the computing node; Storage used Indicates the amount of storage used by the compute node; Storage total Indicates the total storage capacity of the computing node;

[0078] Network bandwidth:

[0079]

[0080] Where: Bandwidth usage Indicates the bandwidth usage of the computing node; Bandwidth used Indicates the bandwidth used by the computing node; Bandwidth total Indicates the total bandwidth of the computing node.

[0081] Step 3: Estimate resource requirements for new tasks using the trained linear regression model. For existing tasks, directly obtain resource requirements from the task description.

[0082] The linear regression model in step 3 is as follows:

[0083] M usage =β0+β1×Task Type +β2×Input Size +β3×Task Complexity +∈;

[0084] Where: M usage represents the predicted resource requirements, including CPU requirements, memory requirements, storage requirements, and network bandwidth requirements; β0, β1, β2, and β3 represent regression coefficients; Task Type Indicates the task type; Input Size Indicates the amount of input data for the task; Task Complexity represents the task complexity; ∈ represents the error term.

[0085] Preferably, the task complexity is calculated as follows:

[0086] Calculate the logical complexity of the task logic logic :

[0087]

[0088] Where: Steps i Indicates the complexity of the i-th step; Weight i represents the weight of the i-th step;

[0089] Data structure complexity:

[0090]

[0091] Where: m represents the total number of data structure operations; Operations j Indicates the number of j-th operation; Cost j represents the complexity cost of the jth operation;

[0092] The task complexity is obtained based on the logic complexity and data structure complexity:

[0093] Complexity total =μ1×Complexity logic +μ2×Complexity data ;

[0094] Where: μ1 and μ2 both represent weights.

[0095] Step 4: Sort the real-time tasks according to their priorities, adopt the least recently loaded node strategy, traverse all nodes to find the node with the least load and assign tasks;

[0096] The step 4 comprises the following steps:

[0097] Step 4.1: Take the weighted sum of the node's CPU, memory, storage, and network bandwidth usage to get the current node load;

[0098] Step 4.2: Sort all computing nodes in ascending order according to load and find the node with the smallest load;

[0099] Step 4.3: Sort each real-time task according to its priority, assign tasks with higher priority first, traverse the sorted node list, and assign the real-time task to the node with the smallest load.

[0100] Step 5: Sort the priority tasks from high to low, adopt a round-robin scheduling strategy, and poll and allocate tasks based on load balancing;

[0101] The steps of allocating priority tasks to computing nodes using the cyclic scheduling strategy are as follows:

[0102] Take a weighted sum of the node's CPU, memory, storage, and network bandwidth usage to get the current node's load;

[0103] Sort the tasks in the priority task set from high to low priority;

[0104] Sort all calculation records according to their load from small to large;

[0105] Task allocation: Take the highest priority tasks in turn, calculate whether the available resources of the current node meet the requirements of the task, and allocate them if they do. If not, calculate whether the next node meets the requirements until the current task is allocated.

[0106] The task allocation step is repeated until all tasks are allocated to corresponding nodes.

[0107] Step 6: Sort common tasks by priority from high to low, use the least task node strategy, traverse all nodes to find the node with the least number of tasks and assign tasks;

[0108] In this embodiment, different types of tasks use different scheduling strategies to optimize their respective characteristics. For example, real-time tasks emphasize timeliness, priority tasks focus on load balancing, and ordinary tasks focus on full utilization of resources. This combination of strategies can comprehensively optimize system performance and meet the needs of different tasks.

[0109] Through load balancing and reasonable task allocation, we can avoid overloading certain nodes, which may lead to system performance degradation or crash. Regularly calculate the load and migrate resources to further ensure the stable operation of the system.

[0110] Different scheduling strategies can be adjusted according to actual conditions, which has high flexibility and scalability. The resource requirement estimation of new tasks and the resource acquisition method of existing tasks also enable the system to dynamically adapt to different workloads.

[0111] Each strategy is optimized for a specific type of task, which can improve the overall efficiency of the system while ensuring the priority of the task. Real-time tasks are processed quickly, high-priority tasks are load balanced, and ordinary tasks are maximized. These measures comprehensively improve the computing efficiency of the system.

[0112] By adopting the optimal scheduling methods for real-time tasks, priority tasks and ordinary tasks, we can not only ensure the timely processing of high-priority tasks, but also achieve system load balancing and efficient use of resources, ultimately improving the computing power scheduling effect and overall system performance in the distributed computing environment.

[0113] Step 7: Regularly calculate the current load of each node. If the load of a node exceeds the threshold, perform resource migration and migrate the task to the node with the lowest load.

[0114] Calculate the current load of each node:

[0115] Load i =α×CPU usage,i +β×Memory usage,i +γ×Storage usage,i +δ×Bandwidthusage,i ;

[0116] Where: Load i Indicates the current load of the i-th computing node; CPU usage,i Indicates the CPU usage of the i-th computing node; Memory usage,i Indicates the memory usage of the i-th computing node; Storage usage,i Indicates the storage usage of the i-th computing node; Bandwidth usage,i represents the network bandwidth utilization rate of the i-th computing node; α, β, γ, δ represent the weight coefficients of each resource utilization rate;

[0117] Traverse all nodes and check for nodes whose load exceeds the threshold:

[0118] Overloaded Nodes ={Node i |Load i >Load threshold};

[0119] Where: Overloaded Nodes Indicates the set of nodes whose load exceeds the threshold; Load i Indicates the current load of the i-th node; Load threshold Indicates the load threshold;

[0120] For each node whose load exceeds the threshold, a target node with the lowest load is selected for task migration;

[0121] One or more tasks are selected from the nodes whose load exceeds the threshold and migrated to the target node until the load of the node drops below the threshold.

[0122] The present invention obtains the tasks that need to be processed currently through the task management system, and divides the tasks into real-time tasks, priority tasks and ordinary task sets, and schedules the related tasks based on the set corresponding to the current task and the priority in the set using the corresponding scheduling strategy, thereby realizing resource scheduling and allocation in a distributed computing environment; the present invention effectively solves the technical problems of computing resource scheduling and allocation in a distributed environment through the steps of task classification, real-time resource monitoring, resource demand estimation, priority sorting, reasonable scheduling strategy, load calculation and resource migration, ensures timely processing of high priority, system load balancing, efficient use of resources and stable operation of the system, thereby improving the performance and reliability of the entire distributed computing environment.

[0123] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A computing power scheduling method in a distributed computing environment, characterized in that: The following steps are involved: Step 1: The task management system periodically obtains the tasks that need to be processed from the task queue and classifies the tasks to be processed into real-time tasks, priority tasks and ordinary tasks. The priority of real-time tasks is higher than that of priority tasks, and the priority of priority tasks is higher than that of ordinary tasks. Step 2: Collect resource usage of all computing nodes in real time based on the resource monitoring system; Step 3: Estimate resource requirements for new tasks using the trained linear regression model. For existing tasks, directly obtain resource requirements from the task description. Step 4: Sort the real-time tasks according to their priorities, adopt the least recently loaded node strategy, traverse all nodes to find the node with the least load and assign tasks; Step 5: Sort the priority tasks from high to low, adopt a round-robin scheduling strategy, and poll and allocate tasks based on load balancing; Step 6: Sort common tasks by priority from high to low, adopt the strategy of least task nodes, traverse all nodes to find the node with the least number of tasks and assign tasks; Step 7: Regularly calculate the current load of each node. If the load of a node exceeds the threshold, perform resource migration and migrate the task to the node with the lowest load.

2. The computing power scheduling method in a distributed computing environment according to claim 1, characterized in that: The steps of classifying the tasks to be processed into real-time tasks, priority tasks and common task sets in step 1 are as follows: Task acquisition and initialization: The task management system obtains all tasks that need to be processed from the task queue at every fixed time interval; Task classification: classify each task and classify it into corresponding task sets according to its importance and urgency; Task priority determination: Within each task set, sort the tasks within each task set according to their importance and urgency.

3. The computing power scheduling method in a distributed computing environment according to claim 1, characterized in that: The resource usage in step 2 includes: CPU Usage: Where: CPU usage Indicates the CPU usage of the computing node; CPU time,used,i Indicates the usage time of the i-th core; CPU total,time,i represents the total time of the i-th core; n represents the number of cores; Memory usage: Where: Memory usage Indicates the memory usage of the computing node; Memory total Indicates the total memory of the computing node; Memory used Indicates the used memory of the computing node; Storage usage: Where: Storage usage Indicates the storage usage of the computing node; Storage used Indicates the amount of storage used by the compute node; Storage total Indicates the total storage capacity of the computing node; Network bandwidth: Where: Bandwidth usage Indicates the bandwidth usage of the computing node; Bandwidth used Indicates the bandwidth used by the computing node; Banwidth total Indicates the total bandwidth of the computing node.

4. The computing power scheduling method in a distributed computing environment according to claim 1, characterized in that: The linear regression model in step 3 is as follows: M usage =β0+β1×Task Type +β2×Input Size +β3×Task Complexity +∈; Where: M usage represents the predicted resource requirements, including CPU requirements, memory requirements, storage requirements, and network bandwidth requirements; β0, β1, β2, and β3 represent regression coefficients; Task Type Indicates the task type; Input Size Indicates the amount of input data for a task; Task Complexity represents the task complexity; ∈ represents the error term.

5. The computing power scheduling method in a distributed computing environment according to claim 4, characterized in that: The task complexity is calculated as follows: Calculate the logical complexity of the task logic logic : Where: Steps o Indicates the complexity of the i-th step; Weight i represents the weight of the i-th step; Data structure complexity: Where: m represents the total number of data structure operations; Operations j Indicates the number of j-th operation; Costj j represents the complexity cost of the jth operation; The task complexity is obtained based on the logic complexity and data structure complexity: Complexity total =μ1×Complexity logic +μ2×Complexity data ; Where: μ1 and μ2 both represent weights.

6. The computing power scheduling method in a distributed computing environment according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4.1: Take the weighted sum of the node's CPU, memory, storage, and network bandwidth usage to get the current node load; Step 4.2: Sort all computing nodes in ascending order according to load and find the node with the smallest load; Step 4.3: Sort each real-time task according to its priority, assign tasks with higher priority first, traverse the sorted node list, and assign the real-time task to the node with the smallest load.

7. The computing power scheduling method in a distributed computing environment according to claim 1, characterized in that: The steps of allocating priority tasks to computing nodes using the cyclic scheduling strategy are as follows: Take a weighted sum of the node's CPU, memory, storage, and network bandwidth usage to get the current node's load; Sort the tasks in the priority task set from high to low priority; Sort all calculation records according to their load from small to large; Task allocation: Take the highest priority tasks in turn, calculate whether the available resources of the current node meet the requirements of the task, and allocate them if they do. If not, calculate whether the next node meets the requirements until the current task is allocated. The task allocation step is repeated until all tasks are allocated to corresponding nodes.

Images