Load balancing task scheduling method and system based on resource feedback in Gaia system

Abstract

The present invention discloses a load balancing task scheduling method and system based on resource feedback in the Gaia system, which can improve the cluster resource utilization and reduce the execution time of system tasks. The method can reallocate the tasks of each resource node when the cluster is unevenly loaded, thereby improving the cluster resource utilization and reducing the execution time of system tasks. The method monitors the load of resource nodes through the implemented real-time monitoring system Monitor, and divides the cluster into four areas of overload, light load, near saturation and poor saturation according to the area partitioning algorithm. Since the machine load in the overloaded area is too heavy, it affects the execution efficiency of the entire cluster. Therefore, a scheduling strategy based on the artificial firefly algorithm is used to schedule the tasks of the resource nodes in the overloaded area to the resource nodes in the poor saturation area, thereby improving the cluster utilization, execution efficiency and throughput efficiency of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a load balancing task scheduling method based on resource feedback in a Gaia system. Background Art

[0002] In recent years, with the explosive growth of data and the emergence of diverse needs, the development of big data computing systems has received increasing attention. Big data systems employ two computing methods: batch computing and stream computing. These two data computing methods often require different big data computing frameworks, which not only increases maintenance costs but also hinders the development of user needs. Gaia, based on this background and based on a national key R&D program, is a batch-stream integrated big data computing system.

[0003] Task scheduling is a key issue in big data computing systems. The task scheduling strategy in the Gaia system uses a round-robin scheduling algorithm, which assigns tasks to resource nodes sequentially without considering the size of each task or the performance differences between resource nodes. This results in severe data skew, leading to low cluster execution efficiency, poor throughput, and low resource utilization. Tasks in the Gaia system run in a multi-threaded mode, which improves system execution efficiency by splitting complex tasks. However, in distributed computing systems, tasks are not equally divided. Since the size of the tasks assigned to each resource node after the job is split varies greatly, and the performance of each resource node cannot be maintained consistently, distributed computing systems often experience uneven load, meaning some resource nodes are overwhelmed with tasks and others are idle.

[0004] Existing methods ignore the heterogeneity of tasks, which can easily lead to problems such as low cluster execution efficiency, poor throughput, and low resource utilization. Summary of the Invention

[0005] In view of this, the present invention provides a load balancing task scheduling method based on resource feedback in a Gaia system, which can improve cluster resource utilization and reduce the execution time of system tasks.

[0006] To achieve the above object, the technical solution of the present invention includes the following steps:

[0007] Step 1: Perform real-time resource performance monitoring on all resource nodes in the Gaia cluster system, calculate the load value of each resource node, and then obtain the load value of the entire Gaia cluster;

[0008] Store the load value of each resource node and the load balancing value of the cluster in the MongoDB database;

[0009] Step 2: Based on the offset of each resource node's load value relative to the entire Gaia cluster and the pre-set high and low thresholds, the Gaia cluster is divided into four regions: overload, light load, near saturation, and poor saturation. These regions are denoted as UPGroup, LPGroup, NSGroup, and DSGroup respectively.

[0010] Step 3: Based on the artificial firefly algorithm, task i on node j in the overloaded region UPGroup is scheduled to node p in the slightly saturated region DSGroup. The UPGroups are sorted in descending order by load value and are scheduled to the DSGroups in sequence until the cluster load value C_Load is less than the preset threshold β.

[0011] In the artificial firefly algorithm, task i on node j in the overload area UPGroup is used as the firefly, and node p in the DSGroup area is used as the target area of ​​the firefly. The objective function of the task is determined by the number of tasks of node j in UPGroup, the priority of task i in node j, the load value of the target node p, and the number of high-priority tasks on the target node p.

[0012] Furthermore, step 1 specifically includes the following steps:

[0013] Step 1.1: The resource monitoring system in the Gaia cluster collects performance indicators of the number of CPU cores, CPU utilization, memory utilization, and total memory size of each resource node in the Gaia cluster every 10 seconds. The comprehensive value of these four indicators represents the real-time resource performance of the current resource node.

[0014] Step 1.2: Calculate the load value of the resource node based on the four performance indicators.

[0015] Step 1.3: Put the calculated load value of the resource node into the MongoDB database.

[0016] Step 1.4: Based on the load value of each resource node, the load value of the entire Gaia cluster is calculated and stored in the MongoDB database. If the load value of the Gaia cluster is less than the set balance threshold, the load balancing task scheduling method ends; otherwise, proceed to step 2.

[0017] Furthermore, step 2 specifically includes the following steps:

[0018] Step 2.1: Calculate the average load value of the Gaia cluster based on the load value of each resource node in the Gaia cluster.

[0019] Step 2.2: Based on the load value of each resource node and the average load value of the Gaia cluster, calculate the load deviation of each resource node and the offset relative to the threshold.

[0020] Step 2.3: Set the high and low thresholds of the Gaia cluster, and allocate all resource nodes to four areas: overload, poor load, oversaturation, and near saturation according to their own offsets. These areas are denoted as UPGroup, LPGroup, NSGroup, and DSGroup respectively. The high and low thresholds of the Gaia cluster are heuristic empirical values.

[0021] Furthermore, the high threshold is set to h_threshold≥0.

[0022] If the offset of resource node j is greater than or equal to the high threshold h_threshold, node j belongs to the overload UPGroup area.

[0023] If the offset of the resource node j is less than the high threshold h_threshold and greater than or equal to 0, the node j belongs to the lightly loaded LPGroup area.

[0024] Set the lower threshold l_threshold ≤ 0.

[0025] If the offset of resource node j is less than the low threshold l_threshold, node j belongs to the poorly saturated DSGroup region.

[0026] If the offset of resource node j is greater than the low threshold l_threshold and less than 0, node j belongs to the near-saturation NSGroup area.

[0027] Furthermore, step 3 includes the following specific steps:

[0028] Step 3.1: Sort the resource nodes assigned to the overloaded area UPGroup from high to low according to the load value, and sort the tasks in a resource node from high to low according to the priority.

[0029] Step 3.2: Schedule the tasks in the waiting queue of the resource nodes in the overload area UPGroup to the poorly saturated area DSGroup in sequence.

[0030] Step 3.3: Read the current real-time resource performance metrics from the MongoDB database every 10 seconds.

[0031] Step 3.4: Update the current area division and repeat steps 3.1, 3.2 and 3.3 until the overload area UPGroup is empty or the poorly saturated area DSGroup is empty, and the load balancing task scheduling method is completed.

[0032] Furthermore, step 3 includes the following specific steps:

[0033] S301: Input firefly set N = {n1, n2, ..., n m}, m is the total number of fireflies; the number of iterations M and the set of overload area UPGroup, light load area LPGroup, near saturation area NSGroup, and poor saturation area DSGroup generated by the area partitioning algorithm in the previous section, sort the tasks in UPGroup according to the load value, set the tasks in UPGroup as fireflies, set the DSGroup area as the luminous area, find the optimal solution in the luminous area and schedule the tasks to the optimal resource node; first initialize the firefly step size s, the disappearance rate ρ of the fluorescein, the update rate γ of the fluorescein, and the update rate β of the decision domain. For each firefly in the firefly set, initialize its fluorescein to l0, the initial decision domain to r0, and the initial position to x i (t).

[0034] The fluorescein value of firefly i at time t is l t (t), ρ represents the disappearance rate of the fluorescent color in fireflies, γ represents the renewal rate of fluorescein, f(x i (t)) represents the objective function value of the position of firefly i at time t, l i (t-1) is the fluorescein value of firefly i at time t-1;

[0035] l i (t)=(1-ρ)l i (t-1)+γf(x i (t)), i=1, 2,…,m (7)

[0036] The neighbor set G of firefly i at time t i (t), where x i (t) represents the location of firefly i at time t, represents the decision domain range of firefly i at time t, x j (t) represents the position of firefly j at time t, and the fluorescein value of firefly j at time t is l j (t);

[0037]

[0038] The decision domain of the firefly at time t The update formula is formula (9), where r s represents the receptive domain range of the firefly, β represents the update rate of the decision domain, g t-1 represents the neighborhood threshold of the firefly at time t-1, G i (t-1) represents the neighborhood of the firefly at time t-1;

[0039]

[0040] For the firefly i in the set, the firefly's fluorescein value, neighborhood set and decision domain radius need to be updated according to formula (7), formula (8) and formula (9) respectively. For the neighborhood set G of the current firefly i, i (t) To calculate each firefly i moves to the neighborhood set G i The probability p of each individual j in (t) ij (t), and select the j direction with the highest probability as the moving direction of firefly i, update the position and the decision domain range of firefly t+1 Until the number of iterations M is completed, the global optimal solution is selected, that is, the most suitable resource node in the poorly saturated area; when UPGroup and DSGroup are not empty, the resource nodes in the overloaded area UPGroup are sorted from high to low according to the load value, and the tasks in the resource node are sorted from high to low according to the priority. The assigned tasks corresponding to the nodes in UPGroup are simulated as fireflies, and DSGroup is defined as the regional movement range of the fireflies.

[0041] S302: Schedule the tasks in the waiting queue of the nodes in the overload area to the poorly saturated area in sequence, and move firefly i to the position of the global optimal solution; where firefly i corresponds to task i in UPGroup, and the position of the global optimal solution corresponds to resource node j in DSGroup.

[0042] S303: Read the current resource performance indicators from MongoDB every 10 seconds; at this time, the real-time performance of the system changes, obtain the real-time resource node load value information from the MongoDB database, and re-partition the region according to the number of current task queues of the resource node.

[0043] S304: Update the current area division and repeat S301, S302 and S303 until the overload area is empty or the poor saturation area is empty, and the overall algorithm scheduling is completed.

[0044] The present invention also provides a load balancing task scheduling system based on resource feedback in a Gaia system, which includes a real-time resource monitoring module, a region division module and a scheduling module.

[0045] The real-time resource monitoring module is used to perform real-time resource performance monitoring on all resource nodes in the Gaia cluster system, calculate the load value of each resource node, and then obtain the load value of the entire Gaia cluster; and store the load value of each resource node and the load balancing value of the cluster in the MongoDB database.

[0046] The regional division module is used to divide the Gaia cluster into four regions: overload, light load, near saturation and poor saturation, which are respectively recorded as UPGroup, LPGroup, NSGroup and DSGroup.

[0047] The scheduling module schedules task i on node j in the overloaded area UPGroup to node p in the poorly saturated area DSGroup based on the artificial firefly algorithm. UPGroups are sorted in descending order according to the size of the load value and scheduled to DSGroups in turn until the cluster load value C_Load is less than the preset threshold β.

[0048] In the artificial firefly algorithm, task i on node j in the overload area UPGroup is used as the firefly, and node p in the DSGroup area is used as the target area of ​​the firefly. The objective function of the task is determined by the number of tasks of node j in UPGroup, the priority of task i in node j, the load value of the target node p, and the number of high-priority tasks on the target node p.

[0049] Furthermore, the real-time resource monitoring module includes a real-time cluster performance monitoring unit, a load value calculation unit, and a MongoDB storage unit.

[0050] The real-time cluster performance monitoring unit counts the performance indicators of the number of CPU cores, CPU utilization, memory utilization, and total memory size of each resource node in the Gaia cluster every 10 seconds. The comprehensive value of these four indicators represents the real-time resource performance of the current resource node.

[0051] The load value calculation unit is used to calculate the load value of the resource node according to the four performance indicators, and calculate the load value of the entire Gaia cluster based on the load value of each resource node.

[0052] MongoDB storage unit, used to store the load value of resource nodes and the load value of the entire Gaia cluster.

[0053] When the real-time resource monitoring module determines that the load value of the Gaia cluster is greater than or equal to the set balance threshold, it calls the region division module.

[0054] Beneficial effects:

[0055] 1. The present invention proposes a load balancing task scheduling method based on resource feedback. This method can reallocate the tasks of each resource node when the cluster is unevenly loaded, thereby improving the cluster resource utilization and reducing the execution time of system tasks. This method monitors the load of resource nodes through the implemented real-time monitoring system Monitor, and divides the cluster into four areas: overload, light load, near saturation, and poor saturation according to the area division algorithm. Since the machine load in the overloaded area is too heavy, it affects the execution efficiency of the entire cluster. Therefore, a scheduling strategy based on the artificial firefly algorithm is used to schedule the tasks of the resource nodes in the overloaded area to the resource nodes in the poor saturation area, thereby improving the cluster utilization, execution efficiency, and throughput efficiency of the system.

[0056] 2. The present invention designs a real-time monitoring system Monitor, which can monitor the performance indicators of each slave node (Slave) in real time, such as the number of CPU cores, CPU utilization, memory utilization, total memory size, etc., so as to obtain the load size of each resource node.

[0057] 2. This paper proposes a strategy for partitioning resource nodes based on their load. Each resource node in the Gaia cluster is divided into four zones: overloaded, lightly loaded, nearly saturated, and nearly saturated, based on their load relative to the cluster's average load. This divides the entire cluster into zones where load reduction is needed, zones where load can be increased, and zones where resource nodes can just handle the load. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic diagram of the overall framework of a task scheduling method based on resource feedback in a Gaia system adopted by the present invention;

[0059] Figure 2 This is a schematic diagram of the overall process of a task scheduling method based on resource feedback in a Gaia system adopted by the present invention;

[0060] Figure 3 A schematic diagram of a specific operation process of a task scheduling method based on resource feedback in a Gaia system according to an embodiment of the present invention;

[0061] Figure 4 Schematic diagram of the execution process of the real-time resource monitoring stage in an embodiment of the present invention;

[0062] Figure 5 A schematic diagram of the MongoDB data execution process in an embodiment of the present invention;

[0063] Figure 6Schematic diagram of the execution process of the region division stage in an embodiment of the present invention;

[0064] Figure 7 Schematic diagram of the execution process of the task scheduling algorithm stage based on artificial fireflies in an embodiment of the present invention; DETAILED DESCRIPTION

[0065] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0066] This example applies a load-balanced task scheduling method based on resource feedback to the Gaia big data analytics system. Gaia is a highly efficient, scalable, and next-generation big data analytics system for coexisting multiple computing models. Task scheduling is a key issue in distributed computing, and improving the efficiency of Gaia task scheduling is a challenge that must be addressed in the field of big data processing. This example incorporates the unique characteristics of the Gaia platform and applies a load-balanced task scheduling method based on resource feedback to reduce cluster load imbalances and improve task execution efficiency, throughput efficiency, and overall cluster resource utilization.

[0067] In this embodiment, a load balancing task scheduling method based on resource feedback in a Gaia system is described. The overall framework is as follows: Figure 1 As shown, the present invention first obtains the load value of each resource node and the load balance value of the entire cluster by real-time monitoring of the performance of the cluster, and stores the load value information in real-time in the MongoDB database. The smaller the load balance value of the cluster, the more balanced the load of the entire cluster. Therefore, when the load balance value is less than the threshold, it means that the cluster is in a balanced state and the algorithm ends; otherwise, the entire cluster is divided according to the principles of the regional division algorithm, and the cluster is divided into four regions: overload, light load, near saturation, and poor saturation. Finally, a task scheduling algorithm based on the artificial firefly algorithm is used in combination with the priority of the task to migrate the tasks in the overloaded area to the resource nodes in the poor saturation area, so as to reduce the load balance value of the entire cluster and improve the overall execution efficiency of the task. The specific modules in this embodiment include three parts: real-time resource monitoring, regional division, and load balancing task scheduling based on artificial fireflies.

[0068] The specific operation process of the embodiment of the present invention is as follows Figure 2 As shown, the following steps are included:

[0069] Step 1: Perform real-time resource performance monitoring on all resource nodes in the Gaia cluster system, calculate the load value of each resource node, and then obtain the load value of the entire Gaia cluster.

[0070] The load value of each resource node and the load balancing value of the cluster are stored in the MongoDB database. Step 1 is used to collect important real-time system information for the subsequent regional division stage and the task scheduling stage based on the artificial firefly algorithm. It is divided into the following specific steps:

[0071] Step 1.1: The resource monitoring system collects statistics every 10 seconds on the number of CPU cores, CPU utilization, memory utilization, and total memory size of each node in the cluster. These four metrics reflect the current load capacity of the node. Therefore, the combined value of these four metrics can be used to represent the real-time performance of the resource node.

[0072] Step 1.2: Calculate the node's load value based on these performance indicators. The load value represents the amount of data that the node can run per unit time. A smaller load value indicates a better load-bearing ability of the node, meaning it can handle more tasks. Conversely, a larger load value indicates a worse load-bearing ability of the node, and the number of tasks performed by the node needs to be reduced.

[0073] Step 1.3: Put the calculated load value information into the MongoDB database.

[0074] Step 1.4: Based on the load value of each resource node, the load value of the entire cluster can be calculated. If the cluster load value is less than the threshold, it means that the cluster load is in a balanced state and the algorithm ends. Otherwise, it enters the regional division stage.

[0075] Step 2: Based on the offset of each resource node's load value relative to the entire Gaia cluster and the pre-set high and low thresholds, the Gaia cluster is divided into four regions: overload, light load, near saturation, and poor saturation. They are denoted as UPGroup, LPGroup, NSGroup, and DSGroup respectively.

[0076] Step 2 includes the following specific steps:

[0077] Step 2.1: The average load value of the cluster can be calculated based on the load value in the cluster.

[0078] Step 2.2: Based on the individual load values ​​and the overall average load value obtained in step 2.1, the load deviation of each resource node and the offset relative to the threshold can be calculated.

[0079] Step 2.3: Allocate all resource nodes to four zones: overload, underload, oversaturation, and near-saturation, based on their offsets and the corresponding relationship between the high and low thresholds of the cluster. The high and low thresholds of the cluster are heuristic empirical values.

[0080] Step 3: Based on the artificial firefly algorithm, task i on node j in the overloaded region UPGroup is scheduled to node p in the slightly saturated region DSGroup. The UPGroups are sorted in descending order by load value and are scheduled to the DSGroups in sequence until the cluster load value C_Load is less than the preset threshold β.

[0081] In the artificial firefly algorithm, task i on node j in the overload area UPGroup is used as the firefly, and node p in the DSGroup area is used as the target area of ​​the firefly. The objective function of the task is determined by the number of tasks of node j in UPGroup, the priority of task i in node j, the load value of the target node p, and the number of high-priority tasks on the target node p.

[0082] For the four regions of overload, poor load, near saturation, and poor saturation obtained from the previous region division stage, a resource node assigned to the overload region means that the load borne by this resource node is too heavy. A resource node assigned to the poor load or near saturation region means that the load of this resource node is within the acceptable range. A resource node assigned to the poor saturation region means that the resource of this node is relatively idle and can handle more tasks.

[0083] Step 3.1: Sort the nodes assigned to the overloaded area by load value from high to low, and sort the tasks in a node by priority from high to low;

[0084] Step 3.2: Schedule the tasks in the waiting queue of the nodes in the overly complex area to the poorly saturated area in sequence;

[0085] Step 3.3: Read the current resource performance metrics from MongoDB every 10 seconds.

[0086] Step 3.4: Update the current area division and repeat steps 3.1, 3.2 and 3.3 until the overload area is empty or the poor saturation area is empty, and the overall algorithm scheduling is completed.

[0087] Example 1

[0088] In an embodiment of the present invention, a method for scheduling load-balanced tasks based on resource feedback in a Gaia system includes the following steps:

[0089] Step 1: Real-time resource monitoring phase, which collects important system real-time information for the subsequent regional division phase and task scheduling phase based on artificial firefly algorithm. The execution process of the real-time resource monitoring phase is as follows: Figure 4 As shown, there are 6 resource nodes in the cluster of this embodiment;

[0090] Step 1.1: The resource monitoring system collects real-time resource performance data for each resource node every 10 seconds. The most representative indicators of computing resources are the number of CPU cores, CPU utilization, memory utilization, and total memory size. These four indicators reflect the current load capacity of the node. Therefore, the comprehensive value of the four indicators, CPU cores, CPU utilization, memory utilization, and total memory size, can be used to represent the real-time performance of the resource node. The calculation of resource node performance indicators is shown in formula (1), where C1 and C2 are constants, Core_Num j represents the number of CPU cores of resource node j, ω 1j represents the CPU utilization of resource node j, ω 2j represents the memory utilization of resource node j, ω 3j Indicates the total memory size of resource node j and the performance indicator Metric of the resource node j Determined by the number of CPU cores, CPU utilization, and free memory size, Metric j Perform normalization processing;

[0091]

[0092] Step 1.2: Based on these performance indicators, the load value of each resource node is calculated using a formula to obtain the load balancing value of the entire cluster. The load value represents the task load on the resource node relative to the node's current performance. If the load value is smaller, the node has a better ability to bear the load, that is, it can bear more tasks; conversely, the larger the load value, the worse the node's ability to bear the load, and the tasks executed by the node need to be reduced. The definition of the load value is shown in Formula 2. TaskNum j Indicates the number of tasks in the task queue on resource node j, Metric j Represents the performance indicator of resource node j, Metric j The larger the value, the better the resource performance is, and the more tasks can be processed per unit time. Conversely, the lower the value, the worse the resource performance is, and the fewer tasks can be processed per unit time. The cluster load average is defined as shown in Formula 3.

[0093]

[0094]

[0095] Step 1.3: Put the calculated load value information into the MongoDB database. The actual execution process of the MongoDB task is as follows: Figure 5As shown in the figure, the application only knows that it is connected to a mongod. The mongod is the router used to control the sharded cluster and provide queries. The mongod router stores the correspondence between data and shards and is responsible for distributing data requests. When the mongod responds to the request, the router collects all the data and returns it to the client.

[0096] Step 1.4: Based on the load value of each resource node, the load value of the entire cluster can be calculated. If the cluster load value is less than the threshold, it means that the cluster load is in a balanced state and the algorithm ends. Otherwise, for each resource node j in the cluster, the getOperatingSystemMXBea method in the ManagementFactory management factory class of the management interface for monitoring and managing the Java virtual machine (JVM) is used to obtain the underlying performance indicators of the resource node and calculate the number of CPU cores Core_Num of node j. j 、CPU utilization ω 1j 、Free memory ω 2j The cluster load value indicates the degree of load balancing of the entire cluster. As shown in Formula 4, if the cluster load value C_Load is less than the threshold β, it means that the overall cluster load is in a balanced state, and the algorithm ends. Otherwise, the larger the cluster load value, the higher the degree of cluster load imbalance. At this time, the load value of each node and the cluster load value need to be stored in the MongoDB database to prepare for the next stage of regional division and task rescheduling.

[0097]

[0098] Step 2: In the area division phase, the entire cluster is divided into four areas: overload, light load, near saturation and poor saturation, which are marked as UPGroup, LPGroup, NSGroup and DSGroup respectively. The execution process of the area division phase is as follows: Figure 6 As shown; In this embodiment, the cluster consists of 6 TaskManagers. According to the resource monitoring system proposed in the previous section, the real-time load value of each TaskManager is stored in the MongoDB database, and the load value of each node in the current resource is re-extracted from the database every 10 seconds. jThe cluster load value C_Load is used to calculate the offset of each node relative to the cluster average load. In this example, the offsets of TaskManager 1, TaskManager 2, TaskManager 3, TaskManager 4, TaskManager 5, and TaskManager 6 are 0.91, 0.71, 0.81, 0.13, 0.21, and 0.88, respectively. The high threshold of the cluster is 0.83 and the low threshold is 0.27. According to the zoning algorithm, the allocation result is as follows: Figure 4 As shown in the right figure, TaskManager 1 and TaskManager 6 are assigned to UPGroup, TaskManager 2 and TaskManager 3 are assigned to NSGroup, and TaskManager 4 and TaskManager 5 are assigned to DSGroup.

[0099] Step 2.1: Calculate the load value of each resource node j and cluster load average The difference δ j , as shown in Formula 5;

[0100]

[0101] Step 2.2: Based on the individual load values ​​and the overall average load value obtained in step 2.1, the load deviation of each resource node and the offset relative to the threshold can be calculated;

[0102] Step 2.3: All resource nodes are assigned to four regions: overload, poor load, oversaturation, and near saturation according to their own offsets and the corresponding relationship between the high and low thresholds of the cluster. The high and low thresholds of the cluster are heuristic empirical values. The cluster regions are divided by defining the offset, as shown in Formula 6. If the offset of resource node j is greater than or equal to the high threshold h_threshold (h_threshold ≥ 0), then node j belongs to the overload UPGroup region; if the offset of resource node j is less than the high threshold h_threshold and greater than or equal to 0, then node j belongs to the light load LPGroup region; if the offset of resource node j is less than the low threshold l_threshold (l_threshold ≤ 0), then node j belongs to the poor saturation DSGroup region; if the offset of resource node j is greater than the low threshold l_threshold and less than 0, then node j belongs to the near saturation NSGroup region;

[0103] offset j =δ j -β,j=1,2,…m (6)

[0104] Step 3: Task scheduling phase based on artificial firefly algorithm. The execution process of this task scheduling phase is as follows: Figure 7 As shown in the figure, based on the area partitioning algorithm in the previous section, task i on node j in the overloaded area UPGroup is scheduled to node p in the poorly saturated area DSGroup according to the task scheduling strategy based on the artificial firefly algorithm. UPGroup is sorted in descending order according to the size of the load value and scheduled to DSGroup in sequence until the cluster load value C_Load is less than the threshold β. In this algorithm, task i is defined as a firefly, and node p in the DSGroup area is defined as the target area of ​​the firefly. The objective function of the task is determined by the number of tasks on node j in UPGroup, the priority of task i in node j, the load value of the target node p, and the number of high-priority tasks on the target node p.

[0105] Let N = {n1, n2, ..., n m} is a set of fireflies, the fluorescein of each firefly is initialized to l0, and the decision domain is r0. The fluorescein value of firefly i at time t is shown in formula 7, where ρ represents the disappearance rate of the fluorescent color in the firefly, γ represents the update rate of the fluorescein, and f(x i (t)) represents the objective function value of the position of firefly i at time t.

[0106] l i (t)=(1-ρ)l i (t-1)+γf(x i (t)), i=1, 2,…,m (7)

[0107] The neighbor set G of firefly i at time t i (t), as shown in Formula 8, where x i (t) represents the location of firefly i at time t, Represents the decision domain range of firefly i at time t.

[0108]

[0109] The decision domain of the firefly at time t The update formula is shown in Formula 9, where r s represents the receptive domain range of the firefly, β represents the update rate of the decision domain, g t-1 represents the neighborhood threshold of the firefly at time t-1, G i (t-1) represents the neighborhood of the firefly at time t-1.

[0110]

[0111] Step 3.1: Input firefly set N = {n1, n2, ..., n m}, the number of iterations M, and the set of overload areas UPGroup, light load areas LPGroup, near saturation areas NSGroup, and poorly saturated areas DSGroup generated by the area partitioning algorithm in the previous section. Sort the tasks in the UPGroup according to the load value, set the tasks in the UPGroup as fireflies, set the DSGroup area as the luminous area, find the optimal solution in the luminous area and schedule the tasks to the optimal resource node. During the algorithm, it is first necessary to initialize the firefly step size s, the disappearance rate ρ of the fluorescein, the update rate γ of the fluorescein, and the update rate β of the decision domain. For each firefly in the firefly set, it is necessary to initialize its fluorescein to l0, the initial decision domain to r0, and the initial position to x. i (t), for the firefly i in the set, the firefly's fluorescein value, neighborhood set and decision region radius need to be updated according to formula 7, formula 8 and formula 9 respectively. For the neighborhood set G of the firefly i (t) To calculate each firefly i moves to the neighborhood set G i The probability p of each individual j in (t) ij (t), and select the j direction with the highest probability as the moving direction of firefly i, update the position and the decision domain range of firefly t+1 Until the number of iterations M is completed, the global optimal solution is selected, that is, the most suitable resource node in the poorly saturated area; Figure 7 Step (a) is the original regional allocation and the corresponding scheduling of tasks and resource nodes in this embodiment. When both UPGroup and DSGroup are not empty, the resource nodes in the overloaded area UPGroup are sorted from high to low according to the load value, and the tasks in the resource nodes are sorted from high to low according to the priority. The assigned tasks corresponding to the nodes in the UPGroup are simulated as fireflies, and DSGroup is defined as the regional movement range of the fireflies.

[0112] Step 3.2: Schedule the tasks in the waiting queue of the nodes in the overly complex area to the poorly saturated area in sequence, and move firefly i (task i in UPGroup) to the position of the global optimal solution (resource node j in DSGroup); in this example, the resource nodes in UPGroup are TaskManager 1 and TaskManager 6, and their offsets are 0.91 and 0.87 respectively. The priorities of tasks Task1, Task2, and Task3 in TaskManager 1 are low, low, and high respectively, and the priority of task Task10 in TaskManager 1 is high; the offsets of TaskManager 4 and TaskManager 5 in DSGroup are 0.13 and 0.28 respectively, the resource nodes in NSGroup are TaskManager 2 and TaskManager 3, and the LPGroup area is empty. Because the order of rescheduling tasks in the UPGroup is first sorted by the offset in the resource node from high to low, and then sorted by the priority of the tasks in the same resource node from high to low, the first task scheduled in the UPGroup is Task 3 in TaskManager 1. Since the offset of TaskManager 4 in the DSGroup is lower than that of TaskManager 5, Task 3 in TaskManager 1 is scheduled to TaskManager 4.

[0113] Step 3.3: Read the current resource performance indicators from MongoDB every 10 seconds. If the system's real-time performance changes, obtain the real-time resource node load value information from the MongoDB database and repartition the nodes based on the number of task queues in the node. The cluster task allocation is now the same as in step (b). TaskManager 1's offset is updated to 0.64 and is in the LPGroup area. The next task to be scheduled is Task 10 of TaskManager 6 in the UPGroup. The offsets of TaskManager 4 and TaskManager 5 in the DSGroup are 0.21 and 0.26, respectively, so Task 10 is scheduled to TaskManager 4.

[0114] Step 3.4: Update the current region division and repeat steps 3.1, 3.2, and 3.3 until the overload region is empty or the poorly saturated region is empty. The overall algorithm scheduling is complete. In this embodiment of the present invention, due to changes in resource node performance, the cluster is repartitioned. TaskManager 4 is assigned to the LPGroup and TaskManager 6 is assigned to the DSGroup region. At this point, the UPGroup is empty, and the algorithm ends.

[0115] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A load balancing task scheduling method based on resource feedback in a Gaia system, characterized in that: The method includes the following steps: Step 1: Perform real-time resource performance monitoring on all resource nodes in the Gaia cluster system, calculate the load value of each resource node, and then obtain the load value of the entire Gaia cluster; Store the load value of each resource node and the load balancing value of the cluster in the MongoDB database; Step 2: Based on the offset of each resource node's load value relative to the entire Gaia cluster and the pre-set high and low thresholds, the Gaia cluster is divided into four regions: overload, light load, near saturation, and poor saturation. These regions are denoted as UPGroup, LPGroup, NSGroup, and DSGroup respectively. Step 3: Based on the artificial firefly algorithm, task i on node j in the overloaded region UPGroup is scheduled to node p in the slightly saturated region DSGroup. Tasks on the UPGroup are sorted in descending order by load value and are scheduled to the DSGroup in sequence until the cluster load value C_Load is less than the preset threshold. In the artificial firefly algorithm, task i on node j in the overload area UPGroup is used as a firefly, and node p on the DSGroup area is used as the target area of ​​the firefly. The objective function of the task is determined by the number of tasks of node j on UPGroup, the priority of task i in node j, the load value of the target node p, and the number of high-priority tasks on the target node p.

2. The load balancing task scheduling method according to claim 1, wherein: The step 1 specifically includes the following steps: Step 1.1: The resource monitoring system in the Gaia cluster collects statistics every 10 seconds on the performance indicators of the number of CPU cores, CPU utilization, memory utilization, and total memory size of each resource node in the Gaia cluster. The comprehensive value of these four indicators represents the real-time resource performance of the current resource node. Step 1.2: Calculate the load value of the resource node based on the four performance indicators; Step 1.3: Store the calculated resource node load value into the MongoDB database; Step 1.4: Based on the load value of each resource node, the load value of the entire Gaia cluster is calculated and stored in the MongoDB database. If the load value of the Gaia cluster is less than the set balance threshold, the load balancing task scheduling method ends; otherwise, proceed to step 2.

3. The load balancing task scheduling method according to claim 1 or 2, wherein: The step 2 specifically includes the following steps: Step 2.1: Calculate the average load value of the Gaia cluster based on the load value of each resource node in the Gaia cluster; Step 2.2: Based on the load value of each resource node and the average load value of the Gaia cluster, calculate the load deviation of each resource node and the offset relative to the threshold; Step 2.3: Set the high and low thresholds of the Gaia cluster, and allocate all resource nodes to four areas: overload, poor load, oversaturation, and near saturation according to their own offsets. These areas are denoted as UPGroup, LPGroup, NSGroup, and DSGroup respectively. The high and low thresholds of the Gaia cluster are heuristic empirical values.

4. The load balancing task scheduling method according to claim 3, wherein: Set the high threshold to h_threshold ≥ 0; If the offset of resource node j is greater than or equal to the high threshold h_threshold, node j belongs to the overload UPGroup area; If the offset of resource node j is less than the high threshold h_threshold and greater than or equal to 0, node j belongs to the lightly loaded LPGroup area; Set the low threshold l_threshold≤0; If the offset of resource node j is less than the low threshold l_threshold, node j belongs to the poorly saturated DSGroup region; If the offset of resource node j is greater than the low threshold l_threshold and less than 0, node j belongs to the near-saturation NSGroup area.

5. The load balancing task scheduling method according to any one of claims 1 to 4, characterized in that: The step 3 includes the following specific steps: Step 3.1: Sort the resource nodes assigned to the overloaded UPGroup from high to low according to their load values, and sort the tasks in a resource node from high to low according to their priorities; Step 3.2: Schedule the tasks in the waiting queue of the resource nodes in the overloaded area UPGroup to the poorly saturated area DSGroup in sequence; Step 3.3: Read the current real-time resource performance indicators from the MongoDB database every 10 seconds; Step 3.4: Update the current area division and repeat steps 3.1, 3.2 and 3.3 until the overload area UPGroup is empty or the poorly saturated area DSGroup is empty, and the load balancing task scheduling method is completed.

6. The load balancing task scheduling method according to any one of claims 1 to 4, characterized in that: The step 3 includes the following specific steps: S301: Input firefly set N = {n1, n2, ..., n m }, m is the total number of fireflies; the number of iterations M and the set of overload area UPGroup, light load area LPGroup, near saturation area NSGroup, and poor saturation area DSGroup generated by the area partitioning algorithm in the previous section, sort the tasks in UPGroup according to the load value, set the tasks in UPGroup as fireflies, set the DSGroup area as the luminous area, find the optimal solution in the luminous area and schedule the tasks to the optimal resource node; first initialize the firefly step size s, the disappearance rate ρ of the fluorescein, the update rate γ of the fluorescein, and the update rate β of the decision domain. For each firefly in the firefly set, initialize its fluorescein to l0, the initial decision domain to r0, and the initial position to x i (t), The fluorescein value of firefly i at time t is l i (t), ρ represents the disappearance rate of the fluorescent color in fireflies, γ represents the renewal rate of fluorescein, f(x i (t)) represents the objective function value of the position of firefly i at time t, l i (t-1) is the fluorescein value of firefly i at time t-1; l i (t) =(1- ρ) l i (t-1)+γf (x i (t)), i=1,2, …,m (7) The neighbor set G of firefly i at time t i (t), where x i (t) represents the location of firefly i at time t, r d i (t) represents the decision domain of firefly i at time t, x j (t) represents the position of firefly j at time t, and the fluorescein value of firefly j at time t is l j (t); G i (t) =() {j :|| x j (t)- x i (t)|| < r d i (t); l i (t)< l j (t)} (8) The decision domain r of the firefly at time t d i (t) The update formula is formula (9), where r s represents the receptive domain range of the firefly, β represents the update rate of the decision domain, g t-1 represents the neighborhood threshold of the firefly at time t-1, G i (t-1) represents the neighborhood of the firefly at time t-1; r d i (t) = min{r s , max{0, r d i (t-1), β(g t-1 -|G i (t-1)|)}} (9) For the firefly i in the set, the firefly's fluorescein value, neighborhood set and decision domain radius need to be updated according to formula (7), formula (8) and formula (9) respectively. For the neighborhood set G of the current firefly i, i (t) To calculate each firefly i moves to the neighborhood set G i The probability p of each individual j in (t) ij (t), and select the direction j with the highest probability as the moving direction of firefly i, update the position and the decision domain range r of firefly t+1 d i (t+1), until the number of iterations M is completed, the global optimal solution is selected, that is, the most suitable resource node in the poorly saturated area; when both UPGroup and DSGroup are not empty, the resource nodes in the overloaded area UPGroup are sorted from high to low according to the load value, and the tasks in the resource node are sorted from high to low according to the priority. The assigned tasks corresponding to the nodes in the UPGroup are simulated as fireflies, and DSGroup is defined as the regional movement range of the firefly; S302: Schedule tasks in the waiting queue of nodes in the overloaded region to the poorly saturated region in sequence, and move firefly i to the position of the global optimal solution; where firefly i corresponds to task i in the UPGroup, and the position of the global optimal solution corresponds to resource node j in the DSGroup; S303: Read the current resource performance indicators from MongoDB every 10 seconds. If the real-time performance of the system changes, obtain the real-time resource node load value information from the MongoDB database and re-partition the resource nodes based on the number of task queues in the resource nodes. S304: Update the current area division and repeat S301, S302 and S303 until the overload area is empty or the poor saturation area is empty, and the overall algorithm scheduling is completed.

7. The load balancing task scheduling system based on resource feedback in the Gaia system is characterized by: Includes real-time resource monitoring module, area division module and scheduling module; The real-time resource monitoring module is used to perform real-time resource performance monitoring on all resource nodes in the Gaia cluster system and calculate the load value of each resource node to obtain the load value of the entire Gaia cluster; The load value of each resource node and the load balancing value of the cluster are stored in the MongoDB database; The region division module is used to divide the Gaia cluster into four regions: overload, light load, near saturation, and poor saturation, which are denoted as UPGroup, LPGroup, NSGroup, and DSGroup respectively. The scheduling module schedules task i on node j in the overloaded region UPGroup to node p in the poorly saturated region DSGroup based on the artificial firefly algorithm. UPGroups are sorted in descending order according to load values ​​and are scheduled to DSGroups in sequence until the cluster load value C_Load is less than a preset threshold. In the artificial firefly algorithm, task i on node j in the overload area UPGroup is used as a firefly, and node p on the DSGroup area is used as the target area of ​​the firefly. The objective function of the task is determined by the number of tasks of node j on UPGroup, the priority of task i in node j, the load value of the target node p, and the number of high-priority tasks on the target node p.

8. The load balancing task scheduling system according to claim 7, wherein: The real-time resource monitoring module includes a real-time cluster performance monitoring unit, a load value calculation unit, and a MongoDB storage unit; The real-time cluster performance monitoring unit counts the performance indicators of the number of CPU cores, CPU utilization, memory utilization, and total memory size of each resource node in the Gaia cluster every 10 seconds. The comprehensive value of these four indicators represents the real-time resource performance of the current resource node; The load value calculation unit is used to calculate the load value of the resource node according to the four performance indicators, and calculate the load value of the entire Gaia cluster based on the load value of each resource node; The MongoDB storage unit is used to store the load value of the resource node and the load value of the entire Gaia cluster; The real-time resource monitoring module calls the region division module when determining that the load value of the Gaia cluster is greater than or equal to the set balance threshold.

Images