A task scheduling method and device for distributed load balancing

Abstract

The application discloses a kind of distributed load balancing task scheduling method and device, comprising: scheduling center is determined from the key performance indicators of each performance indicator of node and the task type matching of to-be-executed task, based on the performance data of the key performance indicators of each node of node monitor acquisition, determine the resource utilization of each node under key performance indicators, determine the weight calculation rule matched with resource utilization, so as to increase the flexibility, accuracy and reliability of selected node, based on the performance data of each performance indicator of each node of node monitor acquisition, determine the weighted weight of each node under each performance indicator, for any node, based on weight calculation rule and the weighted weight of node under each performance indicator, determine the scheduling weight of node, according to the scheduling weight of each node, determine the node of to-be-executed task execution, so as to increase the accuracy and comprehensiveness of selected node.

Description

Technical Field

[0001] This invention relates to the field of financial technology (Fintech), and more particularly to a distributed load balancing task scheduling method and apparatus. Background Technology

[0002] With the development of computer technology, more and more technologies (such as blockchain, cloud computing or big data) are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology, and big data technology is no exception. However, due to the security and real-time requirements of the financial and payment industries, higher requirements are also placed on the task scheduling methods in big data technology.

[0003] Currently, distributed task scheduling methods include the following three approaches:

[0004] 1. Quartz Job Scheduler: An open-source task scheduling framework written in Java, used to implement many-to-many relationships between tasks and triggers, enabling multiple tasks to be associated with different triggers.

[0005] 2. Elastic-job: An elastic distributed task scheduling system based on database distributed functionality. It uses Zookeeper as a registry center for registering, controlling, and coordinating the execution of distributed tasks.

[0006] 3. Saturn: A task scheduling system based on a Zookeeper cluster architecture. It is used for job management, statistical reporting, system configuration, and other functions.

[0007] However, the first method (Quartz Job scheduler) for distributed task scheduling suffers from load imbalance among nodes. The second method (Elastic-job) and the third method (Saturn) are based on cluster architectures, which require multiple scheduling servers to participate, making the distributed system complex and unable to achieve lightweight task scheduling.

[0008] Most importantly, existing technologies cannot flexibly select the node to execute a task based on the task type while achieving load balancing across nodes. The node selection is inflexible and inaccurate.

[0009] Therefore, a task scheduling method is needed to increase the flexibility and accuracy of node selection while achieving load balancing across nodes. Summary of the Invention

[0010] This invention provides a distributed load balancing task scheduling method and apparatus to achieve load balancing among nodes and increase the flexibility and accuracy of node selection.

[0011] In a first aspect, embodiments of the present invention provide a task scheduling method for distributed load balancing, comprising:

[0012] The scheduling center determines the key performance indicators that match the task type of the task to be executed from the various performance indicators of the nodes.

[0013] The scheduling center determines the resource utilization rate of each node under the key performance indicators based on the performance data of the key performance indicators collected by the node monitors of each node.

[0014] The scheduling center determines a weight calculation rule that matches the resource utilization rate; wherein, in the weight calculation rule, the indicator weights of the key performance indicators are positively correlated with the resource utilization rate; and the indicator weights of the non-key performance indicators are negatively correlated with the resource utilization rate.

[0015] The scheduling center determines the weighting of each node under each performance indicator based on the performance data of each performance indicator collected by the node monitor of each node.

[0016] The scheduling center determines the scheduling weight of any node based on the weight calculation rules and the weighted weight of the node under each performance index.

[0017] The scheduling center determines the node to execute the task to be executed based on the scheduling weight of each node.

[0018] In the above technical solution, when selecting nodes to execute tasks, key performance indicators are determined based on the task type. Based on these key performance indicators, the scheduling weight of each node is determined by combining the weighted weights of other performance indicators. This increases the flexibility, accuracy, and reliability of node selection. Specifically, the weight calculation rules are matched according to resource utilization, which is equivalent to matching the indicator weights of key performance indicators with the indicator weights of non-key performance indicators to obtain the scheduling weight, thereby increasing the accuracy and comprehensiveness of node selection.

[0019] The scheduling center determines the resource utilization rate of each node under the key performance indicators based on the performance data of the key performance indicators collected by the node monitors of each node, including:

[0020] For any given node, the scheduling center determines the individual resource utilization rate of the node under the key performance indicators based on the node's maximum available resources and actual available resources under the key performance indicators.

[0021] The scheduling center determines the resource utilization rate of each node under the key performance indicators based on the individual resource utilization rate of each node under the key performance indicators.

[0022] In the above technical solution, the resource utilization rate is determined based on the maximum data and actual available data of all nodes for key performance indicators, which can reflect the usage of all nodes. The resource utilization rate is used to match the weight calculation rules, thereby improving the comprehensiveness and accuracy of the matching weight calculation rules.

[0023] The scheduling center determines the weighted weight of each node for each performance indicator based on the performance data of each node's node monitor, including:

[0024] For any performance metric, the scheduling center sorts the actual available resources of each node under the performance metric and divides the sorting results into multiple intervals.

[0025] The weighted weight of the node under the performance index is determined based on the preset value of the interval where the node is located.

[0026] In the above technical solution, the complexity of assigning weights to each node is reduced by dividing the interval, and the weights are assigned by sorting to distinguish the current performance of each node for any performance indicator, so as to select nodes with good performance in subsequent node selection, thereby increasing the accuracy of node selection.

[0027] The scheduling center determines the scheduling weight of any node based on the weight calculation rules and the weighted weights of the node under various performance indicators, including:

[0028] For any given node, the scheduling center retrieves the node's scheduling weight in the previous task scheduling from the task scheduling database; and determines the node's scheduling weight based on the scheduling weight in the previous task scheduling, the weight of each performance indicator, and the weighted average weight of each performance indicator.

[0029] In the above technical solution, the performance of a node is determined by introducing the scheduling weight in the previous task scheduling. If the scheduling weight in the previous task scheduling is large, it indicates that the node has good performance and high task execution efficiency. If the scheduling weight in the previous task scheduling is small, it indicates that the node has poor performance and low task execution efficiency. This method is used to determine the node with better performance and increase the accuracy of node selection.

[0030] Optionally, there may be multiple weight calculation rules;

[0031] When resource utilization exceeds a set threshold, the weights of each performance indicator are equal.

[0032] When the resource utilization rate is lower than the set threshold, the weight of key performance indicators is higher than the weight of non-key performance indicators.

[0033] In the above technical solution, the weight calculation rules include the weights of key performance indicators and non-key performance indicators, which are used to indicate the importance of performance indicators. This enables the selection of nodes based on task type, improving the flexibility and accuracy of node selection.

[0034] Optionally, after determining the node to execute the task to be executed, the method further includes:

[0035] The scheduling center adjusts the scheduling weight of the node executing the task to be executed according to the reduction adjustment rules, and obtains the scheduling weight of the node executing the task in the previous task scheduling.

[0036] In the above technical solution, for a node that is determined to perform a task, the scheduling weight of the node is reduced so that the reduced scheduling weight can be used as the scheduling weight when selecting a node in the next time. This prevents the scheduling weight from increasing indefinitely, which would cause other nodes to be unable to be selected.

[0037] Optionally, the same node can be configured with two node monitors that serve as both primary and backup for each other;

[0038] For any node monitor, after the node monitoring center determines that the node monitor is abnormal, it sends an overwrite file to the node monitor; the overwrite file is used to instruct the node monitor to restart; after determining that both the primary and backup node monitors have failed to restart, an alarm message is issued.

[0039] In the above technical solution, a real-time monitoring node monitor is used to prevent the performance data of each node from not being updated in real time, i.e., non-real-time data, thereby reducing the probability of abnormal node selection and increasing the accuracy of node selection.

[0040] Secondly, embodiments of the present invention provide a distributed load balancing task scheduling device, comprising:

[0041] The matching module is used to determine the key performance indicators that match the task type of the task to be executed from the various performance indicators of the node.

[0042] The processing module is used to determine the resource utilization rate of each node under the key performance indicators based on the performance data of the key performance indicators collected by the node monitor of each node.

[0043] Determine a weighting calculation rule that matches the resource utilization rate; wherein, in the weighting calculation rule, the indicator weights of the key performance indicators are positively correlated with the resource utilization rate; and the indicator weights of the non-key performance indicators are negatively correlated with the resource utilization rate.

[0044] Based on the performance data of each node collected by the node monitor of each node, the weighting weight of each node under each performance indicator is determined.

[0045] For any given node, the scheduling weight of the node is determined based on the weight calculation rules and the weighted weight of the node under each performance metric.

[0046] Based on the scheduling weights of each node, the node to execute the task to be executed is determined.

[0047] Optionally, the processing module is specifically used for:

[0048] For any node, determine the individual resource utilization rate of the node under the key performance indicators based on the node's maximum available resources and actual available resources under the key performance indicators.

[0049] The resource utilization rate of each node under the key performance indicators is determined based on the individual resource utilization rate of each node under the key performance indicators.

[0050] Optionally, the processing module is specifically used for:

[0051] For any performance metric, sort the actual available resources of each node under the performance metric, and divide the sorting results into multiple intervals;

[0052] The weighted weight of the node under the performance index is determined based on the preset value of the interval where the node is located.

[0053] Optionally, the processing module is specifically used for:

[0054] For any given node, retrieve the scheduling weight of that node in the previous task scheduling from the task scheduling database;

[0055] The scheduling weight of the node is determined based on the scheduling weight in the previous task scheduling, the indicator weight of each performance indicator, and the weighted weight of each performance indicator.

[0056] Optionally, there may be multiple weight calculation rules;

[0057] When resource utilization exceeds a set threshold, the weights of each performance indicator are equal.

[0058] When the resource utilization rate is lower than the set threshold, the weight of key performance indicators is higher than the weight of non-key performance indicators.

[0059] Optionally, the processing module is further configured to:

[0060] After determining the node to execute the task to be executed, the scheduling weight of the node to execute the task to be executed is adjusted according to the reduction adjustment rule to obtain the scheduling weight of the node in the previous task scheduling.

[0061] Optionally, the processing module is further configured to:

[0062] After determining the node to execute the task, two node monitors are set up for the same node, serving as both primary and backup.

[0063] For any node monitor, after the node monitoring center determines that the node monitor is abnormal, it sends an overwrite file to the node monitor; the overwrite file is used to instruct the node monitor to restart; after determining that both the primary and backup node monitors have failed to restart, an alarm message is issued.

[0064] Thirdly, embodiments of the present invention also provide a computer device, comprising:

[0065] Memory, used to store program instructions;

[0066] The processor is used to call the program instructions stored in the memory and execute the above-mentioned distributed load balancing task scheduling method according to the obtained program.

[0067] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described distributed load balancing task scheduling method. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present invention;

[0070] Figure 2 A flowchart illustrating a distributed load balancing task scheduling method provided in an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of task generation provided in an embodiment of the present invention;

[0072] Figure 4 A schematic diagram illustrating the execution of a task according to an embodiment of the present invention;

[0073] Figure 5 This is a schematic diagram of a distributed load balancing task scheduling device provided in an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0075] In existing technologies, distributed task scheduling methods suffer from two main problems. Firstly, there is an imbalance issue. Secondly, distributed task scheduling methods based on cluster architecture perform load balancing on a server-by-server basis. However, cluster architecture requires the participation of multiple servers, making distributed systems complex and unable to achieve lightweight task scheduling.

[0076] Furthermore, regardless of the method used in the existing technology, it is impossible to flexibly select the node to execute the task based on the task type when performing load balancing across nodes. The node selection has poor flexibility and low accuracy.

[0077] Therefore, a task scheduling method is needed, based on a scheduling server, to achieve lightweight selection of nodes to execute tasks based on task type, thereby increasing the flexibility and accuracy of node selection.

[0078] Figure 1 An exemplary system architecture applicable to an embodiment of the present invention is shown, which includes a task scheduling database 110, a scheduling center 120, a node monitoring center 130, and a business processing database 160.

[0079] Figure 1 The invention exemplarily illustrates three distributed nodes: node a141, node b142, and node c143. It also illustrates the node monitors corresponding to the distributed nodes, including monitor 151 for node a, monitor 152 for node b, and monitor 153 for node c. It should be noted that the invention exemplifies three distributed nodes, and no specific quantity is limited here. Furthermore, no specific quantity is limited for any node. For example, there can be two or three monitors for node a141.

[0080] The task scheduling database 110 is used to obtain and store the performance data of each node's performance indicators periodically reported by the node monitoring center 130.

[0081] The scheduling center 120 is used to obtain the task type reported by a certain node, match the key performance indicators according to the task type, and then retrieve the performance data of each performance indicator of each node in the task scheduling database 110. Based on the performance data of each performance indicator of each node and the weight calculation rules, the node to execute the task is determined.

[0082] The node monitoring center 130 is used to store the performance data of each node's periodically reported performance indicators to the task scheduling database 110.

[0083] Any node monitor is used to periodically monitor the performance data of each performance indicator of the corresponding node and report it to the node monitoring center 130. The node monitoring center then stores the performance data of each performance indicator in the task scheduling database 110.

[0084] The business processing database 160 is used to specify any node to send the user-defined task type and task information to the scheduling center 120.

[0085] It should be noted that the above Figure 1 The structure shown is implemented on a single server, meaning this application is implemented on a single server to achieve lightweight load balancing of distributed nodes. Furthermore, the above... Figure 1 The structure shown is merely an example, and the embodiments of the present invention are not limited thereto.

[0086] Based on the above description Figure 2 An exemplary flowchart of a distributed load balancing task scheduling method provided by an embodiment of the present invention is shown. This process can be executed by a distributed load balancing task scheduling device.

[0087] like Figure 2 As shown, the process specifically includes:

[0088] Step 210: The scheduling center determines the key performance indicators that match the task type of the task to be executed from the various performance indicators of the node.

[0089] In this embodiment of the invention, the task type corresponds to one of the performance indicators. Therefore, the performance indicator corresponding to the task type can be matched from the performance indicators according to the task type, and the performance indicator corresponding to the task type is determined as the key performance indicator.

[0090] Step 220: The scheduling center determines the resource utilization rate of each node under the key performance indicators based on the performance data of the key performance indicators collected by the node monitors of each node.

[0091] In this embodiment of the invention, for the key performance indicators corresponding to the task type, the resource utilization rate is determined based on the performance data of the key performance indicators of all nodes, which is equivalent to determining the overall operating status of the server.

[0092] Step 230: The scheduling center determines a weight calculation rule that matches the resource utilization rate.

[0093] In this embodiment of the invention, the weight of key performance indicators in the weight calculation rule is positively correlated with resource utilization; the weight of non-key performance indicators in the weight calculation rule is negatively correlated with resource utilization.

[0094] Step 240: The scheduling center determines the weighted weight of each node under each performance indicator based on the performance data of each performance indicator collected by the node monitor of each node.

[0095] In this embodiment of the invention, for any performance index, each node has a corresponding weighted weight, which is used to calculate the scheduling weight of each node.

[0096] Step 250: The scheduling center determines the scheduling weight of any node based on the weight calculation rules and the weighted weight of the node under each performance index.

[0097] In this embodiment of the invention, for any node, the scheduling weight of the node is determined by different preset index weights of key performance indicators and other performance indicators in the weight calculation rules, based on the performance data of key performance indicators determined by the task type, the performance data of other performance indicators, and the corresponding preset index weights, so as to reflect the importance of key performance indicators.

[0098] Step 260: The scheduling center determines the node to execute the task to be executed based on the scheduling weight of each node.

[0099] In step 210, the task type may include CPU type, IO type and memory type. The task type refers to the importance of the resources consumed by the task to be executed. For example, if the user determines that the task type of a certain task to be executed is CPU type, it means that the task to be executed is more dependent on CPU resources. That is, the better the CPU resources, the higher the execution efficiency of the task to be executed.

[0100] The task types correspond to various performance metrics; therefore, each node includes three performance metrics: CPU performance metrics, I / O performance metrics, and memory performance metrics. It should be noted that this embodiment of the invention exemplifies three types and their corresponding performance metrics, but is not limited to three; it can also include four, five, etc., and can be configured according to specific application scenarios.

[0101] In this embodiment of the invention, the task type is determined by the user based on the task information, and then the task is generated through any node, which then reports the task type to the scheduling center.

[0102] For example, Figure 3 An exemplary schematic diagram of task generation is shown, such as... Figure 3 As shown, it includes:

[0103] Step 310: Trigger node to generate task.

[0104] The task scheduler in the Quartz job is used to trigger node s, which in turn generates a task, including the task type and task information.

[0105] Step 320: The task is reported to the dispatch center.

[0106] Node s will report the generated tasks to the scheduling center.

[0107] Step 330: Select the node to execute the task.

[0108] The dispatch center, based on the task type, uses the above... Figure 2 The technical solution shown determines the node c that will execute the task, and then sends the task information to node c, which will then execute the task.

[0109] In step 220, after determining the key performance indicators (KPIs) based on the task type, resource utilization is determined based on the performance data of the KPIs. For any performance indicator, two performance data points are included: maximum available resources and actual available resources.

[0110] For example, for CPU performance metrics, each node includes two performance data points: maximum CPU utilization and actual CPU availability. For IO performance metrics, each node includes two performance data points: maximum IO utilization and actual IO availability. For memory performance metrics, each node includes two performance data points: maximum used memory and actual available memory.

[0111] Specifically, for any given node, the scheduling center determines the individual resource utilization rate of the node under the key performance indicators based on the node's maximum available resources and actual available resources under the key performance indicators; and determines the resource utilization rate of each node under the key performance indicators based on the individual resource utilization rate of each node under the key performance indicators.

[0112] In one feasible approach, the individual resource utilization rate of each node is determined, and then the average of the individual resource utilization rates is calculated, which is used as the resource utilization rate of each node.

[0113] Taking CPU performance as a key performance indicator as an example, for node a1, the actual CPU availability of node a1 is compared with the maximum CPU availability. The ratio is used as the individual resource utilization of node a1. If there are 9 nodes, the average of the individual resource utilization of the 9 nodes is calculated and used as the resource utilization of the 9 nodes.

[0114] Taking IO performance as a key performance indicator as an example, for node a2, the actual IO availability rate of node a2 is compared with the maximum IO availability rate. The ratio is used as the individual resource utilization rate of node a2. If there are 7 nodes, the individual resource utilization rate of the 7 nodes is averaged and the average value is used as the resource utilization rate of the 7 nodes.

[0115] Taking memory performance as a key performance indicator as an example, for node a3, the actual available memory of node a3 is compared with the maximum available memory to obtain the ratio. The ratio is used as the individual resource utilization rate of node a3. If there are 5 nodes, the individual resource utilization rate of the 5 nodes is averaged and the average value is used as the resource utilization rate of the 5 nodes.

[0116] In another feasible approach, the actual available resources of the key performance indicators of each node are summed to obtain a first value, and the maximum available resources of the key performance indicators of each node are summed to obtain a second value. The ratio of the first value to the second value is used as the resource utilization rate of each node.

[0117] Taking CPU performance as a key performance indicator as an example, we calculate the sum of the actual CPU availability of nodes a1, a2, ..., a9, and the sum of the maximum CPU utilization of nodes a1, a2, ..., a9. We then compare the sum of the actual CPU availability with the sum of the maximum CPU utilization, and the quotient is taken as the resource utilization rate of each node under the CPU performance indicator. Resource utilization rate can also be understood as resource idle rate.

[0118] In step 230, there are multiple weight calculation rules; when the resource utilization rate is higher than the set threshold, the weights of each performance indicator are the same; when the resource utilization rate is lower than the set threshold, the weights of key performance indicators are higher than the weights of non-key performance indicators.

[0119] In this embodiment of the invention, when the resource utilization rate is higher than a set threshold, it proves that the memory resources of each node are sufficient, so there is no need to favor the selection of nodes based on memory performance indicators. When the resource utilization rate is lower than the set threshold, it is necessary to favor the selection of nodes based on memory performance indicators. Therefore, the indicator weight of key performance indicators needs to be higher than the indicator weight of non-key performance indicators.

[0120] Specifically, in the weighting calculation rules, the weights of key performance indicators are positively correlated with resource utilization, while the weights of non-key performance indicators are negatively correlated with resource utilization. In other words, the higher the resource utilization, the higher the weight of the key performance indicator; conversely, the lower the resource utilization, the lower the weight of the non-key performance indicator.

[0121] Taking memory performance as a key performance indicator as an example, the resource utilization rate s of each node in memory performance is determined according to the above technical solution. Then, the weight calculation rules are matched according to the resource utilization rate s, as described below:

[0122] When determining the resource utilization rate s > 0.5 (set threshold), the weight calculation rule described in the following formula (1) is matched;

[0123] D = p + w m +w CPU +w IO (1);

[0124] When the resource utilization rate is determined to be 0.4 < s < 0.5, the weight calculation rule described in the following formula (2) shall be applied.

[0125] D = p + 1.2w m +0.9w CPU +0.9w IO (2);

[0126] When the resource utilization rate is determined to be 0.3 < s < 0.4, the weight calculation rule described in the following formula (3) shall be applied.

[0127] D = p + 1.5w m +0.75w CPU +0.75w IO (3);

[0128] When the resource utilization rate is determined to be 0.2 < s < 0.3, the weight calculation rule described in the following formula (4) shall be applied.

[0129] D = p + 1.8w m +0.6w CPU +0.6w IO (4);

[0130] When the resource utilization rate s < 0.2 is determined, the weight calculation rule described in the following formula (5) shall be matched;

[0131] D = p + 2.2w m +0.4w CPU +0.4w IO (5);

[0132] Among them, w m For the weighted average of memory performance metrics, w CPU w is the weighted average of the node's CPU performance metrics. IO The weighted average of node I / O performance metrics, and w m The parameter being multiplied is the weight of the key performance indicator, and is related to w. CPU and w IO The parameters multiplied are the weights of non-critical performance indicators.

[0133] It should be noted that w CPU and w IO Different weights can be set for non-critical performance metrics, for example, with w CPU The parameter for multiplication is 0.4, and w IO The multiplication parameter is 0.3. If the performance index increases, the index weight can also increase, but no specific limit is made here.

[0134] In this embodiment of the invention, for any node, the weight calculation rule of the node is matched according to the individual resource utilization rate of the node. This means that each node needs to be matched with the weight calculation rule once, rather than matching the weight calculation rule uniformly for all nodes. This will increase the computational load on the scheduling center, but it will also increase the accuracy of node selection. It is suitable for use when there are few nodes.

[0135] In step 240, for any performance metric, the weight of each node is determined according to the ranking, that is, for any performance metric, each node is assigned a weight.

[0136] Specifically, for any performance metric, the scheduling center sorts the actual available resources of each node under the performance metric and divides the sorting results into multiple intervals; based on the preset value of the interval where the node is located, the weighted weight of the node under the performance metric is determined.

[0137] Taking the three performance metrics given above as examples, for the performance data under any performance metric, each node is sorted according to the data and divided into multiple intervals. The weighting weight of the node is determined according to the preset value of the interval where the node is located.

[0138] In one feasible approach, nodes are sorted according to their maximum available resources and divided into multiple intervals. For any given interval, the preset value is used as the weighted weight of the nodes within that interval, which is equivalent to assigning a weighted weight to the nodes within each interval.

[0139] In this embodiment of the invention, nodes are sorted according to their actual available resources and divided into multiple intervals to determine the weighted weight of each node. The sorting rule can be from largest to smallest or from smallest to largest, and is not specifically limited here.

[0140] For example, the distributed nodes include a1, a2, ..., a9. Based on the CPU performance metrics, the nodes are sorted from largest to smallest according to the actual CPU availability, resulting in a1, a2, ..., a9. Then, the nodes are divided into intervals.

[0141] In one feasible approach, the distributed nodes are divided into a predetermined number of intervals based on their number. In another feasible approach, all distributed nodes are divided into intervals based on the predetermined number of distributed nodes.

[0142] In this embodiment of the invention, in conjunction with the above further examples, the distributed nodes a1, a2, ..., a9 are divided into three intervals: a1, a2, a3 are the first interval, a4, a5, a6 are the second interval, and a7, a8, a9 are the third interval.

[0143] The preset value for the first interval is 3, the preset value for the second interval is 2, and the preset value for the third interval is 1. Therefore, for CPU performance metrics, the weighted average weight for nodes a1, a2, and a3 is 3, the weighted average weight for nodes a4, a5, and a6 is 2, and the weighted average weight for nodes a7, a8, and a9 is 1.

[0144] Taking IO performance metrics as an example, the nodes are sorted from largest to smallest based on their actual IO availability, resulting in a9, a8, ..., a1. These nodes are then divided into intervals. Based on preset values ​​for each interval, the weighting weights for nodes a1, a2, and a3 are determined to be 1; for nodes a4, a5, and a6, the weighting weight is 2; and for nodes a7, a8, and a9, the weighting weight is 3. The specific method for determining these weighting weights will not be elaborated upon again.

[0145] Taking memory performance metrics as an example, the nodes are sorted from largest to smallest based on actual IO availability, resulting in the order a4, a5, a6, a3, a1, a2, a8, a7, a9. These nodes are then divided into intervals. Based on preset values ​​for each interval, the weighting weights for nodes a1, a2, and a3 are determined to be 2; for nodes a4, a5, and a6, the weighting weight is 3; and for nodes a7, a8, and a9, the weighting weight is 1. The specific method for determining these weighting weights will not be elaborated upon again.

[0146] In step 250, the scheduling weight of each node is determined based on the weighted weight of each node and the matching weight calculation rules.

[0147] Specifically, for any given node, the scheduling center retrieves the node's scheduling weight in the previous task scheduling from the task scheduling database. Based on the scheduling weight in the previous task scheduling, the weights of each performance indicator, and the weighted average of each performance indicator, the scheduling weight of the node is determined.

[0148] As shown in formulas (1) to (5) above, where D is the scheduling weight of the node and p is the scheduling weight of the node in the previous task scheduling. Then, the scheduling weight of each node can be determined according to the corresponding formula.

[0149] It should be noted that if a node is a new node, meaning it does not have a scheduling weight in the previous task scheduling, then its preset initial scheduling weight is used as the scheduling weight in the previous task scheduling, that is, the initial scheduling weight (such as 10, 15, etc.) is used as p. By assigning an initial scheduling weight, the scheduling weight is determined, giving the new node the possibility of being selected to execute the task to be executed, thus improving the convenience of adding new nodes in this invention.

[0150] In step 260, after obtaining the scheduling weight of each node, the node to be executed is determined according to the scheduling weight of the node. For example, the node with the largest scheduling weight is selected as the node to execute the task.

[0151] To prevent the node scheduling weight from increasing indefinitely, the scheduling weight of the node to be executed is reduced after the node is determined to perform the task.

[0152] Specifically, after determining the node to execute the task to be executed, the scheduling center adjusts the scheduling weight of the node to execute the task to be executed according to the reduction adjustment rules, so as to obtain the scheduling weight of the node to execute the task in the previous task scheduling.

[0153] In one feasible approach, the scheduling weight is reduced by a preset value. Specifically, the difference between the scheduling weight and the preset value is determined, and the difference is updated as the scheduling weight of the node. For example, if the scheduling weight of node a before it is determined to be the node to execute the task is 20, and the preset value is 2, then the difference is 18. After node a is determined to be the node to execute the task, the scheduling weight of node a is 18.

[0154] In another feasible approach, the scheduling weight is reduced by a preset ratio. Specifically, the product of the scheduling weight and the preset ratio is determined, and the product is updated as the scheduling weight of the node. For example, if the scheduling weight of node a before it is determined to be the node to execute the task is 20 and the preset ratio is 80%, then the product is 16. After node a is determined to be the node to execute the task, the scheduling weight of node a is 16.

[0155] In this embodiment of the invention, for any node, the performance data of each performance indicator of the node is obtained periodically through periodic monitoring by multiple node monitors.

[0156] Specifically, the same node is configured with two node monitors that act as both primary and backup for each other;

[0157] For any node monitor, after the node monitoring center determines that the node monitor is abnormal, it sends an overwrite file to the node monitor; the overwrite file is used to instruct the node monitor to restart; after determining that both the primary and backup node monitors have failed to restart, an alarm message is issued.

[0158] In combination with the above Figure 1 As described in the example, the node a monitor 151 includes two node monitors that serve as primary and backup to each other. These monitors are used to take turns periodically monitoring the performance data of node a's memory performance metrics, CPU performance metrics, and I / O performance metrics, as well as its scheduling weight, and then sending the data to the node monitoring center.

[0159] In one scenario, the node monitor shuts down normally. Before node a shuts down normally, it sends a shutdown command to the scheduling center. Upon receiving the shutdown command, the scheduling center changes the status of node a in the task scheduling database to terminated and notifies the node monitoring center to remove node a from the database. Ultimately, the node monitoring center shuts down both node monitors for node a.

[0160] In another scenario, a node monitor shuts down abnormally. If the central monitor periodically scans the reports from each node monitor, and if a monitor on node a fails to report performance data for multiple (e.g., two) periods, an overwrite file is generated and sent to that node monitor. If the node monitor fails to restart, an alarm message is issued, indicating an anomaly and that the node is operating in single-monitor mode.

[0161] If it is determined that all node monitors for node a are abnormal, then change the status of node a to terminated and remove node a from the service.

[0162] In this embodiment of the invention, for a new node, the scheduling center needs to issue an instruction to update the node's status to start or wait to execute a task, i.e. register the new node. After the status update is completed, the node is sent to the node monitoring center, which then configures multiple node monitors for the node to obtain the performance data of various indicators of the node in real time.

[0163] For example, a node will also update its own state after receiving a task. Figure 4 An exemplary schematic diagram of a task execution is shown, such as... Figure 4 As shown, it includes:

[0164] Step 410: The node receives the task.

[0165] After the scheduling center determines the node b to execute the task, it sends the task to node b so that node b can receive the task.

[0166] Step 420: The scheduling center updates the node status.

[0167] After node b receives the task, the scheduling center updates node b's status to "processing".

[0168] Step 430: The node notifies the scheduling center of the task status.

[0169] Node b executes the task and sends completion information to the scheduling center after the task is completed.

[0170] Step 440: The scheduling center determines the node status and updates the node status.

[0171] The scheduling center determines whether node b has other tasks to execute. If so, the state of node b is not changed. If not, the state of node b is updated to the ready state, which means that the task can be executed at any time.

[0172] Based on the same technological concept Figure 3 An exemplary schematic diagram of a distributed load balancing task scheduling device provided in an embodiment of the present invention is shown. This device can execute the above-described distributed load balancing task scheduling method.

[0173] like Figure 5 As shown, the device specifically includes:

[0174] The matching module 510 is used to determine the key performance indicators that match the task type of the task to be executed from the various performance indicators of the node.

[0175] Processing module 520 is used to determine the resource utilization rate of each node under the key performance indicators based on the performance data of the key performance indicators collected by the node monitor of each node.

[0176] Determine a weighting calculation rule that matches the resource utilization rate; wherein, in the weighting calculation rule, the indicator weights of the key performance indicators are positively correlated with the resource utilization rate; and the indicator weights of the non-key performance indicators are negatively correlated with the resource utilization rate.

[0177] Based on the performance data of each node collected by the node monitor of each node, the weighting weight of each node under each performance indicator is determined.

[0178] For any given node, the scheduling weight of the node is determined based on the weight calculation rules and the weighted weight of the node under each performance metric.

[0179] Based on the scheduling weights of each node, the node to execute the task to be executed is determined.

[0180] Optionally, the processing module 520 is specifically used for:

[0181] For any node, determine the individual resource utilization rate of the node under the key performance indicators based on the node's maximum available resources and actual available resources under the key performance indicators.

[0182] The resource utilization rate of each node under the key performance indicators is determined based on the individual resource utilization rate of each node under the key performance indicators.

[0183] Optionally, the processing module 520 is specifically used for:

[0184] For any performance metric, sort the actual available resources of each node under the performance metric, and divide the sorting results into multiple intervals;

[0185] The weighted weight of the node under the performance index is determined based on the preset value of the interval where the node is located.

[0186] Optionally, the processing module 520 is specifically used for:

[0187] For any given node, retrieve the scheduling weight of that node in the previous task scheduling from the task scheduling database;

[0188] The scheduling weight of the node is determined based on the scheduling weight in the previous task scheduling, the indicator weight of each performance indicator, and the weighted weight of each performance indicator.

[0189] Optionally, there may be multiple weight calculation rules;

[0190] When resource utilization exceeds a set threshold, the weights of each performance indicator are equal.

[0191] When the resource utilization rate is lower than the set threshold, the weight of key performance indicators is higher than the weight of non-key performance indicators.

[0192] Optionally, the processing module 520 is further configured to:

[0193] After determining the node to execute the task to be executed, the scheduling weight of the node to execute the task to be executed is adjusted according to the reduction adjustment rule to obtain the scheduling weight of the node in the previous task scheduling.

[0194] Optionally, the processing module 520 is further configured to:

[0195] After determining the node to execute the task, two node monitors are set up for the same node, serving as both primary and backup.

[0196] For any node monitor, after the node monitoring center determines that the node monitor is abnormal, it sends an overwrite file to the node monitor; the overwrite file is used to instruct the node monitor to restart; after determining that both the primary and backup node monitors have failed to restart, an alarm message is issued.

[0197] Based on the same technical concept, embodiments of the present invention also provide a computer device, including:

[0198] Memory, used to store program instructions;

[0199] The processor is used to call the program instructions stored in the memory and execute the above-mentioned distributed load balancing task scheduling method according to the obtained program.

[0200] Based on the same technical concept, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which are used to cause a computer to execute the above-described distributed load balancing task scheduling method.

[0201] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0203] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0204] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0205] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A task scheduling method for distributed load balancing, characterized in that, include: The scheduling center determines the key performance indicators that match the task type of the task to be executed from the various performance indicators of the nodes. The scheduling center determines the resource utilization rate of each node under the key performance indicators based on the performance data of the key performance indicators collected by the node monitors of each node. The scheduling center determines a weight calculation rule that matches the resource utilization rate; wherein, in the weight calculation rule, the indicator weights of the key performance indicators are positively correlated with the resource utilization rate; the indicator weights of the non-key performance indicators are negatively correlated with the resource utilization rate; there are multiple weight calculation rules; when the resource utilization rate is higher than a set threshold, the indicator weights of each performance indicator are the same; when the resource utilization rate is lower than the set threshold, the indicator weights of the key performance indicators are higher than the indicator weights of the non-key performance indicators. The scheduling center determines the weighting of each node under each performance indicator based on the performance data of each performance indicator collected by the node monitor of each node. The scheduling center determines the scheduling weight of any node based on the weight calculation rules and the weighted weight of the node under each performance index. The scheduling center determines the node to execute the task to be executed based on the scheduling weight of each node.

2. The method as described in claim 1, characterized in that, The scheduling center determines the resource utilization rate of each node under the key performance indicators based on the performance data of the key performance indicators collected by the node monitors of each node, including: For any given node, the scheduling center determines the individual resource utilization rate of the node under the key performance indicators based on the node's maximum available resources and actual available resources under the key performance indicators. The scheduling center determines the resource utilization rate of each node under the key performance indicators based on the individual resource utilization rate of each node under the key performance indicators.

3. The method as described in claim 1, characterized in that, The scheduling center determines the weighted weight of each node for each performance indicator based on the performance data of each node's node monitor, including: For any performance metric, the scheduling center sorts the actual available resources of each node under the performance metric and divides the sorting results into multiple intervals. The weighted weight of the node under the performance index is determined based on the preset value of the interval where the node is located.

4. The method as described in claim 1, characterized in that, The scheduling center determines the scheduling weight of any node based on the weight calculation rules and the weighted weights of the node under various performance indicators, including: For any given node, the scheduling center retrieves the scheduling weight of that node in the previous task scheduling from the task scheduling database. The scheduling weight of the node is determined based on the scheduling weight in the previous task scheduling, the indicator weight of each performance indicator, and the weighted weight of each performance indicator.

5. The method as described in claim 1, characterized in that, After determining the node to execute the task to be executed, the process also includes: The scheduling center adjusts the scheduling weight of the node executing the task to be executed according to the reduction adjustment rules, and obtains the scheduling weight of the node executing the task in the previous task scheduling.

6. The method as described in claim 1, characterized in that, After determining the node to execute the task to be executed, the process also includes: Two node monitors are set up on the same node, one as the primary and one as the backup. For any node monitor, after the node monitoring center determines that the node monitor is abnormal, it sends an overwrite file to the node monitor; the overwrite file is used to instruct the node monitor to restart; after determining that both the primary and backup node monitors have failed to restart, an alarm message is issued.

7. A distributed load balancing task scheduling device, characterized in that, include: The matching module is used to determine the key performance indicators that match the task type of the task to be executed from the various performance indicators of the node. The processing module is used to determine the resource utilization rate of each node under the key performance indicators based on the performance data of the key performance indicators collected by the node monitor of each node. A weighting calculation rule matching the resource utilization rate is determined; wherein, the weights of the key performance indicators in the weighting calculation rule are positively correlated with the resource utilization rate; the weights of the non-key performance indicators in the weighting calculation rule are negatively correlated with the resource utilization rate; there are multiple weighting calculation rules; when the resource utilization rate is higher than a set threshold, the weights of each performance indicator are the same; when the resource utilization rate is lower than the set threshold, the weights of the key performance indicators are higher than the weights of the non-key performance indicators. Based on the performance data of each node collected by the node monitor, the weighting weight of each node under each performance indicator is determined. For any given node, the scheduling weight of the node is determined based on the weight calculation rules and the weighted weight of the node under each performance metric. Based on the scheduling weights of each node, the node to execute the task to be executed is determined.

8. A computer device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method according to any one of claims 1 to 6.

Images