Cluster load balancing method and device, electronic equipment and storage medium

Abstract

The invention discloses a cluster load balancing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining initial load index data of a node in a cluster, carrying out the load information updating processing of the initial load index data according to a load reporting threshold value, and obtaining target load index data, and performing node energy consumption sensing processing on the target load index data according to the energy consumption model to obtain node energy consumption information, and allocating nodes from the cluster according to the node energy consumption information to perform task scheduling. The embodiment of the invention can effectively reduce the energy consumption of the cluster and balance the load of each node of the cluster, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a cluster load balancing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of computer technology and its widespread application, the user scale and traffic are growing, and the concurrent access volume of Web services is also increasing. A single server can no longer meet normal needs. Cluster deployment solutions are widely used due to their high reliability and scalability. In related technologies, the load balancing mechanism of Web clusters is mainly based on user experience, but rarely considers the energy consumption of clusters, resulting in high energy consumption of clusters and affecting the load balancing of clusters.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a cluster load balancing method, device, electronic device and storage medium, which can reduce cluster consumption and improve scheduling efficiency.

[0005] To achieve the above object, an embodiment of the present application provides a cluster load balancing method, the method comprising:

[0006] Get the initial load indicator data of the nodes in the cluster;

[0007] Performing load information updating processing on the initial load indicator data according to the load reporting threshold to obtain target load indicator data;

[0008] Perform node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information;

[0009] Nodes are allocated from the cluster for task scheduling according to the node energy consumption information.

[0010] In some embodiments, performing load information update processing on the initial load indicator data according to the load reporting threshold to obtain target load indicator data includes the following steps:

[0011] The load information monitoring module periodically collects and processes the node load information to obtain the collected period load information;

[0012] Performing load capacity change calculation processing on the collected periodic load information to obtain a node load capacity change rate;

[0013] When the change rate of the node load capacity is greater than the load reporting threshold, the collection period load information is reported and processed to update the target load indicator data.

[0014] In some embodiments, the step of performing load capacity change calculation processing on the collected periodic load information to obtain a node load capacity change rate comprises the following steps:

[0015] Obtaining the load capacity at the start time and the load capacity at the end time from the acquisition period load information;

[0016] The load capacity at the end time and the load capacity at the start time are subtracted from each other, and the calculated difference is divided by the cycle time value and the absolute value is calculated to obtain the node load capacity change rate.

[0017] In some embodiments, performing node energy consumption perception processing on the target load indicator data according to the energy consumption model to obtain node energy consumption information includes the following steps:

[0018] Perform node energy consumption calculation processing on the target load index data according to the energy consumption model to obtain a node energy consumption value;

[0019] The node energy consumption values ​​are sorted in ascending order by an asynchronously executed sub-thread and stored in an ordered set to obtain the node energy consumption information.

[0020] In some embodiments, before performing node energy consumption perception processing on the target load indicator data according to the energy consumption model to obtain node energy consumption information, the method further includes constructing the energy consumption model, which specifically includes the following steps:

[0021] Build an experimental cluster environment;

[0022] Collecting and processing node power consumption information of the experimental cluster environment to obtain power consumption sampling data;

[0023] The power consumption sampling data is modeled according to a polynomial regression model to obtain the energy consumption model.

[0024] In some embodiments, allocating nodes from the cluster for task scheduling according to the node energy consumption information includes the following steps:

[0025] Calculate the average energy consumption of cluster nodes according to the node energy consumption information;

[0026] Perform node screening on the cluster according to the average energy consumption of the cluster nodes to obtain an initial node set;

[0027] Performing traversal and sorting processing on the initial node set to obtain the allocation node with the largest node remaining load rate;

[0028] Performing task scheduling on the allocation node.

[0029] In some embodiments, the traversal and sorting of the initial node set to obtain the allocation node with the largest node remaining load rate includes the following steps:

[0030] The initial node set is screened according to the running status of the node service instance to obtain a target node set;

[0031] Calculating the node residual load rate of the target node set to obtain the residual load rate;

[0032] The target node set is sorted according to the remaining load rate to obtain an allocation node with the largest remaining load rate.

[0033] To achieve the above object, another aspect of an embodiment of the present application provides a cluster load balancing device, the device comprising:

[0034] The first module is used to obtain the initial load indicator data of the nodes in the cluster;

[0035] The second module is used to perform load information update processing on the initial load indicator data according to the load reporting threshold to obtain target load indicator data;

[0036] The third module is used to perform node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information;

[0037] The fourth module is used to allocate nodes from the cluster for task scheduling according to the node energy consumption information.

[0038] In some embodiments, the second module is used to perform load information update processing on the initial load indicator data according to the load reporting threshold to obtain target load indicator data, including:

[0039] The first unit is used to periodically collect and process the node load information through the load information monitoring module to obtain the collected period load information;

[0040] The second unit is used to calculate and process the load capacity change of the collected period load information to obtain a node load capacity change rate;

[0041] The third unit is used to report the collection period load information when the node load capacity change rate is greater than the load reporting threshold, and update the target load indicator data.

[0042] In some embodiments, the second unit is used to calculate the load capacity change of the collection period load information to obtain the node load capacity change rate, including:

[0043] A first subunit is used to obtain the load capacity at the start time and the load capacity at the end time from the acquisition period load information;

[0044] The second subunit is used to subtract the load capacity at the end time from the load capacity at the start time, and divide the calculated difference with the cycle time value and perform absolute value calculation to obtain the node load capacity change rate.

[0045] In some embodiments, the third module is used to perform node energy consumption perception processing on the target load indicator data according to the energy consumption model to obtain node energy consumption information, including:

[0046] A fourth unit is used to perform node energy consumption calculation processing on the target load index data according to the energy consumption model to obtain a node energy consumption value;

[0047] The fifth unit is used to sort the node energy consumption values ​​in ascending order through asynchronously executed sub-threads and store them into an ordered set to obtain the node energy consumption information.

[0048] In some embodiments, before the third module is used to perform node energy consumption perception processing on the target load index data according to the energy consumption model to obtain the node energy consumption information, the device also includes a fifth module for constructing the energy consumption model, which specifically includes:

[0049] The third subunit is used to build the experimental cluster environment;

[0050] A fourth subunit is used to collect and process node power consumption information of the experimental cluster environment to obtain power consumption sampling data;

[0051] The fifth subunit is used to perform modeling processing on the power consumption sampling data according to a polynomial regression model to obtain the energy consumption model.

[0052] In some embodiments, the fourth module, used to allocate nodes from the cluster for task scheduling according to the node energy consumption information, includes:

[0053] A sixth unit is used to calculate the average energy consumption of the cluster nodes according to the node energy consumption information;

[0054] A seventh unit is used for performing node screening processing on the cluster according to the average energy consumption of the cluster nodes to obtain an initial node set;

[0055] An eighth unit is used to perform traversal and sorting processing on the initial node set to obtain an allocation node with the largest node residual load rate;

[0056] The ninth unit is used to schedule tasks for the allocation node.

[0057] In some embodiments, the eighth unit is used to perform traversal and sorting processing on the initial node set to obtain the allocation node with the largest node residual load rate, including:

[0058] A sixth subunit is used to filter the initial node set according to the running status of the node service instance to obtain a target node set;

[0059] A seventh subunit is used to perform node residual load rate calculation processing on the target node set to obtain a residual load rate;

[0060] An eighth subunit is used to sort the target node set according to the residual load rate to obtain an allocation node with the largest node residual load rate.

[0061] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0062] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0063] The embodiments of the present application include at least the following beneficial effects: The present application provides a cluster load balancing method, device, electronic device and storage medium. The scheme obtains the initial load index data of the nodes in the cluster, updates the load information of the initial load index data according to the load reporting threshold, and obtains the target load index data. It can monitor the load status of each node in real time and update the load status based on the load reporting threshold, thereby saving cluster network resource overhead; in addition, the scheme performs node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information, allocates nodes from the cluster for task scheduling according to the node energy consumption information, and can reasonably schedule tasks according to energy consumption and load conditions, while effectively reducing the cluster energy consumption. Balance the load of each node in the cluster, improve scheduling efficiency, reduce cluster operating costs, and realize green and energy-saving cluster management. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flow chart of a cluster load balancing method provided in an embodiment of the present application;

[0065] Figure 2 is a schematic diagram of an energy consumption model application provided in an embodiment of the present application;

[0066] Figure 3 yes Figure 1Flow chart of step S102 in FIG.

[0067] Figure 4 yes Figure 1 Flow chart of step S103 in FIG.

[0068] Figure 5 It is a topological diagram of an experimental cluster environment provided in an embodiment of the present application;

[0069] Figure 6 It is a fitting diagram of a polynomial regression model provided in an embodiment of the present application;

[0070] Figure 7 yes Figure 1 Flow chart of step S104 in FIG.

[0071] Figure 8 It is a flow chart of an energy consumption-aware scheduling algorithm provided in an embodiment of the present application;

[0072] Fig. 9 It is a structural diagram of a cluster load balancing device provided in an embodiment of the present application;

[0073] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0075] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0076] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0078] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0079] 1) Web cluster is a server cluster composed of multiple Web servers, which is used to process a large number of Web requests to improve the performance and reliability of the system.

[0080] 2) Load balancing is the process of distributing requests among multiple servers to ensure that the load on each server is as even as possible, thereby optimizing resource usage and improving system reliability and performance.

[0081] 3) Energy consumption perception, which is used to monitor and analyze system energy consumption in real time, and optimize scheduling based on energy consumption data to achieve energy saving.

[0082] With the rapid development of computer technology and its widespread application, the user scale and traffic are growing, and the concurrent access volume of Web services is also increasing. A single server can no longer meet normal needs. Cluster deployment solutions are widely used due to their high reliability and scalability. As an important infrastructure for resource storage and data computing, server clusters bear a large energy consumption pressure while responding to a large number of task requests. Cluster energy consumption issues are becoming an increasingly important challenge.

[0083] In related technologies, the load balancing mechanism of Web clusters is mainly based on user experience, but rarely considers the energy consumption of the cluster, resulting in high energy consumption of the cluster, which affects the load balancing of the cluster. Therefore, task scheduling must ensure that the cluster reduces energy consumption while ensuring user experience and load balancing of the cluster. These requirements are very challenging for scheduling algorithms.

[0084] In view of this, a cluster load balancing method, device, electronic device and storage medium are provided in the embodiments of the present application. The scheme obtains the initial load index data of the nodes in the cluster, updates the load information of the initial load index data according to the load reporting threshold, and obtains the target load index data. It can monitor the load status of each node in real time, and update the load status based on the load reporting threshold, thereby saving the cluster network resource overhead; in addition, the scheme performs node energy consumption perception processing on the target load index data according to the energy consumption model to obtain the node energy consumption information, allocates nodes from the cluster for task scheduling according to the node energy consumption information, and can reasonably schedule tasks according to energy consumption and load status, while effectively reducing the cluster energy consumption. The load of each node in the cluster is balanced, the scheduling efficiency is improved, the cluster operating cost is reduced, and green and energy-saving cluster management is realized.

[0085] A cluster load balancing method provided in an embodiment of the present application relates to the field of computer technology. By real-time monitoring and analyzing the energy consumption of server nodes, combined with workload data, the task scheduling strategy of the server is dynamically adjusted to reduce the energy consumption of the computing cluster, while maintaining the load balance and efficient operation of the system. A cluster load balancing method provided in an embodiment of the present application can be applied to a server cluster. In some embodiments, the server cluster can be configured as a server cluster or a distributed system composed of multiple physical servers. The servers in the server cluster can be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers that provide basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network, but is not limited to the above forms.

[0086] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0087] Figure 1is an optional flow chart of a cluster load balancing method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0088] Step S101, obtaining initial load indicator data of nodes in the cluster;

[0089] Step S102, performing load information update processing on the initial load indicator data according to the load reporting threshold value to obtain target load indicator data;

[0090] Step S103, performing node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information;

[0091] Step S104: Allocate nodes from the cluster for task scheduling according to the node energy consumption information.

[0092] In steps S101 to S104 shown in the embodiment of the present application, the initial load index data of the nodes in the cluster are obtained through the load collector, and the load data of each node in the cluster is periodically collected through the load collector, and the initial load index data is updated with the load information through the preset load reporting threshold to obtain the target load index data. The target load index data is the load index data of each node in the cluster that needs to be calculated and processed within the task scheduling cycle, and is used to input the energy consumption model to calculate the energy consumption value of each node. In order to ensure the energy saving and consumption reduction of the algorithm and the load balancing of each node, the embodiment of the present application improves the update of the load information through the load capacity change rate and the load reporting threshold when monitoring the load information of each node in the cluster, so as to save the cluster network resource overhead. Then, according to the pre-constructed energy consumption model, the target load index data is processed by node energy consumption perception to obtain the node energy consumption information, so as to allocate nodes from the cluster for task scheduling according to the node energy consumption information. By calculating the energy consumption information of the nodes, the embodiment of the present application can preferentially allocate tasks to nodes with low energy consumption values ​​and high remaining load rates for processing, so as to balance the load of each node and the cluster energy consumption. Please refer to Figure 2 , Figure 2 This is a schematic diagram of an energy consumption model application. The load information of each node in the server cluster is collected by a load collector and sent to the energy consumption model. The energy consumption data of each node is calculated according to the energy consumption model and uploaded to the scheduler. The scheduler combines the load data and the energy consumption data to perform task scheduling. By introducing the energy consumption model into the task scheduling algorithm, the embodiment of the present application can integrate the resources of each node in the cluster and calculate the energy consumption through the load balancer, so as to schedule the nodes according to the node energy consumption to balance the load of each node and the cluster energy consumption.

[0093] In step S101 of some embodiments, the initial load indicator data of the nodes in the cluster can be obtained through the load collector. The initial load indicator data can be the load indicator data of each node when the server cluster starts running, or it can be the load indicator data of each node collected by the load collector from a specified time. The load indicator data includes indicator values ​​such as the central processing unit (CPU) utilization and memory utilization of each node in the cluster, but is not limited to this.

[0094] See also Figure 3 In step S102 of some embodiments, performing load information update processing on the initial load indicator data according to the load reporting threshold to obtain target load indicator data includes the following steps:

[0095] Step S301, periodically collect and process node load information through a load information monitoring module to obtain collection period load information;

[0096] Step S302, performing load capacity change calculation processing on the collected periodic load information to obtain a node load capacity change rate;

[0097] Step S303: When the change rate of the node load capacity is greater than the load reporting threshold, the collection period load information is reported and the target load index data is updated.

[0098] In an embodiment of the present application, the load information of each node in the cluster is periodically collected and processed by the load information monitoring module to obtain the collection period load information, which includes the load information of the node at each moment in the collection period. The collection period can be set to ten minutes, and can also be adjusted according to actual conditions, but is not limited to this. Then the load capacity change calculation process is performed on the collection period load information to obtain the node load capacity change rate, and the node load capacity change rate is used to characterize the change of the load index data of the server node in the collection period. When the node load capacity change rate calculated by the server node is greater than the preset load reporting threshold, the collection period load information is reported and processed, so as to update the initial load index data to obtain the target load index data. If the load capacity change rate calculated by the server node does not exceed the preset load reporting threshold, the server node does not need to report the load information in this collection period, and only needs to update the local load information for the calculation of the next load capacity change rate. The embodiment of the present application monitors the load information by defining the load reporting threshold, and updates the load information of the node with a large load capacity change, which can reduce the consumption of network resources.

[0099] In step S302 of some embodiments, performing load capacity change calculation processing on the collected periodic load information to obtain the node load capacity change rate includes the following steps:

[0100] Obtaining the load capacity at the start time and the load capacity at the end time from the acquisition period load information;

[0101] The load capacity at the end time and the load capacity at the start time are subtracted from each other, and the calculated difference is divided by the cycle time value and the absolute value is calculated to obtain the node load capacity change rate.

[0102] In an embodiment of the present application, when monitoring the load information of each node in the cluster, a collection period is set to collect the load information of each node in the cluster, and the load status of some nodes of the back-end server may not change significantly during the collection period. Therefore, in order to save the network resources of the cluster, the node load capacity change rate of each node in the cluster is calculated to determine whether it is necessary to report the load information of the node at that moment to the load balancer, which is used to integrate the resources of each node in the cluster and calculate the energy consumption. If the load information processing module of the load balancer does not receive the node load information within the collection period, the previous load information continues to be used, so as to reduce the network resource consumption of the entire cluster due to frequent load information collection. Specifically, the load capacity at the start time and the load capacity at the end time are obtained from the collection period load information, the load capacity at the end time and the load capacity at the start time are subtracted, and the calculated difference is divided by the cycle time value and the absolute value is calculated to obtain the node load capacity change rate. The calculation formula for the node load capacity change rate is as follows:

[0103]

[0104] Where ΔCL represents the change rate of node load capacity, represents the load capacity of node i at time t1, that is, the load capacity at the starting time, represents the load capacity of node i at time t2, that is, the load capacity at the end time, t1 represents the start time of the collection period, and t2 represents the end time of the collection period. The embodiment of the present application calculates the node load capacity change rate of each node during the collection period, and can select and report the load information according to the calculated node load capacity change rate to save cluster network resource overhead.

[0105] See also Figure 4 In step S103 of some embodiments, performing node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information includes the following steps:

[0106] Step S401, performing node energy consumption calculation processing on the target load index data according to the energy consumption model to obtain a node energy consumption value;

[0107] Step S402: sorting the node energy consumption values ​​in ascending order through the asynchronously executed sub-thread and storing them into an ordered set to obtain the node energy consumption information.

[0108] In an embodiment of the present application, a node energy consumption perception module can be set up, and the node energy consumption perception module is used to process the data of the load information monitoring module to obtain the load index data of each node in the cluster, and calculate the node energy consumption according to the energy consumption model. Specifically, the target load index data is subjected to node energy consumption calculation processing through the pre-constructed energy consumption model to obtain the node energy consumption value, and then the node energy consumption value of each node is sorted. Since the average time complexity of the sorting algorithm is high, in order to reduce the overhead of the algorithm on the entire system, the embodiment of the present application starts an asynchronously executed sub-thread to store the calculated node energy consumption value into the Redis database, in which the ordered set data structure of Redis is mainly used. The value-score stored in the ordered set represents the node identifier and the node energy consumption value respectively. The ordered set will be sorted in ascending order by default according to the size of the score (node ​​energy consumption value), thereby obtaining the final node energy consumption information. The embodiment of the present application calculates the node energy consumption value through the energy consumption model, and uses an ordered set to store the node energy consumption value and node energy consumption information, which can monitor the energy consumption of each node in the cluster, thereby facilitating subsequent task scheduling according to the node energy consumption and improving the scheduling efficiency.

[0109] In some embodiments, before performing node energy consumption perception processing on the target load indicator data according to the energy consumption model to obtain node energy consumption information, the method further includes constructing the energy consumption model, which specifically includes the following steps:

[0110] Build an experimental cluster environment;

[0111] Collecting and processing node power consumption information of the experimental cluster environment to obtain power consumption sampling data;

[0112] The power consumption sampling data is modeled according to a polynomial regression model to obtain the energy consumption model.

[0113] In the embodiment of the present application, the experimental cluster environment is constructed as follows: Figure 5As shown, it mainly includes a laptop as a client node 501, the client node uses the JMeter stress testing tool to send HTTP requests, and simulates multi-threading to perform stress testing on the back-end server; a Ribbon load balancing server 502, wherein the load balancing server has a built-in energy-saving scheduling algorithm for energy consumption awareness provided by the embodiment of the present application; three back-end servers, including a first server node 503, a second server node 504 and a third server node 505, respectively deploy the same Web service to process client HTTP requests, and each server node is connected to an HP9800 electrical parameter meter 506, which can count the power consumption of the target server in real time, connect to the power supply 507 through a Micro-USB to USB cable, and transmit power consumption data to the computing node 508 for power collection in real time. This experiment builds a heterogeneous Web cluster environment, which mainly includes a client node, a Ribbon load balancing server, three Web servers, etc. The relevant configuration information of each node is shown in Table 1 below:

[0114]

[0115] Table 1 Cluster node configuration information table

[0116] Among them, the power consumption detector can be used to collect the power consumption information of the target host in real time. It will display the current current, voltage, power and other data, and can transmit the information to the computing node that collects the power information in time through its own Micro-USB interface. The computing node that collects power can use the power consumption monitor data analysis system to view the power consumption trend in real time, and can manually control the power consumption data sampling rate, count the sampling time, current, voltage, active power and other data, and export them to excel, txt and other files. The power consumption data collected by the computing node in the load range 34-38 of the embodiment of the present application is shown in Table 2 below:

[0117] Receiving time Current Voltage Power Factor Active Power Annual electricity consumption Active energy Reactive energy 11:54:58 0.32247 241.40 0.94018 73.19065 293.86047 1.94971 0.51236 11:55:00 0.33133 241.40 0.89202 71.35191 286.47794 1.94981 0.51236 11:55:02 0.32351 241.30 0.95035 74.19621 297.89783 1.94981 0.51236 11:55:04 0.3237 241.30 0.90223 70.47563 282.95966 1.94981 0.51236 11:55:06 0.31797 241.30 0.95054 72.93208 292.82233 1.94992 0.51236 11:55:08 0.31227 241.40 0.76836 57.92046 232.55063 1.94992 0.51236 11:55:10 0.3311 241.30 0.95453 76.26481 306.20322 1.94992 0.51236 11:55:12 0.32512 241.30 0.92985 72.94644 292.87997 1.95002 0.51236 11:55:14 0.32047 241.30 0.87255 67.48767 270.96301 1.95002 0.51236 11:55:16 0.32116 241.40 0.85008 65.89314 264.56094 1.95013 0.51236

[0118] Table 2 Details of electricity consumption sampling data

[0119] Finally, a model is established based on the power consumption data of a certain node above, and a polynomial regression model is used to model the power consumption sampling data of the node. When the polynomial regression model is set to degree 2, the goodness of fit R 2 is 0.9882. Substitute the regression coefficient and intercept into the binary second-order polynomial equation to obtain the energy consumption model of the node. The expression of the energy consumption model is shown in the following formula:

[0120]

[0121] In the formula, y represents energy consumption, x1 represents CPU usage, and x2 represents memory usage. Figure 6 , a visualization result after fitting the polynomial model is as follows Figure 6 The embodiment of the present application integrates the constructed energy consumption model into the scheduling model, so as to dynamically perceive the node energy consumption, thereby improving the effect of cluster load balancing.

[0122] See also Figure 7 In step S104 of some embodiments, allocating nodes from the cluster for task scheduling according to the node energy consumption information includes the following steps:

[0123] Step S701, calculating the average energy consumption of cluster nodes according to the node energy consumption information;

[0124] Step S702, performing node screening processing on the cluster according to the average energy consumption of the cluster nodes to obtain an initial node set;

[0125] Step S703, traversing and sorting the initial node set to obtain the allocation node with the largest node remaining load rate;

[0126] Step S704: performing task scheduling on the allocation node.

[0127] In an embodiment of the present application, energy consumption information of all nodes is obtained from the Redis database, and the average energy consumption of the cluster nodes is calculated, wherein the calculation formula of the average energy consumption (Average Energy Consumption, AEC) of the cluster nodes is as follows:

[0128]

[0129] Where N is the number of nodes in the cluster, EC i(i=0,1,...,N) represents the current energy consumption of the i-th node in the cluster, and AEC represents the average energy consumption of each node in the current cluster. Then query the low-power nodes in the cluster that have a smaller energy consumption value than the average energy consumption value of the cluster nodes to obtain the initial node set, traverse each low-power node in the initial node set to determine whether the node instance is still running. If the service instance is not running, delete the relevant energy consumption data in Redis. Otherwise, calculate the node's remaining load rate according to the result of the load information collection module, where the size of the remaining load rate determines the current node's remaining load capacity. The higher the value, the more tasks can be accepted. Finally, sort the initial node set according to the remaining load rate, and obtain the allocation node with the largest node remaining load rate, so that the task scheduling module can schedule the allocation node. It can be imagined that when the allocation node needs to continue to process tasks after the task scheduling is performed on the allocation node, the allocation node can be deleted from the initial node set, and the node with the largest node remaining load rate in the initial node set can be selected as the allocation node for allocation. The embodiment of the present application allocates nodes from the cluster for task scheduling according to the node energy consumption information, and can preferentially schedule tasks to low-energy consumption nodes, thereby reducing the overall energy consumption of the cluster and achieving the purpose of energy saving.

[0130] In step S703 of some embodiments, the traversal and sorting of the initial node set to obtain the allocation node with the maximum node residual load rate includes the following steps:

[0131] The initial node set is screened according to the running status of the node service instance to obtain a target node set;

[0132] Calculating the node residual load rate of the target node set to obtain the residual load rate;

[0133] The target node set is sorted according to the remaining load rate to obtain an allocation node with the largest remaining load rate.

[0134] In the embodiment of the present application, the initial node set is screened according to the running status of the node service instance to determine whether the service instance of each node in the initial node set is still running. If the service instance is not running, the relevant energy consumption data in Redis is deleted, and the node is deleted from the initial node set to obtain the target node set. Then, the target node set is processed to calculate the node residual load rate to obtain the residual load rate. First, the inherent load capacity of the i-th node in the cluster is calculated. The inherent load capacity is calculated as follows:

[0135]

[0136] In the formula, They represent the maximum occupancy of CPU, memory, and disk IO of the i-th node respectively. α1, α2, and α3 are the weights of CPU, memory, and disk IO of the i-th node respectively, and they satisfy α1+α2+α3=1, where The default size is 1.

[0137] Then calculate the current load capacity of each node, where the expression of the current load capacity of the i-th node is as follows:

[0138]

[0139] In the formula, They represent the current occupancy rates of the CPU, memory, and disk IO of the i-th node respectively. β1, β2, and β3 are the weights of the CPU, memory, and disk IO of the i-th node respectively, and they satisfy β1+β2+β3=1.

[0140] The remaining load capacity of the i-th node is the inherent load capacity Subtract the current load capacity The calculation formula of the node's remaining load capacity is as follows:

[0141]

[0142] Then the residual load ratio RLR of the i-th node is i Remaining load capacity and inherent load capacity The expression of the residual load rate is as follows:

[0143]

[0144] In an ideal state, the remaining load ratio of each node in the entire cluster is equal, satisfying: RLR1=RLR2=...=RLR N ; In the embodiment of the present application, by considering the load condition of each node and performing task scheduling according to the remaining load rate of each node, tasks can be preferentially assigned to nodes with high remaining load capacity, so that the algorithm can achieve cluster load balancing as much as possible while reducing energy consumption, thereby improving the effect of cluster load balancing.

[0145] The following is a detailed description of the embodiments of the present invention with reference to specific application examples:

[0146] The present application embodiment can be applied to the load balancing application scenario of the cluster, please refer to Figure 8, by collecting load information from each server in the cluster during operation, and calculating the load capacity change rate, it is determined whether the load capacity change rate is less than the reporting threshold. If so, there is no need to report the load information collected in real time, otherwise the load information is sent to the KafKa distributed messaging system, and the CPU usage, memory usage, disk usage and other data are obtained by processing the load information, and then energy consumption perception processing is performed, and the node energy consumption of each node in the cluster is calculated according to the energy consumption model, and the energy consumption data in the KafKa distributed messaging system is updated, and the data of the KafKa distributed messaging system can be stored in the Redis database. When the server node receives an HTTP request, that is, when the node needs to be scheduled, it is determined whether there is energy consumption data in the Redis database. If not, the server node is scheduled according to the polling strategy. Otherwise, the average energy consumption is calculated, and the low-energy consumption node set is obtained according to the average energy consumption. The set is traversed to determine whether there are service instances of the nodes in the set that are still running. If so, the remaining load rate of the node is calculated, otherwise the node energy consumption data in the Redis database is deleted. The traversed node set is sorted according to the remaining load rate, the node with the largest remaining load rate is obtained, and the task is scheduled for the node.

[0147] See also Fig. 9 The embodiment of the present application further provides a cluster load balancing device, which can implement the above cluster load balancing method, and the device includes:

[0148] The first module 901 is used to obtain initial load indicator data of nodes in the cluster;

[0149] The second module 902 is used to perform load information update processing on the initial load indicator data according to the load reporting threshold to obtain target load indicator data;

[0150] The third module 903 is used to perform node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information;

[0151] The fourth module 904 is used to allocate nodes from the cluster for task scheduling according to the node energy consumption information.

[0152] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0153] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above cluster load balancing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0154] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0155] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0156] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0157] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the cluster load balancing method of the embodiment of this application;

[0158] Input / output interface 1003, used to implement information input and output;

[0159] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0160] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0161] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0162] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned cluster load balancing method is implemented.

[0163] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0164] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] The embodiments of the present application provide a cluster load balancing method, device, electronic device and storage medium. The scheme obtains initial load index data of nodes in the cluster, updates the initial load index data according to the load reporting threshold, and obtains target load index data. It can monitor the load status of each node in real time and update the load status based on the load reporting threshold, saving cluster network resource overhead. In addition, the scheme performs node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information, allocates nodes from the cluster for task scheduling according to the node energy consumption information, and can reasonably schedule tasks according to energy consumption and load conditions, effectively reducing cluster energy consumption while balancing the load of each node in the cluster, improving scheduling efficiency, reducing cluster operating costs, and realizing green and energy-saving cluster management.

[0166] The embodiment of the present application introduces the energy consumption model into the task scheduling algorithm so that the scheduling process can perceive the energy consumption of the node. By calculating the energy consumption of each node, tasks are preferentially assigned to low-energy consumption nodes to minimize energy consumption. A load balancer can also be set to integrate the resources of each node in the cluster and calculate the energy consumption of each node. It can monitor the load of each node in real time and reasonably schedule tasks according to the energy consumption and load conditions. At the same time, in order to ensure energy-saving effects and load balancing, the embodiment of the present invention proposes a load capacity change rate and a load reporting threshold. These parameters are used to dynamically adjust the load reporting frequency of the node to reduce the consumption of network resources.

[0167] The embodiment of the present application significantly reduces the overall energy consumption of the Web cluster by scheduling tasks to low-energy nodes first, achieving the purpose of energy saving, and through the load information monitoring module and the residual load rate calculation module, the load balancing of each node is guaranteed, avoiding the situation where individual nodes are overloaded or idle. In addition, the embodiment of the present application is not only energy-aware, but also has efficient task allocation capabilities. By optimizing the load reporting mechanism, the consumption of network resources is reduced, the utilization efficiency of the network is improved, and the response speed and overall efficiency of task scheduling are improved. Furthermore, the embodiment of the present application reduces energy consumption and improves resource utilization through task scheduling, reduces operating costs, and realizes green and energy-saving computing cluster management.

[0168] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0169] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0170] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0172] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0173] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0174] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0178] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A cluster load balancing method, characterized in that: The method comprises the following steps: Get the initial load indicator data of the nodes in the cluster; Performing load information updating processing on the initial load indicator data according to the load reporting threshold to obtain target load indicator data; Perform node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information; Nodes are allocated from the cluster for task scheduling according to the node energy consumption information.

2. The method according to claim 1, characterized in that The step of performing load information updating processing on the initial load indicator data according to the load reporting threshold to obtain target load indicator data comprises the following steps: The load information monitoring module periodically collects and processes the node load information to obtain the collected period load information; Performing load capacity change calculation processing on the collected periodic load information to obtain a node load capacity change rate; When the change rate of the node load capacity is greater than the load reporting threshold, the collection period load information is reported and processed to update the target load indicator data.

3. The method according to claim 2, characterized in that The step of performing load capacity change calculation processing on the collected periodic load information to obtain a node load capacity change rate comprises the following steps: Obtaining the load capacity at the start time and the load capacity at the end time from the acquisition period load information; The load capacity at the end time and the load capacity at the start time are subtracted from each other, and the calculated difference is divided by the cycle time value and the absolute value is calculated to obtain the node load capacity change rate.

4. The method according to claim 1, characterized in that The step of performing node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information includes the following steps: Perform node energy consumption calculation processing on the target load index data according to the energy consumption model to obtain a node energy consumption value; The node energy consumption values ​​are sorted in ascending order by an asynchronously executed sub-thread and stored in an ordered set to obtain the node energy consumption information.

5. The method according to claim 1, characterized in that Before performing node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information, the method further includes constructing the energy consumption model, which specifically includes the following steps: Build an experimental cluster environment; Collecting and processing node power consumption information of the experimental cluster environment to obtain power consumption sampling data; The power consumption sampling data is modeled according to a polynomial regression model to obtain the energy consumption model.

6. The method according to claim 1, characterized in that The allocating nodes from the cluster for task scheduling according to the node energy consumption information comprises the following steps: Calculate the average energy consumption of cluster nodes according to the node energy consumption information; Perform node screening on the cluster according to the average energy consumption of the cluster nodes to obtain an initial node set; Performing traversal and sorting processing on the initial node set to obtain the allocation node with the largest node remaining load rate; Performing task scheduling on the allocation node.

7. The method according to claim 6, characterized in that The traversal and sorting of the initial node set to obtain the allocation node with the largest node residual load rate comprises the following steps: The initial node set is screened according to the running status of the node service instance to obtain a target node set; Calculating the node residual load rate of the target node set to obtain the residual load rate; The target node set is sorted according to the remaining load rate to obtain an allocation node with the largest remaining load rate.

8. A cluster load balancing device, characterized in that: The device comprises: The first module is used to obtain the initial load indicator data of the nodes in the cluster; The second module is used to perform load information update processing on the initial load indicator data according to the load reporting threshold to obtain target load indicator data; The third module is used to perform node energy consumption perception processing on the target load index data according to the energy consumption model to obtain node energy consumption information; The fourth module is used to allocate nodes from the cluster for task scheduling according to the node energy consumption information.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images