Self-adaptive load balancing method and system for distributed system
By using an adaptive load balancing method, the performance metrics of replica nodes are dynamically perceived and intelligently scheduled, solving the problem that traditional strategies fail to consider real-time load and performance fluctuations, and achieving lower microservice response latency and higher load balancing quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-03-13
AI Technical Summary
Traditional load balancing strategies fail to effectively consider the real-time load and performance fluctuations of replica nodes in large-scale online data-intensive applications, resulting in high tail latency, especially under high load or heterogeneous hardware environments.
An adaptive load balancing method is adopted, which dynamically senses the performance indicators of replica nodes through a detection module and a performance prediction module, uses a multiple linear regression model to predict latency, and performs intelligent scheduling based on performance scores, dynamically adjusting the detection time interval to optimize load balancing.
It effectively reduces the response latency of microservices, especially tail latency, improves the quality of load balancing and the response speed of the system, and reduces the impact of latency on enterprise revenue.
Smart Images
Figure CN121664809A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of distributed computing and cloud computing technology, specifically relating to an adaptive load balancing method and system for distributed systems. Background Technology
[0002] In modern large-scale online data-intensive applications (such as Apache Cassandra and Microsoft Bing), a single user request is typically broken down into multiple parallel microservices (lightweight computation processes). These microservices can compute in parallel, with each microservice calculating the correlation between the user request and the data to obtain a partial result. The application then merges the results from these microservices to generate the final response to the user request.
[0003] Meanwhile, in large-scale online data-intensive applications, enterprises typically adopt a distributed system design, deploying data redundantly across multiple geographically dispersed replica nodes to improve overall system availability and fault tolerance. In this case, each microservice can request any replica node. Therefore, considering the characteristics of large-scale online data-intensive applications, a user request is usually split into multiple microservices that process in parallel, and each microservice needs to access a different cluster of server replica nodes.
[0004] Online data-intensive applications typically focus on response latency when processing user requests. Systems often set maximum latency limits; results from lightweight computations exceeding these limits are discarded, impacting the quality of the final user response. Furthermore, user request response latency can affect enterprise revenue. For example, in Microsoft Bing, a 500-millisecond increase in user request response latency results in approximately $3.16 million in lost revenue. In summary, latency, especially tail latency, affects not only the quality of user request responses but also enterprise revenue. Therefore, a key task for large-scale online data-intensive applications is reducing microservice response latency, and load balancing plays a crucial role in reducing microservice response latency, particularly tail latency.
[0005] When a client initiates a microservice read request, the system needs to select the optimal node from multiple replica nodes to achieve the shortest response latency. Traditional load balancing strategies such as "nearest selection," "dynamic snitch," or "round-robin" often use only simple static global configurations or only consider simple parameters such as network latency (RTT), ignoring dynamic performance factors such as the real-time load, I / O pressure, and replica node dynamics of each replica node. This results in high tail latency for traditional load balancing strategies, especially under high load or heterogeneous hardware environments.
[0006] Therefore, a load balancing strategy is urgently needed to solve the above-mentioned technical problems. Summary of the Invention
[0007] In view of the above, the purpose of this invention is to provide an adaptive load balancing method and system for distributed systems, which adopts a load balancing strategy that can dynamically sense the impact of instantaneous performance fluctuations of replica nodes on response latency and perform intelligent scheduling accordingly. This load balancing strategy can enhance the ability of nodes to respond to future performance fluctuations and improve the quality of microservice load balancing selection.
[0008] To achieve the above-mentioned objectives, the embodiments provide an adaptive load balancing system for distributed systems, which configures a load balancer for each client in the distributed system. The load balancer includes a detection module and a performance prediction module. The detection module is responsible for sending detection requests to all server replica nodes at certain detection intervals to obtain the performance index data of each replica node. Based on the performance index data, it recalculates the current load of the entire distributed system and dynamically adjusts and updates its own detection interval according to the current load. At the same time, it synchronizes the performance index data to the performance prediction module. The performance prediction module is responsible for receiving real-time performance metric data from the detection module, recalculating the performance score of each replica node through the performance prediction model, and sorting all replica nodes according to their performance scores so that the microservices can make load balancing requests in a timely manner.
[0009] Preferably, the collected performance metrics include: network round-trip latency from the client to the server replica node, overall response latency of the server replica node, and request queuing depth, wherein the overall response latency includes the queuing latency of the replica node and the microservice processing latency.
[0010] Preferably, recalculating the current load of the distributed system based on performance index data includes: For each replica node s, its normalized load at time t Defined as the ratio of the current queue length to the maximum queue length, i.e.: in, For replica nodes In time The real-time queue length is specifically fed back by the server in the probe response; This represents the maximum allowed queue length for replica nodes. The client maintains a smooth load estimate for each replica. As the current load of the calculation; in, It is the smoothing factor for the exponential moving average.
[0011] Preferably, the detection time interval for dynamically adjusting and updating itself based on the current load includes: in,, Indicates the basic detection interval. It is the load sensitivity coefficient, which controls the degree to which the load amplifies the interval. When the current load is... That is, during idle time, the time interval The system maintains the shortest detection time interval; when That is, when fully loaded, The system maintains the longest possible detection interval.
[0012] Preferably, the performance score of each replica node is recalculated using a performance prediction model, including: The performance prediction model includes a delayed prediction model and a transformation operation. The delayed prediction model is based on historical data of performance indicators and is established using a multiple linear regression model. The latency prediction model is used to calculate the latency data for each server replica node based on the current performance metric data. The latency prediction model is as follows: in, For replica nodes Prediction delay; The regression coefficients are calculated from historical data using the least squares method. For clients to replica nodes Network round-trip time; For nodes Overall response delay This includes queuing latency for replica nodes and microservice processing latency; Calculate a performance score based on latency data, and predict latency. Convert to performance score .
[0013] Preferably, the prediction delay is calculated using the following formula. Convert to performance score : in, To prevent small constants from being divided by zero, a higher score indicates a faster expected response for that node.
[0014] Preferably, all replica nodes are sorted according to performance scores so that microservices can make load balancing requests in a timely manner, including: The client selects the replica node with the highest performance score from all available replica nodes to send a request, or randomly selects from the Top-K high-scoring nodes to achieve load distribution.
[0015] To achieve the above-mentioned objectives, embodiments of the present invention also provide an adaptive load balancing method for distributed systems, comprising: The detection module periodically sends detection requests to all replica nodes according to the detection time interval. After obtaining the real-time performance index data of the performance detection response, it performs two operations in parallel: (1) Based on the real-time performance index data of the detection response, it evaluates the current load of the system and adjusts the detection time interval of the detection module according to the real-time load of the system; (2) It synchronizes the real-time performance index data obtained in real time to the performance prediction module. The performance prediction module receives real-time performance metrics data from replica nodes and microservice requests simultaneously, and performs two operations in parallel: (1) When the performance prediction module does not need to update the model parameters, it recalculates the performance scores of all replica nodes through the performance prediction model and processes all replica nodes in order of score ranking so as to make a load balancing selection as quickly as possible; at the same time, when it is determined that the model parameters in the performance prediction module need to be updated, it recalculates the autoregressive parameters using the probed real-time performance metrics data to ensure that the performance prediction module can make the best load balancing selection; (2) The performance prediction module accepts microservice requests and tries its best to make load balancing requests as quickly as possible.
[0016] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention employs an adaptive load-aware load balancing strategy. This strategy considers the instantaneous performance fluctuations of server replica nodes, predicts the performance scores of replica nodes, and simultaneously takes into account the impact of performance probe requests to replica nodes on the system load, thereby reducing tail latency during microservice request distribution. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart illustrating the adaptive load balancing method provided in the embodiment; Figure 2 This is a schematic diagram of the workflow of the detection module provided in the embodiment; Figure 3This is a schematic diagram of the workflow of the performance prediction module provided in the embodiment. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0020] The distributed system has multiple clients and multiple server replica node clusters, and each server replica node cluster has multiple replica nodes for task computation.
[0021] In the adaptive load balancing scheme for distributed systems provided in the embodiment, a load balancer is configured for each client. The load balancer includes a detection module and a performance prediction module, which realizes adaptive load balancing of microservices on the client in the distributed system.
[0022] The detection module is responsible for sending detection requests to all server replica clusters at certain detection intervals to obtain performance metrics data for each replica node. These performance metrics data include network round-trip latency from the client to the replica node, overall response latency of the replica node, and request queuing depth. The request queuing depth includes real-time queue length n and maximum queue length N. The overall response latency includes queuing latency of the replica node and microservice processing latency.
[0023] After obtaining performance metric data from each replica node, the probing module evaluates and recalculates the current load of the entire distributed system based on this data, and dynamically adjusts its probing interval accordingly. Simultaneously, the probing module synchronizes the performance metric data to the performance prediction module.
[0024] This includes calculating the current load of the distributed system based on performance metric data, including: For each replica node s, its normalized load at time t Defined as the ratio of the current queue length to the maximum queue length, i.e.: in, For replica nodes In time The real-time queue length is specifically fed back by the server in the probe response; This represents the maximum allowed queue length for replica nodes. The client maintains a smooth load estimate for each replica. As the current load of the calculation; in, It is the smoothing factor for the exponential moving average.
[0025] The detection time interval is dynamically adjusted based on the current load, including: in,, Indicates the basic detection interval. It is the load sensitivity coefficient, which controls the degree to which the load amplifies the interval. When the current load is... That is, during idle time, the time interval The system maintains the shortest detection time interval; when That is, when fully loaded, The system maintains the longest possible detection interval.
[0026] The performance prediction module receives real-time performance metric data from the probing module and recalculates the performance score of each replica node using the performance prediction model. The module then sorts all replica nodes according to their performance scores, enabling microservices to make timely load balancing requests. Furthermore, the performance prediction module periodically updates the relevant parameters of the autoregressive coefficient formula in the performance prediction model based on real-time performance data to ensure that it calculates more accurate scores.
[0027] The performance prediction model includes a latency prediction model and a conversion operation. The latency prediction model is based on historical performance data and is built using a multiple linear regression model. Specifically, the performance prediction model recalculates the performance score of each replica node, including: First, the latency data for each server replica node is calculated based on the current performance metric data using a latency prediction model. The latency prediction model is as follows: in, For replica nodes Prediction delay; The regression coefficients are calculated from historical data using the least squares method. For clients to replica nodes Network round-trip time; For nodes Overall response delay This includes queuing latency for replica nodes and microservice processing latency; Then, a performance score is calculated based on the latency data, and the predicted latency is transformed through a transformation operation. Convert to performance score In practice, the following formula is used to calculate the prediction delay. Convert to performance score : in, To prevent small constants from being divided by zero, a higher score indicates a faster expected response for that node.
[0028] The performance prediction module receives microservice requests in real time. Given the performance scores of each replica node, the module strives to make load balancing selections as quickly as possible to prevent excessive latency in microservice request responses during the load balancing process. The load balancing selection made by the performance prediction module is based on real-time data from the replica nodes, aiming for the selected replica node to complete and respond to microservice requests at the fastest speed. Specifically, it selects the replica node with the highest performance score from all available replica nodes to send requests, or randomly selects from the Top-K high-scoring nodes to achieve load distribution.
[0029] Specifically, adaptive load balancing methods include: S1, the detection module, based on a timer, periodically sends detection requests to all replica nodes according to the detection time interval. After obtaining real-time performance indicator data of the performance detection response, it performs parallel operations such as... Figure 3 The two steps shown are as follows: (1) Evaluate the current load of the system based on the real-time performance index data of the detection response, and adjust the detection time interval of the detection module based on the real-time load of the system; (2) Synchronize the real-time performance index data obtained in real time to the performance prediction module.
[0030] S2, the performance prediction module can simultaneously receive two different inputs: real-time performance metric data from replica nodes and microservice requests, and then perform parallel processing such as... Figure 2 The two-step operation shown: (1) When the performance prediction module does not need to update the model parameters, it recalculates the performance scores of all replica nodes through the performance prediction model and sorts all replica nodes by score to make the load balancing selection as quickly as possible. At the same time, when it is determined that the model parameters in the performance prediction module need to be updated, it recalculates the autoregressive parameters using the real-time performance index data (i.e., probe data) to ensure that the performance prediction module can make the best load balancing selection.
[0031] (2) The performance prediction module accepts microservice requests and tries its best to make load balancing requests as quickly as possible. Based on the performance score ranking, the load balancing selection made by the performance prediction module will allocate the microservice to the node with the best real-time performance and expect to get a response with the shortest latency.
[0032] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An adaptive load balancing system for distributed systems, characterized in that, Configure a load balancer for each client in the distributed system. The load balancer includes a probing module and a performance prediction module. The detection module is responsible for sending detection requests to all server replica nodes at certain detection intervals to obtain the performance index data of each replica node. Based on the performance index data, it recalculates the current load of the entire distributed system and dynamically adjusts and updates its own detection interval according to the current load. At the same time, it synchronizes the performance index data to the performance prediction module. The performance prediction module is responsible for receiving real-time performance metric data from the detection module, recalculating the performance score of each replica node through the performance prediction model, and sorting all replica nodes according to their performance scores so that the microservices can make load balancing requests in a timely manner.
2. The adaptive load balancing system for distributed systems according to claim 1, characterized in that, The collected performance metrics include: network round-trip latency from the client to the server replica node, overall response latency of the server replica node, and request queuing depth. The overall response latency includes the queuing latency of the replica node and the microservice processing latency.
3. The adaptive load balancing system for distributed systems according to claim 1, characterized in that, Recalculate the current load of the distributed system based on performance metric data, including: For each replica node s, its normalized load at time t Defined as the ratio of the current queue length to the maximum queue length, i.e.: in, For replica nodes In time The real-time queue length is specifically fed back by the server in the probe response; This represents the maximum allowed queue length for replica nodes. The client maintains a smooth load estimate for each replica. As the current load of the calculation; in, It is the smoothing factor for the exponential moving average.
4. The adaptive load balancing system for distributed systems according to claim 1, characterized in that, The detection interval is dynamically adjusted based on the current load, including: in,, Indicates the basic detection interval. It is the load sensitivity coefficient, which controls the degree to which the load amplifies the interval. When the current load is... That is, during idle time, the time interval The system maintains the shortest detection time interval; when That is, when fully loaded, The system maintains the longest possible detection interval.
5. The adaptive load balancing system for distributed systems according to claim 1, characterized in that, The performance score of each replica node is recalculated using a performance prediction model, including: The performance prediction model includes a delayed prediction model and a transformation operation. The delayed prediction model is based on historical data of performance indicators and is established using a multiple linear regression model. The latency prediction model is used to calculate the latency data for each server replica node based on the current performance metric data. The latency prediction model is as follows: in, For replica nodes Prediction delay; The regression coefficients are calculated from historical data using the least squares method. For clients to replica nodes Network round-trip time; For nodes Overall response delay This includes queuing latency for replica nodes and microservice processing latency; Calculate a performance score based on latency data, and predict latency. Convert to performance score .
6. The adaptive load balancing system for distributed systems according to claim 5, characterized in that, The following formula will be used to predict the delay. Convert to performance score : in, To prevent small constants from being divided by zero, a higher score indicates a faster expected response for that node.
7. The adaptive load balancing system for distributed systems according to claim 1, characterized in that, Sort all replica nodes according to performance scores so that microservices can make load balancing requests in a timely manner, including: The client selects the replica node with the highest performance score from all available replica nodes to send a request, or randomly selects from the Top-K high-scoring nodes to achieve load distribution.
8. An adaptive load balancing method for distributed systems, characterized in that, The method employs the system described in any one of claims 1-7 and includes the following steps: The detection module periodically sends detection requests to all replica nodes according to the detection time interval. After obtaining the real-time performance index data of the performance detection response, it performs two operations in parallel: (1) Based on the real-time performance index data of the detection response, it evaluates the current load of the system and adjusts the detection time interval of the detection module according to the real-time load of the system; (2) It synchronizes the real-time performance index data obtained in real time to the performance prediction module. The performance prediction module receives real-time performance metrics data from replica nodes and microservice requests simultaneously, and performs two operations in parallel: (1) When the performance prediction module does not need to update the model parameters, it recalculates the performance scores of all replica nodes through the performance prediction model and processes all replica nodes in order of score ranking so as to make a load balancing selection as quickly as possible; at the same time, when it is determined that the model parameters in the performance prediction module need to be updated, it recalculates the autoregressive parameters using the probed real-time performance metrics data to ensure that the performance prediction module can make the best load balancing selection; (2) The performance prediction module accepts microservice requests and tries its best to make load balancing requests as quickly as possible.