Load balancing method and device based on reinforcement learning, electronic equipment and medium

Abstract

The invention discloses a load balancing method and device based on reinforcement learning, electronic equipment and a medium, and relates to the field of distributed systems.The method comprises the steps that a request instruction is received and responded, and current moment state information of a system is obtained according to the request instruction, and determining an expected cumulative reward value set based on the state-action value corresponding information, the discrete action set and the current state information, then determining a target action based on the expected cumulative reward value set and a preset limiting condition, and determining a target node according to the target action after the target action is determined, so that the target node responds to the request. Therefore, according to the method, traffic scheduling real-time decision and dynamic selection of nodes are carried out through a dynamic reward mechanism and reinforcement learning, the technical problems of response lag and local overload of static load balancing under burst traffic are solved, and the technical effect of global adaptive load balancing is achieved.

Description

Technical Field

[0001] This invention relates to the field of distributed systems, and more particularly to a load balancing method, apparatus, electronic device, and medium based on reinforcement learning. Background Technology

[0002] In distributed systems, dynamic load balancing is a key technology for improving performance and energy efficiency. Traditional load balancing mechanisms such as round-robin or weighted least connections have two major drawbacks: First, static decision-making is lagging and cannot respond to sudden surges in node request volume in a timely manner, resulting in too many requests being concentrated on a single shard, leading to node overload. Second, the local optimum trap is based only on instantaneous load distribution, ignoring historical states and global resource coupling, thus making it difficult to adapt to dynamic environments. Summary of the Invention

[0003] This application provides a load balancing method, apparatus, electronic device, and medium based on reinforcement learning, which at least solves the problems of delayed response and local overload in load balancing under burst traffic in related technologies, and achieves global load balancing.

[0004] This application provides a load balancing method based on reinforcement learning, comprising: obtaining the current state information of the system based on a request instruction; determining the expected cumulative reward value set based on state-action value correspondence information, a discrete action set, and the current state information, wherein the state-action value correspondence information is based on reinforcement learning output, and the discrete action set is used to represent the node selection strategy corresponding to different actions; determining the target action based on the expected cumulative reward value set and preset constraints; and determining the target node based on the target action.

[0005] This application also provides a load balancing device based on reinforcement learning, comprising: an acquisition module for acquiring the current state information of the system based on a request instruction; a first determination module for determining an expected cumulative reward value set based on state-action value correspondence information, a discrete action set, and the current state information, wherein the state-action value correspondence information is based on reinforcement learning output, and the discrete action set is used to characterize the node selection strategy corresponding to different actions; a second determination module for determining a target action based on the expected cumulative reward value set and preset constraints; and a third determination module for determining a target node based on the target action.

[0006] This application also provides an electronic device, including a memory for storing a computer program; and a processor for implementing the steps of the aforementioned reinforcement learning-based load balancing method when executing the computer program.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the aforementioned reinforcement learning-based load balancing method.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned reinforcement learning-based load balancing method.

[0009] This application achieves dynamic allocation of user requests by determining the expected cumulative reward value set based on the current state information and reinforcement learning to obtain a discrete action set, determining the target action based on the constraints, and determining the target node based on the target action so that the target node can respond to the request. This solves the problem of response lag and local overload in traditional load balancing under burst traffic and achieves global load balancing. Based on preset constraints as hard constraints, it reduces computational complexity and improves decision-making speed. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a diagram illustrating the architecture of a reinforcement learning model according to some embodiments of this application; Figure 2 Here is a flowchart of a reinforcement learning-based load balancing method according to some embodiments of this application; Figure 3 This is a flowchart of an action filtering mechanism according to some embodiments of this application; Figure 4 This is a block diagram of a reinforcement learning-based load balancing device according to some embodiments of this application; Figure 5 This is a schematic diagram of an electronic device according to some embodiments of this application. Detailed Implementation

[0012] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] The specific application environment architecture or specific hardware architecture on which the execution of the load balancing method depends is described here.

[0016] Reference Figure 1 The reinforcement learning model architecture diagram shown represents the specific application environment architecture upon which the execution of this load balancing method depends, which consists of four closed-loop interconnected components.

[0017] The dynamic perception module integrates multi-source data, such as monitoring, logs, and tracking information. When a request arrives, it outputs the current system status information to the reinforcement learning decision engine to assess the health and load of nodes. Furthermore, when abnormal activity is detected in a node or data shard, such as excessive load or a significant increase in tail latency, the module triggers the decision engine to adjust and migrate new requests to nodes with lower loads, thereby preventing the formation of hotspot nodes.

[0018] Tail latency is defined as a percentile metric for the response time of concurrent requests in a distributed storage scenario. In a distributed storage system, tail latency refers to the response time of the slowest subset of requests in a batch of concurrent requests, typically denoted as P99 or P999. For example, P99 = 200ms means that 99% of requests have a response time of less than or equal to 200ms, with the remaining 1% being even slower. Its calculation is based on recording the end-to-end response time of all requests within a sliding window, sorting the collected latency samples in ascending order, and then taking the Nth percentile value. In engineering implementations, streaming algorithms such as HdrHistogram and T-Digest are used for online output. Furthermore, the tail latency metric, expressed as P99 or P999, quantifies the true end-to-end experience of the slowest 1% or 0.1% of requests in a shared storage system, and is a core baseline in the service level agreement (SLA) framework. When P99 abnormally increases, it can be immediately mapped to root causes such as hotspot nodes, allowing the operations and maintenance system to trigger replica migration, load balancing, or rate limiting strategies to prevent a single node from slowing down the entire cluster.

[0019] The reinforcement learning decision engine takes real-time four-dimensional state as input and evaluates the long-term benefits of all candidate nodes within milliseconds using a pessimistic algorithm. It quickly identifies the safest node and issues a request. Subsequently, it updates the value table online based on the reward signal fed back by the system, forming a closed loop of "perception-decision-execution-feedback". This continuously balances the load, suppresses hotspots, and reduces tail latency, achieving adaptive optimization of the entire cluster.

[0020] After receiving the target action from the reinforcement learning decision engine, the action execution module first parses it into a corresponding set of candidate nodes. For example, if the action is "select the node with the lowest load," the node with the lowest CPU (Central Processing Unit, or controller) utilization is selected; if the action is "select the node with the lowest memory," the node with the lowest memory usage is selected. Similarly, the "optimal latency" and "lowest heat" actions are parsed. Subsequently, the module determines the candidate node selection based on the current node load status, response latency, and the value of the reinforcement learning engine. After determining the final target node, the module sends the request to the target node. After the node executes the request, it records the execution result to update the reinforcement learning decision strategy, completing millisecond-level secure deployment and closed-loop feedback.

[0021] The dynamic reward engine uses a nonlinear multi-objective function to weight response time, throughput, and error rate in real time, and dynamically adjusts the weights and squared penalties for anomalies according to the environment, compressing multi-dimensional performance indicators into a single instant reward to drive reinforcement learning to converge accurately.

[0022] The embodiments of this application provide a load balancing method based on reinforcement learning, the execution flow of the load balancing method, and a detailed description of the method.

[0023] Figure 2 Here is a flowchart of a reinforcement learning-based load balancing method according to some embodiments of this application, with reference to... Figure 2 As shown, this reinforcement learning-based load balancing method may include: S1, obtain the current status information of the system based on the request command.

[0024] Specifically, upon receiving a request instruction, the system responds to it. The system collects monitoring data, logs, and tracing information at fixed time intervals; for example, this interval can be set to 100ms or other reasonable times, depending on business needs. Based on the monitoring data, logs, and tracing information, the system obtains status information, which can be used to assess the health and load of nodes.

[0025] S2, determine the expected cumulative reward value set based on state-action value correspondence information, discrete action set and current state information, where state-action value correspondence information is based on reinforcement learning output, and discrete action set is used to represent the node selection strategy corresponding to different actions.

[0026] Specifically, the acquired state information is passed to the reinforcement learning decision engine. Based on the state-action value correspondence information, the expected cumulative reward value of the current state information under different discrete action sets is obtained, thus obtaining the expected cumulative reward value set. The state-action value correspondence information is the expected cumulative result calculated by the system performing all discrete actions in all possible states, and is exemplarily represented by a two-dimensional mapping table, namely the state-action mapping table. In some embodiments, a finite set of discrete actions is defined, with each action corresponding to a specific node selection or resource allocation strategy.

[0027] For example, the discrete action set is defined as At={a1, a2, ..., an}, where a1, a2, and an represent different actions. The current state information can be represented as a state vector. Based on the current state information (e.g., the state vector St), the expected cumulative reward corresponding to the discrete action set is extracted to form the expected cumulative reward value set {Q(St, a1), Q(St, a2), ..., Q(St, an)}.

[0028] S3 determines the target action based on the expected cumulative reward value set and preset constraints.

[0029] Specifically, to prevent nodes from overheating, hard constraints, or preset limits, are set. These preset limits may include at least one of the following: node CPU utilization and node response latency. The specific limits can be determined based on actual needs. The expected cumulative reward value set contains the expected cumulative reward value for each action. Taking CPU utilization as an example (as a parameter in the preset limits), the CPU utilization of each node is obtained. Nodes with CPU utilization lower than the preset value are filtered out. The expected cumulative reward value for the filtered nodes is determined, and the action corresponding to the maximum expected cumulative reward value is selected as the target action.

[0030] S4, determine the target node based on the target action.

[0031] Specifically, upon receiving a request instruction, the system's current state information can be obtained based on collected monitoring data, logs, and tracking information. Reinforcement learning is then used to acquire the expected cumulative reward value for each action within this current state. This expected cumulative reward value set is then constructed based on the expected cumulative reward values ​​of all actions within the current state. Nodes that meet preset constraints are identified. Then, corresponding actions are determined based on these nodes. The expected cumulative reward value for each action is then determined. Based on the constraints corresponding to the expected cumulative reward value, the target action is determined. The corresponding target node is then identified based on the target action, and finally, the request instruction is responded to based on the target node.

[0032] Therefore, based on the current state information and reinforcement learning to obtain a discrete action set, a set of expected cumulative reward values ​​is determined, and a target action is determined according to the constraints. Based on the target action, a target node is determined so that the target node can respond to the request, thereby realizing dynamic allocation of user requests. This solves the problem of response lag and local overload in traditional load balancing under burst traffic, and realizes global load balancing. Based on preset constraints as hard constraints, computational complexity is reduced and decision-making speed is improved.

[0033] In some embodiments of this application, the target action is determined based on a set of expected cumulative reward values ​​and preset constraints, including: identifying nodes whose processor utilization is less than a preset utilization threshold and whose response latency is less than a preset time threshold as reference nodes; and, if there are multiple reference nodes, selecting the action corresponding to the maximum expected cumulative reward value of the actions corresponding to the multiple reference nodes as the target action. The preset utilization threshold and preset time threshold can be calibrated according to actual conditions; for example, the preset utilization threshold can be 85%, and the preset time threshold can be 100ms.

[0034] Specifically, refer to Figure 3 The flowchart of the action selection mechanism shown first illustrates that the system checks whether the processor (CPU) utilization and response latency of each node meet preset thresholds. A preset utilization threshold (e.g., 85%) is set to determine if a node is overloaded. A preset time threshold (e.g., 100ms) is set to determine if the node's response speed meets the requirements. The system obtains the processor utilization and response latency of each node, identifies nodes with processor utilization less than the preset utilization threshold and response latency less than the preset time threshold, and then selects the action corresponding to the maximum expected cumulative reward value from these nodes as the target action (at).

[0035] For example, taking a discrete action set including {a1, a2, a3, a4} as an example, Q(St, a1) = 2.30, the processor utilization rate of node N1 corresponding to action a1 is 90%, and the response latency is 88ms; Q(St, a2) = 2.31, the processor utilization rate of node N2 corresponding to action a2 is 82%, and the response latency is 95ms; Q(St, a3) = 2.29, the processor utilization rate of node N3 corresponding to action a3 is 75%, and the response latency is 105ms; Q(St, a4) = 2.25, the processor utilization rate of node N4 corresponding to action a4 is 80%, and the response latency is 92ms. The action filtering mechanism first filters out nodes with CPU ≥ 85% or latency ≥ 100ms, then filters out nodes N2 and N4. Next, it obtains the maximum value of the expected cumulative reward value Q(St, a2) corresponding to nodes N2 and N4, and then takes action a2 as the target action. The target node N2 corresponding to a2 sends a route so that the target node N2 can respond to the request command.

[0036] Therefore, this design employs a two-stage screening process: first, applying hard constraints, and then selecting better nodes. The first step is to use a hard threshold to instantly eliminate overloaded or slow nodes, ensuring the system's safety baseline. Subsequently, only the nodes with the highest expected cumulative reward value are selected from the qualified nodes, balancing performance and stability to achieve optimal decision-making under low risk.

[0037] In some embodiments of this application, the above-described reinforcement learning-based load balancing method further includes: determining the expected cumulative reward value of each action in the next state based on reinforcement learning when the target node executes a request instruction, wherein the next state is used to characterize the state information of the system after executing the request instruction; determining the future discount reward based on the product of the minimum expected cumulative reward value of all actions in the next state and the discount factor; determining the target reward value based on the sum of the immediate reward and the future discount reward; determining the error value of the target action based on the difference between the target reward value and the expected cumulative reward value of the target action in the current state; and updating the expected cumulative reward value of the target action based on the expected cumulative reward value of the target action in the current state, the learning rate, and the error value.

[0038] Specifically, after identifying the target node, the system controls the target node to execute request commands. After the target node completes the execution of the request commands, monitoring data, logs, and tracking information are collected, and the next state information of the system is generated. Based on the next state information, the expected cumulative reward values ​​corresponding to the discrete action sets are extracted to form the expected cumulative reward value set for the next state, and the minimum value in the expected cumulative reward value set for the next state is obtained. Then, the minimum expected cumulative reward value is multiplied by a discount factor to obtain the future discounted reward. The target reward value is determined based on the sum of the immediate reward and the future discounted reward. Then, the error value of the target action is determined based on the difference between the target reward value and the expected cumulative reward value of the target action in the current state. Finally, the expected cumulative reward value of the target action is updated based on the expected cumulative reward value, the learning rate, and the error value.

[0039] In some embodiments, the discount factor is used to balance the importance of current and future rewards, and the value of the discount factor ranges from 0 to 1. The closer the discount factor is to 0, the more it focuses on mitigating immediate hot spots; the closer it is to 1, the more it balances long-term equilibrium and low latency. Error value TD Error The expected cumulative reward value of the target action in the current state. The larger the gap between the current strategy and the target reward value (Target), the more the strategy needs to be adjusted.

[0040] Therefore, determining future discount rewards based on discount factors can avoid overly optimistic estimations when node states are uncertain. This ensures performance stability under large-scale concurrency while reducing tail latency jitter and resource waste, achieving millisecond-level hotspot migration and global load balancing. The action execution module selects target service nodes through discrete action sets and uses a filtering mechanism to constrain the action space, balancing scheduling diversity and target accuracy.

[0041] Furthermore, in some embodiments of this application, determining to update the expected cumulative reward value of the target action based on the expected cumulative reward value, learning rate, and error value of the target action in the current state includes: obtaining the product of the error value and the learning rate of the target action in the current state to obtain the updated value of the target action; and updating the expected cumulative reward value of the target action based on the sum of the expected cumulative reward value of the target action in the current state and the corresponding updated value.

[0042] Specifically, based on the error value TD of the calculated target action Error With learning rate The product of these values ​​represents the target action update value, where the learning rate is [value]. Used to control the update step size, and ∈(0,1], while α decays over time to improve the stability of later decisions.

[0043] Based on the aforementioned expected cumulative reward value Q(St, at) of the target action, the learning rate α, and the error value TD of the target action... Error Calculate the new expected cumulative reward value and replace the current expected cumulative reward value Q(St, at) in table Q. The formula is: Q(St, at) = Q(St, at) + TD Error , that is, Q(St, at)=Q(St, at)+α·[rt+γ·minQ(S t+1 ,a)-Q(St,at)] Simultaneously, acquire data to generate experience tuples (St, at, rt, s). t+1 This provides parameters for the decision engine in subsequent requests.

[0044] For example, the initial expected cumulative reward is set to Qinit(S, a) = 0, the learning rate is α = 0.3, and a learning rate decay strategy is used. Given a discount factor γ = 0.9, when executing a certain request, assume the current system state is as follows: Current expected cumulative reward value Q: Q(St, at) = 2.3, immediate reward: rt = 0.8, minimum Q value for the next state: minQ(S t+1 α) = 2.0, discount factor: γ = 0.9, learning rate: α = 0.03. The specific calculation process is as follows: Calculate future discount rewards: γ•minQ(S t+1 a) = 0.9 × 2.0 = 1.8; Calculate the target reward value Target: Target = rt + γ•minQ(S) t+1 a) = 0.8 + 1.8 = 2.6; Calculate the error value TD of the target action Error : TD Error =Target-Q(St,at)=2.6-2.3=0.3; Update Q value: Qnew(St,at)=2.3+0.03×0.3=2.309.

[0045] This design employs a reinforcement learning algorithm, updating the minimum expected cumulative reward of the next state. Combined with a learning rate that decreases with the number of decisions, it suppresses overly optimistic estimations when the node state is uncertain, thereby continuously enhancing the system's stability.

[0046] In some embodiments of this application, obtaining the immediate reward includes: obtaining the system response time, throughput, and error rate; and determining the immediate reward based on the system response time and its corresponding weight coefficient, the throughput and its corresponding weight coefficient, and the error rate and its corresponding weight coefficient.

[0047] Furthermore, the immediate reward is determined using the following formula: ; in, Indicates an immediate reward. This represents the weighting coefficient corresponding to the system response time. Indicates the attenuation factor. Indicates system response time. Indicates throughput. This represents the weighting coefficient corresponding to throughput. Represents the hyperbolic tangent function. Indicates the error rate. This represents the weighting coefficient corresponding to the error rate.

[0048] Specifically, a nonlinear multi-objective function (reward function) is used. This involves dynamically adjusting the weights and implementing a squared penalty term to achieve a weighted combination of system performance indicators. See the formula above, where... This represents the response time optimization item. This indicates a throughput improvement item. This represents the error rate penalty. The response time optimization term encourages the system to reduce request response time (RT), and is expressed by the exponential function e. -β•RT The shorter the response time, the higher the reward, and it is sensitive to long-tail latency. β is the weighting coefficient, and β is the decay factor (controlling sensitivity). The throughput enhancement term is used to encourage the system to increase throughput (TP, the number of requests processed per unit time); the tanh function maps the throughput to the interval [0, 1) to avoid numerical explosion. These are weighting coefficients. The error rate penalty term is used to penalize high error rates (ER, such as request failure rate or timeout rate) in the system; the squared term ER... 2 Enhance sensitivity to high error rates. For penalty weights.

[0049] In some embodiments, The weighting coefficient corresponding to the response time can be adjusted according to business needs, for example, in > This indicates a greater focus on node latency optimization. Typically set as a baseline value, this weighting coefficient controls the degree of influence of response time in the reward function. That is, if business requirements are particularly sensitive to response time, this value can be appropriately increased to encourage the system to take action to reduce response time. β is the decay factor, with typical values ​​of β∈[0.1, 1.0], preferably β∈[0.005, 0.02]. The larger the value, the stronger the penalty for long delays, i.e., controlling the rate of RT decay. The larger the value, the higher the system's sensitivity to latency. The weighting coefficient corresponding to throughput, and A trade-off must be struck to avoid excessive pursuit of throughput leading to increased latency. TP represents throughput and needs to be normalized to the tanh input range. Divide by the system's maximum theoretical throughput to ensure that the reward value for throughput is within a reasonable range, preventing the phenomenon of numerical explosion or gradient vanishing caused by directly substituting the original throughput values ​​of tens to millions into the reward function. This is the weighting coefficient corresponding to the error rate, usually set to a large value to ensure that the error rate converges to the safety threshold first. If business requirements are particularly sensitive to the fault tolerance rate, this value can be appropriately increased to strengthen the penalty for the error rate. ER represents the error rate (0≤ER≤1), and the proportional value is used directly.

[0050] For example, the system state after a certain execution is as follows (average over 15 seconds): response time RT = 200ms, throughput TP = 1000 requests / sec, error rate ER = 0.05 (5%). The reward function formula is calculated as follows: Response time optimization items: = = ≈0.135; =1.0 × 0.135 = 0.135; Throughput improvement items: = ≈1.0; =0.8 × 1.0 = 0.8; Error rate penalty items: =0.05 2 =0.0025; =-0.5×0.0025=-0.00125; Timely rewards: =0.135+0.8-0.00125≈0.93375.

[0051] Therefore, the formula determines the immediate reward by comprehensively considering three performance indicators: response time, throughput, and error rate, and adjusts them through weighting coefficients to achieve a comprehensive evaluation of system behavior.

[0052] In some embodiments of this application, the current state information includes: average cluster load, maximum memory pressure, tail latency, and target data shard popularity. Obtaining the current state information of the system includes: obtaining the sum of the products of the processor utilization of all nodes and their corresponding weights, and determining the average cluster load based on the ratio between the sum and the number of nodes; using the maximum value among the memory utilization of all nodes as the maximum memory pressure; using the product of the preset request response time and the latency amplification factor as the tail latency, where the latency amplification factor is greater than 1; and using the ratio of the number of shard accesses to the number of accesses to all nodes as the target data shard popularity.

[0053] Specifically, when a request arrives, the system's current state information is obtained. This state information can be represented as a vector. In this embodiment, the state information is quantized into a four-dimensional state vector, denoted as St=(Lt,Mt,Dt,Ht). Lt represents the cluster's average load, reflecting the overall computing resource utilization of the cluster. Its calculation formula is Lt=∑(node ​​CPU utilization × weight) / number of nodes. Preset weights are used to quantify the actual contribution ratio of different role nodes to the cluster's computing load. For example, the weight of a computing node is set to 1, and the weight of a storage node is set to 0.5. Mt represents the maximum memory pressure. The highest memory utilization value among all nodes is used to identify the most strained memory resource state in a node, avoiding performance bottlenecks caused by insufficient memory. Its calculation formula is Mt = MAX(Node 1 memory utilization, Node 2 memory utilization, ..., Node N memory utilization). Dt represents tail latency, which measures the system's ability to guarantee slow requests. Its calculation formula is Dt = P99 (request response time) × latency amplification factor, where P99 (request response time) refers to the 99th percentile latency of all requests in the tracing information to represent the time taken by the slowest 1% of requests. The latency amplification factor is a preset coefficient greater than or equal to 1, used to correct tail latency. Optionally, when the system timeout rate > 5%, the latency amplification factor is set to 1.5 to amplify the penalty; otherwise, it is 1.0. Ht represents the target data shard heat, which quantifies the access frequency or load of data shards to identify potential hot data. Its calculation formula is Ht = shard access count / total node access count. The state information (e.g., state vector St) collected by the dynamic perception module is injected into the reinforcement learning decision engine.

[0054] The system collects multi-dimensional metrics such as node CPU, memory, tail latency, and shard heat in real time at millisecond intervals, and immediately summarizes them into unified state information (such as state vectors). This provides accurate millisecond-level, multi-dimensional data for subsequent reinforcement learning decisions, ensuring that the load balancing strategy is always executed based on the latest environmental profile, and significantly reducing the probability of hot nodes being generated under sudden traffic surges.

[0055] In some embodiments of this application, the above-described reinforcement learning-based load balancing method further includes: not responding to request commands when it is determined that the node popularity or shard popularity is abnormal based on the target data shard popularity.

[0056] Specifically, if the target data shard's heat value exceeds a preset threshold (optionally 0.8), the node or shard's heat value is determined to be abnormal, and the node will not respond to the request, thus preventing the formation of hot nodes. For example, the system continuously monitors all shards and nodes; when a request arrives, if monitoring detects that the heat value of its target data shard is higher than 0.8, the shard is immediately marked as "overheated" during the request allocation phase. The system no longer directs new requests to the node to which the shard belongs, but instead prioritizes forwarding traffic to other shard replicas with a heat value lower than 0.8.

[0057] Therefore, this mechanism completes the judgment and redirection within a millisecond window, continuously alleviating hot spot pressure and preventing the formation of overheated fragments without manual intervention.

[0058] In some embodiments of this application, the above-described reinforcement learning-based load balancing method further includes: determining a target action based on preset rules when the node processor utilization rate is greater than or equal to a preset utilization rate threshold, or the response latency is greater than or equal to a preset time threshold, wherein the preset rules are used to characterize the business requirements determination of the request instruction.

[0059] Specifically, continue to refer to Figure 3 The flowchart of the action filtering mechanism shown indicates that if the node processor utilization rate is greater than or equal to the aforementioned preset utilization rate threshold, or the response latency is greater than or equal to the aforementioned preset time threshold, then according to the business requirements of the request, the action corresponding to a node is selected from the nodes as the target action.

[0060] Exemplarily, the processor utilization threshold is preset to 85%, and the response latency threshold is preset to 100 ms. The processor utilization of node N1 corresponding to action a1 in the discrete action set is 92%, and the response latency is 72 ms; the processor utilization of node N2 corresponding to action a2 is 78%, and the response latency is 95 ms; the processor utilization of node N3 corresponding to action a3 is 89%, and the response latency is 85 ms; the processor utilization of node N4 corresponding to action a4 is 85%, and the response latency is 110 ms. None of the above four nodes meet the hard constraints for action selection, so the current request business requirements are obtained. For example, if it is sensitive to response latency, the processor utilization requirement for the node is relaxed. Exemplarily, for latency-sensitive requests, the controller threshold can be relaxed to 90%, and the latency threshold remains 100 ms. According to the new node load and response time threshold requirements, the remaining nodes N2 and N3 are involved in the comparison, and these two nodes are sorted according to the response latency, that is, N3 (85 ms) < N2 (95 ms). Finally, the node with the minimum latency is selected, that is, N3 is determined as the target node.

[0061] This design combines the node load situation, response latency time, and the business requirements of the request, and gives priority to ensuring key performance; once the candidate set is empty, the system immediately and precisely relaxes the non-critical indicators according to the business labels, ensuring that at least one node is always within the selectable range without sacrificing the core performance requirements.

[0062] In some embodiments of the present application, the discrete action set includes a first action, a second action, a third action, and a fourth action. Among them, the first action is used to represent selecting the node with the lowest load at the current moment, the second action is used to represent selecting the node with the lowest memory pressure at the current moment, the third action is used to represent selecting the node with the lowest tail latency, and the fourth action is used to represent selecting the node with the lowest data shard heat.

[0063] Specifically, the discrete action set is defined as At = {a1, a2, a3, a4}, where a1 represents selecting the node with the lowest current load, a2 represents selecting the node with the lowest memory pressure, a3 represents selecting the node with the optimal tail latency, and a4 represents selecting the node with the lowest data shard heat. These actions are generated based on the state information (load, memory pressure, tail latency, data heat) provided by the dynamic perception module. The system compresses the infinite node selection problem into four enumerable actions of "select the lowest load, the lowest memory pressure, the optimal tail latency, and the lowest shard heat" through the discrete action set, greatly reducing the search dimension and computing power consumption; the four actions not only incorporate expert experience but also retain the exploration space of reinforcement learning, achieving diverse strategies; a single request only needs to compare four actions to make a decision, ensuring the safety and controllability of each scheduling result.

[0064] In some embodiments of the present application, the reinforcement learning includes a pessimistic Q-Learning decision algorithm.

[0065] Specifically, Q-learning is a classic model-free, offline policy reinforcement learning algorithm that learns which action is most valuable in each state by interacting with the environment. The core is to use the Q-value, or Q(state, action), to measure the value of performing an action in a given state—that is, the expected cumulative reward—to guide the agent in making optimal decisions. The basic principle of this algorithm is that the agent gains experience through interaction with the environment and iteratively adjusts the Q-value according to a pre-defined formula. Simultaneously, the algorithm balances the learning process by trying new actions to discover higher Q-values ​​and selecting the current optimal action to obtain a deterministic reward. As the number of interactions increases, the agent's understanding of the environment becomes more accurate, and the Q-value gradually converges; training ends when the change in Q-value is less than a preset threshold or when the maximum number of iterations is reached.

[0066] Q-learning is characterized by its ability to learn from interaction samples without prior environmental dynamics and reward functions; decoupling of behavioral and target policies to support experience replay; and single-step backtracking at each step to achieve low-complexity, high-data-efficiency global optimal convergence. It is also compatible with tabular or function approximation forms and has online incremental updates and exploration capabilities. Leveraging its natural balancing ability, Q-learning is widely used in scenarios that require learning optimal behavior through trial and error, such as game AI (e.g., Atari games, Go), robot control, and path planning.

[0067] Therefore, by using pessimistic Q-learning decision-making, updating the Q value with the minimum operation, and with the learning rate decaying with the number of decisions, we avoid being overly optimistic when the node state is uncertain, thus improving the system stability.

[0068] The reinforcement learning-based load balancing method in this application involves a system that collects and integrates monitoring, log, and tracing information at fixed intervals to generate status information including cluster load, memory pressure, tail latency, and data shard popularity. Based on this information, pessimistic Q-Learning selects the optimal action from a discrete action set and routes requests to the corresponding nodes in real time. After execution, a dynamic reward function calculates the immediate reward based on response time, throughput, and error rate, and updates the Q-value with a learning rate that decays with the number of decisions, forming a closed loop. This method achieves global load balancing by migrating hotspots in milliseconds under burst traffic.

[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0070] This application also provides a load balancing device based on reinforcement learning, such as... Figure 4As shown, the reinforcement learning-based load balancing device 400 includes: an acquisition module 410, a first determination module 420, a second determination module 430, and a third determination module 440.

[0071] The acquisition module 410 is used to acquire the current state information of the system based on a request command. The first determination module 420 is used to determine the expected cumulative reward value set based on state-action value correspondence information (such as a state-action mapping table), a discrete action set, and the current state information. The state-action value correspondence information is based on reinforcement learning output, and the discrete action set is used to represent the node selection strategy corresponding to different actions. The second determination module 430 is used to determine the target action based on the expected cumulative reward value set and preset constraints. The third determination module 440 is used to determine the target node based on the target action.

[0072] In some embodiments of this application, the second determining module 430 determines the target action based on the expected cumulative reward value set and preset constraints. Specifically, it is used to: determine nodes whose node processor utilization rate is less than a preset utilization rate threshold and whose response latency is less than a preset time threshold as reference nodes; and when there are multiple reference nodes, take the action corresponding to the maximum value of the expected cumulative reward value of the actions corresponding to the multiple reference nodes as the target action.

[0073] In some embodiments of this application, the aforementioned reinforcement learning-based load balancing device further includes: an update module, configured to: determine the expected cumulative reward value of each action in the next state based on reinforcement learning when the target node executes a request instruction, wherein the next state is used to characterize the state information of the system after executing the request instruction; determine the future discount reward based on the product of the minimum expected cumulative reward value of all actions in the next state and the discount factor; determine the target reward value based on the sum of the immediate reward and the future discount reward; determine the error value of the target action based on the difference between the target reward value and the expected cumulative reward value of the target action in the current state; and update the expected cumulative reward value of the target action based on the expected cumulative reward value of the target action in the current state, the learning rate, and the error value.

[0074] In some embodiments of this application, the update module determines to update the expected cumulative reward value of the target action based on the expected cumulative reward value, learning rate, and error value of the target action in the current state. Specifically, it is used to: obtain the product of the error value and the learning rate of the target action in the current state to obtain the updated value of the target action; and update the expected cumulative reward value of the target action based on the sum of the expected cumulative reward value of the target action in the current state and the corresponding updated value.

[0075] In some embodiments of this application, the update module obtains the immediate reward, specifically by: obtaining the system response time, throughput, and error rate; and determining the immediate reward based on the system response time and its corresponding weight coefficient, throughput and its corresponding weight coefficient, and error rate and its corresponding weight coefficient.

[0076] In some embodiments of this application, the update module determines the immediate reward using the following formula: ; in, Indicates an immediate reward. This represents the weighting coefficient corresponding to the system response time. Indicates the attenuation factor. Indicates system response time. Indicates throughput. This represents the weighting coefficient corresponding to throughput. Represents the hyperbolic tangent function. Indicates the error rate. This represents the weighting coefficient corresponding to the error rate.

[0077] In some embodiments of this application, the current state information includes: cluster average load, maximum memory pressure, tail latency, and target data shard popularity. Specifically, the acquisition module 410 acquires the current state information of the system, which is used to: acquire the sum of the products of the processor utilization of all nodes and their corresponding weights, and determine the cluster average load based on the ratio between the sum and the number of nodes; take the maximum value of the memory utilization of all nodes as the maximum memory pressure; take the product of the preset request response time and the latency amplification factor as the tail latency, wherein the latency amplification factor is greater than 1; and take the ratio of the number of shard accesses to the number of accesses of all nodes as the target data shard popularity.

[0078] In some embodiments of this application, the second determining module 430 is further configured to: not respond to the request instruction if it is determined that the node heat or the fragment heat is abnormal based on the target data fragment heat.

[0079] In some embodiments of this application, the second determining module 430 is further configured to: determine the target action based on a preset rule when the node processor utilization rate is greater than or equal to a preset utilization rate threshold, or the response delay is greater than or equal to a preset time threshold, wherein the preset rule is used to characterize the business requirement determination of the request instruction.

[0080] In some embodiments of this application, the discrete action set includes a first action, a second action, a third action, and a fourth action, wherein the first action is used to represent selecting the node with the least load at the current time, the second action is used to represent selecting the node with the least memory pressure at the current time, the third action is used to represent selecting the node with the least latency, and the fourth action is used to represent selecting the node with the lowest data sharding heat.

[0081] In some embodiments of this application, reinforcement learning includes a pessimistic Q-Learning decision algorithm.

[0082] It should be noted that the description of the features in the embodiment of the reinforcement learning-based load balancing device can be found in the relevant description of the embodiment of the reinforcement learning-based load balancing method, and will not be repeated here.

[0083] This application also provides an electronic device, as described in the embodiments. Figure 5 The schematic diagram of the electronic device shown shows that the electronic device 100 includes a memory 110 for storing computer programs and a processor 120 for implementing the steps of the aforementioned reinforcement learning-based load balancing method when executing the computer program.

[0084] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned load balancing method based on reinforcement learning.

[0085] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0086] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned reinforcement learning-based load balancing method.

[0087] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the reinforcement learning-based load balancing method.

[0088] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0089] The foregoing has provided a detailed description of a reinforcement learning-based load balancing method, apparatus, electronic device, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of these embodiments are only intended to aid in understanding the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims.

Claims

1. A load balancing method based on reinforcement learning, characterized in that, The method includes: Obtain the current status information of the system based on the request command; The expected cumulative reward value set is determined based on the state-action value correspondence information, the discrete action set, and the current state information. The state-action value correspondence information is based on the reinforcement learning output, and the discrete action set is used to represent the node selection strategy corresponding to different actions. The target action is determined based on the expected cumulative reward value set and preset constraints. The target node is determined based on the target action.

2. The load balancing method based on reinforcement learning according to claim 1, characterized in that, The determination of the target action based on the expected cumulative reward value set and preset constraints includes: Nodes whose processor utilization is less than a preset utilization threshold and whose response latency is less than a preset time threshold are identified as reference nodes. When there are multiple reference nodes, the action corresponding to the maximum expected cumulative reward value of the actions corresponding to the multiple reference nodes is taken as the target action.

3. The load balancing method based on reinforcement learning according to claim 1, characterized in that, Also includes: When the target node executes the request instruction, the expected cumulative reward value of each action in the next state is determined based on the state-action value correspondence information, wherein the next state is used to characterize the state information of the system after the request instruction is executed; The future discount reward is determined by multiplying the minimum expected cumulative reward value of all actions in the next state by the discount factor. The target reward value is determined based on the sum of the immediate reward and the future discount reward. The error value of the target action is determined based on the difference between the target reward value and the expected cumulative reward value of the target action in the current state; Based on the expected cumulative reward value, learning rate, and error value of the target action in the current state, the expected cumulative reward value of the target action is updated.

4. The load balancing method based on reinforcement learning according to claim 3, characterized in that, The step of updating the expected cumulative reward value of the target action based on the current state, learning rate, and error value includes: The updated value of the target action is obtained by multiplying the error value of the target action in the current state by the learning rate. The expected cumulative reward value of the target action is updated based on the sum of the expected cumulative reward value of the target action in the current state and the corresponding updated value.

5. The load balancing method based on reinforcement learning according to claim 3, characterized in that, Earning instant rewards includes: Obtain system response time, throughput, and error rate; The instant reward is determined based on the system response time and its corresponding weight coefficient, the throughput and its corresponding weight coefficient, and the error rate and its corresponding weight coefficient.

6. The load balancing method based on reinforcement learning according to claim 5, characterized in that, The instant reward is determined using the following formula: in, This refers to the instant reward. This represents the weighting coefficient corresponding to the system response time. Indicates the attenuation factor. This indicates the system response time. This indicates the throughput. This represents the weighting coefficient corresponding to the throughput. Represents the hyperbolic tangent function. This indicates the error rate. This represents the weighting coefficient corresponding to the error rate.

7. The load balancing method based on reinforcement learning according to claim 1, characterized in that, The current status information includes: cluster average load, maximum memory pressure, tail latency, and target data shard heat. The process of obtaining the system's current status information includes: Obtain the sum of the products of the processor utilization of all nodes and their corresponding weights, and determine the average load of the cluster based on the ratio between the sum and the number of nodes; The maximum memory usage among all nodes is taken as the maximum memory pressure; The product of the preset request response time and the delay amplification factor is used as the tail delay, wherein the delay amplification factor is greater than 1; The ratio of the number of accesses to the total number of accesses to all nodes is used as the target data shard popularity.

8. The load balancing method based on reinforcement learning according to claim 7, characterized in that, The method further includes: If the node or shard popularity is determined to be abnormal based on the target data shard popularity, the request command will not be responded to.

9. The load balancing method based on reinforcement learning according to claim 2, characterized in that, The method further includes: When the node processor utilization rate is greater than or equal to the preset utilization rate threshold, or the response latency is greater than or equal to the preset time threshold, the target action is determined based on preset rules, wherein the preset rules are used to characterize the business requirements determination of the request instruction.

10. The load balancing method based on reinforcement learning according to claim 1, characterized in that, The discrete action set includes a first action, a second action, a third action, and a fourth action. The first action is used to select the node with the lowest load at the current time, the second action is used to select the node with the lowest memory pressure at the current time, the third action is used to select the node with the lowest latency, and the fourth action is used to select the node with the lowest data sharding heat.

11. The load balancing method based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning includes the pessimistic Q-Learning decision algorithm.

12. A load balancing device based on reinforcement learning, characterized in that, The device includes: The acquisition module is used to obtain the current status information of the system based on the request command; The first determining module is used to determine the expected cumulative reward value set based on state-action value correspondence information, discrete action set and current state information, wherein the state-action value correspondence information is based on reinforcement learning output, and the discrete action set is used to represent the node selection strategy corresponding to different actions; The second determining module is used to determine the target action based on the expected cumulative reward value set and preset constraints. The third determining module is used to determine the target node based on the target action.

13. An electronic device, characterized in that, include: memory for storing computer programs; A processor, configured to implement the steps of the reinforcement learning-based load balancing method as described in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the reinforcement learning-based load balancing method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the reinforcement learning-based load balancing method as described in any one of claims 1 to 11.

Images