A load balancing method based on reinforcement learning and game theory

Abstract

The application discloses a load balancing method based on reinforcement learning and game theory, comprising the following steps: collecting the global state and load characteristics of a Kubernetes cluster where a Serverless vector database is located, constructing a Markov decision process based on the global state and load characteristics, defining a state space containing cluster state and request characteristics, an action space containing request node allocation operations, and a reward function containing a delay term, a load balancing term and a resource utilization term; dynamically adjusting the hyperparameters of the DPPO algorithm through an evolutionary game model; generating a load balancing scheduling strategy based on the adjusted DPPO algorithm, distributing task requests to target nodes, regularly collecting performance indicators such as average load standard deviation and average response time to calculate the revenue value, and updating the evolutionary game strategy distribution and the DPPO model parameters. The application realizes the dynamic adjustment of the reinforcement learning hyperparameters and optimizes the allocation strategy of the requests to the cluster nodes by fusing the reinforcement learning and the evolutionary game theory.

Description

Technical Field

[0001] This invention relates to the technical fields of artificial intelligence and big data services, and in particular to a load balancing method based on reinforcement learning and game theory. Background Technology

[0002] With the deep integration of serverless architecture and vector databases, vector databases are increasingly widely used in recommendation systems, semantic retrieval, and other fields. They need to handle both computationally intensive tasks (such as high-dimensional vector similarity matching) and I / O-intensive tasks (such as batch data insertion), resulting in significant diversity and volatility in load characteristics. Load balancing, as a core optimization direction for serverless vector databases, aims to rationally distribute dynamic requests across cluster nodes, avoiding single-node overload while maximizing resource utilization and system throughput. Unlike automatic scaling strategies on the resource supply side, load balancing focuses on controlling request allocation. It needs to accurately adapt to the different resource requirements of various tasks, as well as complex load scenarios such as tidal and bursty loads. This places stringent requirements on the dynamic adaptability and decision-making accuracy of the strategy.

[0003] Current load balancing strategies for serverless vector databases suffer from significant technical bottlenecks. Traditional static strategies (such as round-robin and least connections) allocate requests based on a single metric, failing to differentiate the resource consumption characteristics of different loads, which can easily lead to resource waste or local node overload. While dynamic strategies based on reinforcement learning (RL) can learn optimal decisions through environmental interaction, model performance is highly dependent on the configuration of hyperparameters such as learning rate, discount factor, and exploration rate. Traditional static hyperparameter settings struggle to adapt to real-time changes in load characteristics, and in scenarios with periodic load fluctuations or sudden request peaks, problems such as slow training convergence, policy performance oscillations, and even getting stuck in local optima can easily occur, making it impossible to maintain stable service quality in the long term.

[0004] Existing hyperparameter optimization methods also have limitations. Offline hyperparameter tuning methods such as grid search require pre-defined candidate parameter spaces and lack adaptive feedback to load changes during training, making it difficult to cope with the non-stationarity of load in serverless environments. Manual hyperparameter tuning relies on experience and is inefficient, unable to quickly respond to load fluctuations. In addition, the unique factors of vector databases, such as inter-node communication overhead and data locality, further exacerbate the complexity of load balancing. Existing strategies do not fully integrate these characteristics, resulting in insufficient matching between request allocation decisions and actual resource status, making it difficult to balance multi-dimensional optimization objectives such as latency, throughput, and load balancing.

[0005] In summary, existing load balancing strategies have shortcomings in terms of dynamic load adaptability, hyperparameter optimization flexibility, and multi-objective balancing capabilities, failing to meet the high-performance requirements of serverless vector databases in complex scenarios. Therefore, developing a load balancing solution that can dynamically adjust reinforcement learning hyperparameters, accurately adapt to diverse load characteristics, and consider multi-dimensional performance indicators has become crucial to overcoming current technical bottlenecks. Summary of the Invention

[0006] The main objective of this invention is to address the problem that traditional static hyperparameters in serverless vector database load balancing are difficult to adapt to dynamic loads, leading to unstable strategy performance. By integrating reinforcement learning and evolutionary game theory, this invention enables dynamic adjustment of reinforcement learning hyperparameters, optimizes the allocation strategy of requests to cluster nodes, thereby improving load balancing, increasing system throughput, and reducing response latency and task drop rate.

[0007] Based on the first main aspect of the present invention, a load balancing method based on reinforcement learning and game theory is provided, characterized by comprising the following steps executed by a computer system:

[0008] Collect the global status and load characteristics of the Kubernetes cluster where the Serverless vector database is located. The global status includes the CPU utilization, memory utilization, and network bandwidth utilization of each node. The load characteristics include average CPU utilization, total CPU usage, average memory utilization, total memory usage, average network utilization, total bandwidth usage, and concurrency configuration.

[0009] Based on the global state and load characteristics, a Markov decision process is constructed, defining a state space containing cluster state and request characteristics, an action space containing request node allocation operations, and a reward function containing latency, load balancing, and resource utilization.

[0010] The hyperparameters of the reinforcement learning algorithm are dynamically adjusted using an evolutionary game model. These hyperparameters include the learning rate, discount factor, exploration rate, and policy update frequency. The reinforcement learning algorithm is the DPPO algorithm.

[0011] Based on the adjusted DPPO algorithm, a load balancing scheduling strategy is generated to allocate task requests initiated by users or business systems to the Serverless vector database to target nodes. Performance indicators such as average load standard deviation and average response time are collected periodically to calculate the revenue value and update the evolutionary game strategy distribution and DPPO model parameters.

[0012] Optionally, in the aforementioned method, the reward function is specifically:

[0013]

[0014] in, For the first Total reward value for the time step; For delayed terms; For load balancing; For resource utilization items;

[0015] and, , , Rewards and penalties will be given based on the delay in request response; It is an exponential function; It is a fixed coefficient; For the first The load variation coefficient at each time step; For the first The average resource utilization of the cluster at each time step; To achieve optimal resource utilization; Tolerate bandwidth for resource utilization.

[0016] Optionally, in the aforementioned method, the process of dynamically adjusting the hyperparameters of the reinforcement learning algorithm by the evolutionary game model is as follows:

[0017] First, treat the candidate hyperparameter values ​​as game strategies and initialize the strategy probabilities;

[0018] Then, calculate the policy reward based on the DPPO training performance metrics, and update the policy distribution by replicating the following dynamic equation:

[0019]

[0020] in, For the first Time step, the first hyperparameter The rate of change in the proportion of each candidate strategy; For the first Time step, the first hyperparameter The current percentage of each candidate strategy; For the first Time step, the first hyperparameter The individual payoff value of each candidate strategy; For the first Time step, the average payoff of the evolutionary game group;

[0021] Finally press Smoothly update hyperparameters, For the first The final set of hyperparameters for each time step; For smoothing coefficients; No. The expected value of the time step hyperparameter, i.e. the expected value of the optimal policy; For the first The historical hyperparameter set for time steps.

[0022] Optionally, in the aforementioned method, the hyperparameter values ​​range as follows: learning rate 0.001-0.005, discount factor 0.8-0.99, exploration rate 0.005-0.3, and policy update frequency 10-100 steps;

[0023] The training parameters for the DPPO algorithm are: Actor learning rate 3e-3, Critic learning rate 2e-3, discount factor 0.9, GAE factor 0.80, Clip function parameter 0.2, training steps 1000, sampling size 8, and Mini Batch 5.

[0024] Optionally, in the aforementioned method, the action space includes the following three categories:

[0025] Discrete action space: Directly select a single target node;

[0026] Continuous action space: output node assignment probability vector;

[0027] Multi-level action space: including node selection, routing strategy selection and batch processing decision, wherein the routing strategies include minimum latency priority and minimum queue priority.

[0028] Based on a second key aspect of the present invention, a load balancing device based on reinforcement learning and game theory is provided, characterized in that it comprises the following functional module assembly integrated into the computer system of the Kubernetes cluster where the Serverless vector database resides:

[0029] The data acquisition module is used to collect the global status of the Kubernetes cluster, including the CPU / memory / network bandwidth utilization of each node, as well as the load characteristics, including the utilization and total amount of CPU, memory, and network, and the concurrency configuration.

[0030] The MDP building module is used to build MDPs, defining the state space, the action space containing request allocation operations, and the reward function that integrates latency, load balancing, and resource utilization.

[0031] The hyperparameter dynamic adjustment module is used to dynamically optimize the learning rate, discount factor, exploration rate and policy update frequency of the DPPO algorithm through an evolutionary game model.

[0032] The scheduling and execution module is used to generate scheduling strategies based on the optimized DPPO algorithm and allocate task requests initiated by users or business systems to the Serverless vector database to target nodes.

[0033] The strategy update module is used to collect performance indicators, calculate payoff values, and update the evolutionary game strategy distribution and DPPO model parameters.

[0034] Optionally, in the aforementioned apparatus, the reward function of the policy update module adopts the following mathematical expression:

[0035]

[0036] in, For the first Time step, number Individual hyperparameter strategies The overall return value; For the first The actual throughput of the system at each time step; This serves as the baseline value for throughput normalization. For the first The system's average response time at each time step; This is the baseline value for delayed normalization; , , These are the throughput weighting coefficient, the latency weighting coefficient, and the stability weighting coefficient, respectively. , , ; For the first Time step, number Individual hyperparameter strategies The convergence stability index, The negative value of the variance is rewarded.

[0037] Optionally, in the aforementioned apparatus, the scheduling strategy satisfies the following feasibility constraints:

[0038] Task resource requirement vector No more than the remaining capacity of the node ,in, This is the resource requirement vector for the tasks to be assigned. This is the amount of CPU resources required for the task; This is the amount of memory resources required for this task; This refers to the amount of network bandwidth resources required for the task. It is the first The remaining capacity vector of each node; It is the first The remaining CPU capacity of each node; It is the first The remaining memory capacity of each node; It is the first The remaining network bandwidth capacity of each node;

[0039] Furthermore, softmin / softmax is used to map the node processing cost to the assignment probability.

[0040] According to a third key aspect of the present invention, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0041] The memory stores a computer program that, when executed by the processor, causes the processor to perform the aforementioned load balancing method based on reinforcement learning and game theory.

[0042] Based on a fourth key aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon that, when executed, implements the aforementioned load balancing method based on reinforcement learning and game theory.

[0043] Compared to existing technologies, this invention achieves dynamic adaptive optimization of hyperparameters through the deep integration of reinforcement learning (DPPO) and evolutionary game theory, significantly improving the convergence and stability of model training. Compared to traditional fixed hyperparameter or grid search hyperparameter tuning schemes, the evolutionary game mechanism uses a survival-of-the-fittest strategy to continuously increase the proportion of high-yield hyperparameter combinations and suppress inefficient configurations, enabling the DPPO model to maintain a reasonable balance between exploration and utilization throughout the entire training cycle.

[0044] Experimental data show that after 1000 iterations, the cumulative reward value of the model of this invention is significantly higher than that of the grid search hyperparameter tuning model, and the throughput can continue to increase in the later stage of iteration. It effectively avoids the local optima and performance bottleneck problems that static hyperparameter tuning is prone to. At the same time, the smooth update mechanism reduces the training oscillation caused by hyperparameter mutations, ensuring the robustness of policy learning.

[0045] In optimizing core load balancing metrics, this invention achieves multi-dimensional improvements in node load balancing, response latency, and task drop rate. By deeply integrating load characteristics and cluster status into MDP modeling, and combining feasibility constraints to ensure precise matching between request allocation and remaining node capacity, the load distribution across nodes becomes more balanced. The average load standard deviation of query tasks is reduced to a minimum of 0.12, far superior to 0.25 for the fixed parameter strategy. In periodic load scenarios, the average response latency is reduced by 17% compared to the fixed parameter model, and the task drop rate is reduced to 2.4%. When facing sudden loads, through rapid dynamic adjustment of hyperparameters, the system can quickly respond to load changes, reducing the average load standard deviation by 20% compared to the fixed parameter model, and decreasing the task drop rate by approximately 19.5%, effectively avoiding the dual problems of single-node overload and resource waste.

[0046] This invention possesses strong robustness in adapting to various scenarios, accurately addressing the diverse loads and dynamic fluctuations of serverless vector databases. Whether it's computationally intensive near-neighbor retrieval, I / O-intensive batch insertion, or complex join aggregation queries, the scheduling strategy of this invention can allocate requests based on load characteristics, demonstrating optimal or near-optimal performance across various tasks. For the tidal periodic loads and unpredictable burst loads common in real-world business scenarios, the evolutionary game-driven hyperparameter adjustment mechanism can quickly adapt to load change trends, pre-optimize the scheduling strategy, and ensure stable throughput during peak periods while avoiding resource idleness during low-load periods. Compared to traditional strategies, it better meets the load balancing needs of complex dynamic environments.

[0047] Furthermore, this invention achieves an optimal match between system resource utilization and Quality of Service (QoS) by using a weighted balance of throughput, latency, and model stability through a revenue function. While ensuring low latency and low dropout rate, the overall cluster resource utilization remains within a reasonable range, avoiding resource waste or performance degradation caused by excessive pursuit of a single metric. This technology is not only applicable to conventional load scenarios but also supports high-concurrency and complex query business needs, providing core technical support for the large-scale deployment and high-performance operation of serverless vector databases. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0049] Figure 1 The following is an execution flowchart of a resource scheduling method for a Serverless vector database system according to an embodiment of the present invention. Detailed Implementation

[0050] The preferred embodiments of the present invention will be described in detail below to provide a clearer understanding of the purpose, features, and advantages of the present invention. It should be understood that the following embodiments are not intended to limit the scope of the present invention, but are merely illustrative of the essential spirit of the technical solution of the present invention.

[0051] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known techniques associated with the invention may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0052] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0053] The core technical terms and their meanings that may be involved in the following embodiments are as follows:

[0054] Serverless architecture: An architecture that eliminates the need for users to manage underlying servers, allocates resources on demand, and charges based on actual usage.

[0055] Kubernetes cluster (K8s cluster): an open-source container orchestration and management platform consisting of one Master node (control plane) and multiple Worker nodes (worker nodes).

[0056] Reinforcement learning: refers to the Distributed Near-End Policy Optimization Algorithm, or DPPO algorithm, which is the core generative model of the load balancing scheduling strategy of this invention. It learns the optimal request allocation strategy by interacting with the system environment and achieves dynamic adaptation of hyperparameters by combining evolutionary game theory.

[0057] Evolutionary Game Theory (EGT): A theoretical framework for dynamically optimizing hyperparameters in reinforcement learning. It treats candidate hyperparameter values ​​as game strategies and updates the strategy distribution through a survival-of-the-fittest replication dynamic equation to reinforce high-yield parameter combinations.

[0058] Load balancing: Focusing on the control mechanism of request distribution in serverless vector databases, the core is to reasonably distribute user task requests to cluster nodes, avoid single-node overload, and balance latency, throughput and resource utilization.

[0059] Markov Decision Process (MDP): A modeling framework for load balancing problems, including state space, action space, and reward function, providing the decision-making basis for the DPPO algorithm.

[0060] Dynamic hyperparameter adjustment: The core innovative mechanism of this invention optimizes hyperparameters such as learning rate and discount factor of DPPO in real time through evolutionary game theory, avoiding the defect that static parameter tuning cannot adapt to dynamic load.

[0061] Payoff function: The core function for quantifying the merits of hyperparameter strategies in evolutionary game theory. It integrates indicators such as system throughput, response latency, and model convergence stability, and outputs payoff values ​​to guide the update of strategy distribution.

[0062] Feasibility constraints: The prerequisite for load balancing scheduling is that the task resource demand vector should not exceed the remaining capacity vector of the target node, so as to avoid the node from being overloaded or the task being dropped due to insufficient resources.

[0063] Strategy Individual ( ): The hyperparameter combination unit in evolutionary game theory. Each individual corresponds to a complete set of DPPO hyperparameter configurations and is the basic unit for payoff evaluation and strategy evolution.

[0064] like Figure 1 As shown, in one embodiment, a load balancing method based on reinforcement learning and game theory according to the present invention includes the following steps 100-400 executed by a computer system:

[0065] Step 100: Collect the global status and load characteristics of the Kubernetes cluster where the Serverless vector database is located. The global status includes the CPU utilization, memory utilization, and network bandwidth utilization of each node. The load characteristics include average CPU utilization, total CPU usage, average memory utilization, total memory usage, average network utilization, total bandwidth usage, and concurrency configuration.

[0066] Step 200: Construct a Markov decision process based on the global state and load characteristics, and define a state space containing cluster state and request characteristics, an action space containing request node allocation operations, and a reward function containing latency, load balancing, and resource utilization.

[0067] Step 300: Dynamically adjust the hyperparameters of the reinforcement learning algorithm through an evolutionary game model. The hyperparameters include the learning rate, discount factor, exploration rate, and policy update frequency. The reinforcement learning algorithm is the DPPO algorithm.

[0068] Step 400: Generate a load balancing scheduling strategy based on the adjusted DPPO algorithm, allocate task requests initiated by users or business systems to the Serverless vector database to target nodes, periodically collect performance indicators such as average load standard deviation and average response time to calculate revenue values, and update the evolutionary game strategy distribution and DPPO model parameters.

[0069] When collecting global status and load characteristics of the Kubernetes cluster hosting the Serverless vector database, the monitoring system and data collection mechanism of the Kubernetes cluster are relied upon to strictly execute global status and load characteristic collection synchronously at a fixed time granularity of 30 seconds, ensuring the real-time performance and integrity of the data. The collection tool used is Kubernetes MetricsServer. The collection covers all nodes in the cluster (1 Master node and multiple Worker nodes) and the Serverless vector database Pod instances running on the nodes. During the collection process, data is pulled in batches through the communication links between the cluster control plane and each node to avoid performance interference with node operation and database services.

[0070] Among them, the global status collection focuses on the core resource usage of each node: for each node, the CPU utilization (the ratio of the node's used CPU computing power to the total CPU computing power), memory utilization (the ratio of the node's occupied memory to the total memory), and network bandwidth utilization (the ratio of the node's current network transmission rate to the maximum bandwidth) are obtained in real time. These three indicators directly reflect the node's resource load level. After collection, they are classified and summarized by node number to form a cluster global status dataset.

[0071] Load characteristic collection revolves around the task processing characteristics of the Serverless vector database: For all running database Pod instances, the following metrics are calculated: average CPU utilization (average CPU utilization of all Pods), total CPU usage (total CPU computing power actually used by all Pods), average memory utilization (average memory utilization of all Pods), total memory usage (total memory actually used by all Pods), average network utilization (average network bandwidth utilization of all Pods), and total bandwidth usage (total bandwidth used for network transmission by all Pods). Simultaneously, the concurrency configuration parameters (the maximum number of concurrent tasks currently allowed) in the system configuration file are read. These seven metrics together constitute the load characteristic vector. After collection, all data will be associated with timestamps and corresponding business function IDs for storage, providing standardized input data for subsequent Markov Decision Process (MDP) construction.

[0072] After completing the data collection in step 100, the core of step 200 is to construct a Markov Decision Process (MDP) that conforms to the load balancing scenario based on the standardized data collected in step 100. First, it is clarified that the core logic of the MDP is based on a closed loop from the current system state to the execution of allocation actions and finally obtaining performance feedback. This allows the DPPO agent to learn the optimal load balancing strategy through interaction. All definitions are strictly related to the cluster resource state and load characteristics to avoid being divorced from the actual scheduling scenario.

[0073] In possible implementations, the definition of the state space strictly follows a two-dimensional design that considers both the cluster global state and request characteristics. Specifically, the cluster global state collected in step 100 is used... (CPU utilization, memory utilization, and network bandwidth utilization of each node) and load characteristics (Seven metrics, including average CPU utilization and total memory usage, are integrated into a unified state vector, i.e.) This state vector captures both the overall resource load level of the cluster and the resource requirements of the currently scheduled tasks, providing the agent with a comprehensive basis for decision-making.

[0074] The definition of the action space needs to cover the scheduling requirements of different load scenarios. In implementation, the basic discrete action space is first provided: directly output the number of a single target node (such as node 1 to node N) to realize the single-point allocation of requests.

[0075] For scenarios with large load fluctuations, a continuous action space is enabled: the output is an allocation probability vector of length N (number of nodes). ,satisfy and By using softmax mapping, the node processing cost is transformed into allocation probability, thus achieving smooth request distribution.

[0076] For complex serverless scenarios, the action space is expanded to include three sub-actions: target node selection, routing strategy selection (minimum latency priority, minimum queue priority, etc.), and batch processing decision (0 = disabled, 1 = enabled), which improves the flexibility of the strategy and the adaptability of the scenario.

[0077] The reward function quantifies the effectiveness of strategies based on a structure of delay term + load balancing term + resource utilization term. Delay term Calculated based on node response latency: Positive rewards are given when latency is below a preset threshold; otherwise, penalties are applied at a linear slope. Load balancing item. use Calculation, wherein in this embodiment, the following is set , This represents the load variation coefficient (standard deviation / mean) for each node; the higher the load balance, the closer the reward value is to 1. Resource utilization item. Using the normal distribution function Calculation, wherein in this embodiment, the following is set When the average resource utilization rate of the cluster Approaching the optimal threshold The maximum reward is obtained at the specified time. Finally, according to... Summation generates quantitative feedback that drives strategy optimization.

[0078] In most embodiments, the specific implementation of step 300 is based on the evolutionary game model as the core framework, and dynamically optimizes the four types of hyperparameters (learning rate, discount factor, exploration rate, and policy update frequency) of the DPPO algorithm, including policy initialization, payoff calculation, evolution update, and smooth parameter tuning, to ensure that the hyperparameters are adapted to the dynamic load characteristics.

[0079] Before implementation, core constraints need to be clearly defined, including the range of hyperparameter values ​​and the evolution process relying on the replication dynamic equation and smooth update mechanism to avoid hyperparameter mutations affecting model stability.

[0080] First, hyperparameter policies are initialized to construct a virtual population for evolutionary game. For each type of hyperparameter, a candidate policy set is determined: learning rate candidate values ​​are 0.001-0.005, discount factor is 0.8-0.99, exploration rate is 0.005-0.3, and policy update frequency is 10-100 steps. The candidate values ​​of different hyperparameters are then combined into multiple individual policies. (as in step) Each individual corresponds to a complete set of DPPO hyperparameter configurations. During initialization, the probability distribution of all policy individuals is uniform. ( (The total number of strategy individuals) ensures that each hyperparameter combination has an equal opportunity to be verified in the initial stage, which is consistent with the initial setting of multi-strategy competition in evolutionary games.

[0081] Next, strategy returns are calculated to provide a quantitative basis for evolutionary updates. This step follows the return function formula. The weighting coefficients are taken as follows: , , .

[0082] During implementation, each strategy individual The DPPO algorithm was used for one round of training, and the following performance metrics were collected after training: actual system throughput. Average response time Reward fluctuation variance (used to calculate stability index) );pass (Throughput baseline) and (Delay baseline value) for , After normalization, the comprehensive return value of each strategy individual is finally calculated. The higher the return, the more suitable the hyperparameter combination is for the current load scenario.

[0083] Then, a policy evolution update is performed, adjusting the policy probability distribution based on the replication dynamic equation. This step follows the dynamic equation:

[0084]

[0085] in, For the first Time step, the first hyperparameter The rate of change in the proportion of each candidate strategy; For the first Time step, the first hyperparameter The current percentage of each candidate strategy; For the first Time step, the first hyperparameter The individual payoff value of each candidate strategy; For the first Time step, the average payoff of the evolutionary game group.

[0086] When implemented, if the individual's payoff under a certain strategy... Higher than the group average return Its strategy proportion It will increase with iteration ( The probability of a strategy being adopted by the DPPO algorithm increases; conversely, the proportion of strategies with below-average returns gradually decreases, achieving a survival-of-the-fittest strategy selection. During the evolution process, the strategy distribution is recalculated after each iteration to ensure that highly adaptable hyperparameter combinations continue to dominate.

[0087] Finally, a smooth hyperparameter update is performed to generate the current hyperparameter set for the DPPO algorithm. First, based on the evolved policy distribution, the expected values ​​of various hyperparameters are calculated. ,in The updated percentage for individual strategies. This corresponds to the hyperparameter combination. Then, the smooth update formula is applied. Perform the update, where The smoothing coefficient is used to balance historical hyperparameters (which reflect past training experience) with the expected values ​​for the current period. (Adapt to the current load). This step avoids drastic fluctuations in hyperparameters, ensuring the continuity and stability of DPPO model training, and ultimately outputs the adjusted set of hyperparameters, providing an optimized algorithmic basis for the subsequent generation of load balancing scheduling strategies.

[0088] In most possible implementations, the specific implementation of step 400 relies entirely on the distributed architecture of the Kubernetes cluster, and synchronizes node status and model parameters through the parameter server (PS) mechanism to ensure the real-time performance and consistency of scheduling decisions.

[0089] First, a load balancing scheduling strategy is generated, using the adjusted DPPO algorithm as the core, combined with the MDP model defined in step 200 and the hyperparameters (learning rate, discount factor, etc.) optimized in step 300. During implementation, the DPPO agent outputs the optimal action based on the current system state vector through a neural network: if it is a discrete action space, it directly outputs the target node number. If it is a continuous action space, it maps the node processing cost to an allocation probability vector using the softmin / softmax function, achieving smooth request distribution. If it is a multi-level action space, it synchronously outputs node selection, routing strategies (minimum latency priority, etc.), and batch processing decisions.

[0090] The key constraint in step 400 is that the strategy generation must strictly satisfy the feasibility constraint: the task resource requirement vector. No more than the remaining capacity of the node ,in, This is the resource requirement vector for the tasks to be assigned. This is the amount of CPU resources required for the task; This is the amount of memory resources required for this task; This refers to the amount of network bandwidth resources required for the task. It is the first The remaining capacity vector of each node; It is the first The remaining CPU capacity of each node; It is the first The remaining memory capacity of each node; It is the first The remaining network bandwidth capacity of each node, this constraint is expressed by the node's remaining capacity vector. Task resource requirement vector The dimensional comparison ensures that the CPU, memory, and network bandwidth requirements of the task do not exceed the remaining capacity of the target node, thus avoiding node overload.

[0091] Then, task request allocation is performed, and the allocation object is the core task (query, similar nearest neighbor retrieval, batch insertion, join aggregation, etc.) initiated by the user or business system to the Serverless vector database.

[0092] During implementation, based on the generated scheduling strategy, tasks are directed to target nodes through the cluster control plane: the discrete action space directly routes tasks to designated nodes; the continuous action space randomly samples nodes according to the allocation probability vector to achieve dynamic multi-node distribution; and the multi-level action space combines routing strategy and batch processing decision to prioritize the allocation of computationally intensive tasks to nodes with sufficient CPU and prioritize the selection of nodes with sufficient memory for memory-sensitive tasks.

[0093] During the allocation process, the resource status of each node is synchronized in real time through the parameter server (PS). If a sudden increase in node load is detected, the strategy is immediately fine-tuned to prioritize the allocation of new tasks to low-load, high-yield nodes (nodes corresponding to the strategies that have a leading share in the evolutionary game) to avoid single-node overload.

[0094] When periodically collecting performance metrics such as average load standard deviation and average response time to calculate revenue, the collection granularity remains consistent with step 100 (30 seconds / time), and the collection tool still uses Kubernetes Metrics Server. Core collection metrics include: average load standard deviation (quantifying node load balancing, calculated as "standard deviation / mean"), average response time (average time from task submission to completion), task drop rate (the percentage of tasks not processed correctly), and system throughput (the number of successfully processed tasks per unit time). Simultaneously, the reward volatility variance of the DPPO model is recorded (used to calculate stability metrics). All collected data is associated with timestamps, the current hyperparameter strategy ID, and the task type for storage, providing standardized data support for subsequent revenue calculations.

[0095] The return value is calculated based on the following return function formula. During implementation, the collected throughput will be... Average response time Substituting into the formula, where (Throughput baseline value) (Latency baseline value) is set using historical best performance indicators, stability indicators Take the negative value of the reward fluctuation variance; assign weighting coefficients according to the set range. , , This ensures that the revenue value comprehensively reflects throughput, latency, and model stability. After calculation, the revenue value and the corresponding hyperparameter policy individual are... Binding serves as the core basis for updating evolutionary game strategies.

[0096] Finally, the evolutionary game strategy distribution and DPPO model parameters are updated. In the evolutionary game part, based on the calculated payoff values, the dynamic equations are replicated. Update strategy probability distribution: The proportion of high-return strategies continuously increases, while the proportion of low-return strategies gradually decreases. Then, a smoothing update formula is applied. Generate a new set of hyperparameters to ensure the continuity of updates.

[0097] DPPO model parameter updates are achieved by aggregating the global experience pool (state-action-reward data uploaded by each node), using a parameter server to synchronize gradients, and executing Actor / Critic network updates according to the adjusted hyperparameters. When the change in node policy distribution is less than a threshold, the game is considered to have converged; if a sudden load change (such as a sudden surge in requests) is detected, the evolutionary game cycle is re-triggered to ensure that the policy and model parameters always adapt to the dynamic load scenario.

[0098] Various embodiments of the systems and techniques described above in this invention can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0102] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0103] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0104] The acquisition, storage, and application of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0105] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this invention does not impose any limitations on them.

[0106] The technical terms, principles, or means related to the technical solutions of the present invention mentioned in the above embodiments, which are not described in detail above, are all well-known technologies or common practices that are known to those skilled in the art.

[0107] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A load balancing method based on reinforcement learning and game theory, characterized in that, This includes the following steps performed by the computer system: Collect the global status and load characteristics of the Kubernetes cluster where the Serverless vector database is located. The global status includes the CPU utilization, memory utilization, and network bandwidth utilization of each node. The load characteristics include average CPU utilization, total CPU usage, average memory utilization, total memory usage, average network utilization, total bandwidth usage, and concurrency configuration. Based on the global state and load characteristics, a Markov decision process is constructed, defining a state space containing cluster state and request characteristics, an action space containing request node allocation operations, and a reward function containing latency, load balancing, and resource utilization. The hyperparameters of the reinforcement learning algorithm are dynamically adjusted using an evolutionary game model. These hyperparameters include the learning rate, discount factor, exploration rate, and policy update frequency. The reinforcement learning algorithm is the DPPO algorithm. Based on the adjusted DPPO algorithm, a load balancing scheduling strategy is generated to allocate task requests initiated by users or business systems to the Serverless vector database to target nodes. The average load standard deviation and average response time are collected periodically to calculate the revenue value and update the evolutionary game strategy distribution and DPPO algorithm parameters. The reward function is specifically as follows: in, For the first Total reward value for the time step; For delayed terms; For load balancing; For resource utilization items; and, , , Rewards and penalties will be given based on the delay in request response; It is an exponential function; It is a fixed coefficient; For the first The load variation coefficient at each time step; For the first The average resource utilization of the cluster at each time step; To achieve optimal resource utilization; Tolerating bandwidth for resource utilization; The process by which the evolutionary game model dynamically adjusts the hyperparameters of the reinforcement learning algorithm is as follows: First, treat the candidate hyperparameter values ​​as game strategies and initialize the strategy probabilities; Then, calculate the policy reward based on the DPPO training performance metrics, and update the policy distribution by replicating the following dynamic equation: in, For the first Time step, the first hyperparameter The rate of change in the proportion of each candidate strategy; For the first Time step, the first hyperparameter The current percentage of each candidate strategy; For the first Time step, the first hyperparameter Individual payoff values ​​for each candidate strategy; For the first Time step, the average payoff of the evolutionary game group; Finally press Smoothly update hyperparameters, For the first The final set of hyperparameters for the time step; For smoothing coefficients; No. The expected value of the time step hyperparameter, i.e. the expected value of the optimal policy; For the first The historical hyperparameter set of time steps; The action space includes the following three categories: Discrete action space: Directly select a single target node; Continuous action space: output node assignment probability vector; Multi-level action space: including node selection, routing strategy selection and batch processing decision, wherein the routing strategies include minimum latency priority and minimum queue priority.

2. The load balancing method based on reinforcement learning and game theory according to claim 1, characterized in that, The hyperparameter values ​​are: learning rate 0.001-0.005, discount factor 0.8-0.99, exploration rate 0.005-0.3, and policy update frequency 10-100 steps. The training parameters for the DPPO algorithm are: Actor learning rate 3e-3, Critic learning rate 2e-3, discount factor 0.9, GAE factor 0.80, Clip function parameter 0.2, training steps 1000, sampling size 8, and Mini Batch 5.

3. A load balancing device based on reinforcement learning and game theory for implementing the method of claim 1, characterized in that, This includes the following functional modules integrated into the computer system of the Kubernetes cluster where the Serverless vector database resides: The data acquisition module is used to collect the global status of the Kubernetes cluster, including the CPU / memory / network bandwidth utilization of each node, as well as the load characteristics, including the utilization and total amount of CPU, memory, and network, and the concurrency configuration. The MDP building module is used to build MDPs, defining the state space, the action space containing request allocation operations, and the reward function that integrates latency, load balancing, and resource utilization. The hyperparameter dynamic adjustment module is used to dynamically optimize the learning rate, discount factor, exploration rate, and policy update frequency of the DPPO algorithm through an evolutionary game model. The scheduling and execution module is used to generate scheduling strategies based on the optimized DPPO algorithm and allocate task requests initiated by users or business systems to the Serverless vector database to target nodes. The strategy update module is used to collect performance indicators, calculate payout values, and update the evolutionary game strategy distribution and DPPO algorithm parameters.

4. The load balancing device based on reinforcement learning and game theory according to claim 3, characterized in that, The profit function of the strategy update module is expressed by the following mathematical expression: in, For the first Time step, number Individual hyperparameter strategies The overall return value; For the first The actual throughput of the system at each time step; This serves as the baseline value for throughput normalization. For the first The system's average response time at each time step; This is the baseline value for delayed normalization; , , These are the throughput weighting coefficient, the latency weighting coefficient, and the stability weighting coefficient, respectively. , , ; For the first Time step, number Individual hyperparameter strategies The convergence stability index.

5. The load balancing device based on reinforcement learning and game theory according to claim 3, characterized in that, The scheduling strategy satisfies the following feasibility constraints: Task resource requirement vector No more than the remaining capacity of the node ,in, This is the resource requirement vector for the tasks to be assigned. This is the amount of CPU resources required for the task; This is the amount of memory resources required for this task; This refers to the amount of network bandwidth resources required for the task. It is the first The remaining capacity vector of each node; It is the first The remaining CPU capacity of each node; It is the first The remaining memory capacity of each node; It is the first The remaining network bandwidth capacity of each node; Furthermore, softmin / softmax is used to map the node processing cost to the assignment probability.

6. An electronic device, comprising: The processor, communication interface, memory, and communication bus are characterized in that the processor, communication interface, and memory communicate with each other via the communication bus. The memory stores a computer program that, when executed by the processor, causes the processor to perform the load balancing method based on reinforcement learning and game theory as described in claim 1 or 2.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the load balancing method based on reinforcement learning and game theory as described in claim 1 or 2.

Images