A global interface dynamic throttling method based on reinforcement learning
By using a reinforcement learning-based global interface dynamic rate limiting method, and training an agent with a request simulator and deep neural network, dynamic rate limiting based on interface load and features is achieved. This solves the problem that existing rate limiting methods cannot dynamically adjust, and improves system performance and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SICHUAN COMPUTING CLOUD TECHNOLOGY CO LTD
- Filing Date
- 2025-03-14
- Publication Date
- 2026-05-19
AI Technical Summary
Existing rate limiting methods cannot dynamically adjust based on actual load and interface characteristics, resulting in insufficient utilization of system resources and a decline in user experience.
A global interface dynamic rate limiting method based on reinforcement learning is adopted. By constructing a request simulator, collecting state information, establishing an action reward function, and training an agent with a deep neural network, intelligent decision-making is achieved on whether to limit or allow the current request.
It improves system throughput and reduces request access latency, and can dynamically adjust according to interface load and characteristics to avoid system overload, thereby enhancing user experience and network security.
Smart Images
Figure CN120281718B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of interface flow control technology, specifically relating to a global interface dynamic rate limiting method based on reinforcement learning. Background Technology
[0002] In internet applications, API rate limiting is a crucial strategy. Its main purposes are as follows: First, ensuring system stability. When a large number of requests flood in within a short period, exceeding the system's capacity, it can lead to system crashes. API rate limiting effectively controls the number of requests per unit of time, preventing the system from becoming paralyzed due to overload. Second, improving user experience. Without API rate limiting, some users' requests may experience long waiting times or timeouts due to resource contention. Rate limiting ensures that user requests are processed promptly and efficiently, providing users with a smooth and stable service. Third, ensuring network security. Rate limiting can effectively defend against malicious attacks by sending massive amounts of invalid requests, thereby maintaining the security and health of the entire application ecosystem.
[0003] Existing rate limiting methods typically use static rules, which cannot dynamically limit rates based on actual load conditions and interface characteristics. To improve overall resource throughput and reduce average interface latency, there is an urgent need to propose a global dynamic rate limiting method based on reinforcement learning. Summary of the Invention
[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a global interface dynamic rate limiting method based on reinforcement learning. This solves the problem that static rules cannot perceive the interface load and implement rate limiting accordingly, thereby maximizing the throughput of the rate limiting logic and reducing request access latency.
[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a global interface dynamic rate limiting method based on reinforcement learning, comprising the following steps:
[0006] S1. Build a request simulator and send simulated requests to the agent through the request simulator;
[0007] S2. Collect state information, and make decisions based on the current state information through the intelligent agent to establish an action reward function;
[0008] S3. The agent randomly selects an action, and the action reward function obtains a reward value based on the selected action to establish the Q function;
[0009] S4. Train the Q-function using a deep neural network, and obtain the trained agent based on the optimal Q-function;
[0010] S5. Make decisions on simulated requests using a trained agent.
[0011] Furthermore: In S1, the specific method for requesting the simulator to generate a simulation request is as follows:
[0012] Capture HTTP traffic from the production environment using tcpdump, convert the HTTP traffic into a JSON file using Tshark, parse the JSON file, and create simulated requests.
[0013] Furthermore, in S2, the specific method for collecting status information is as follows:
[0014] Obtain status characteristic information, including CPU utilization, memory utilization, disk utilization, network bandwidth utilization, stack space utilization, the set of interfaces to be accessed, and the average response time in milliseconds for all interfaces within a preset time interval;
[0015] The state feature information is converted into a vector, and the values of the vector are regularized to obtain the state information.
[0016] Furthermore: In S2, the expression for establishing the action reward function R is specifically as follows:
[0017]
[0018] In the formula, n is the time interval Δ between the last calculation and the current time. t The number of simulated requests within, r i Let be the reward value for the i-th simulated request;
[0019]
[0020] In the formula, res(i) is the response time of each interface, and i is the Δ t The sequence number of each request is avg(t-1), where t is the time.
[0021] Furthermore: In S3, the method for updating the Q function Q(S,A) is as follows:
[0022] Q(S,A)'←Q(S,A)+α[R+γmax Q(S',A')-Q(S,A)]
[0023] In the formula, Q(S,A)' is the updated Q function, α is the first fixed value, γ is the second fixed value, S' is the next state, A' is the next action, A is the action, S is the state, including allowing passage and limiting flow, and max Q(S',A') is the value with the largest expected reward among all possible actions taken in the next state S'.
[0024] Furthermore, in S4, the method for training the Q-function using a deep neural network is as follows:
[0025] S41. Define a neural network;
[0026] S42. Obtain the state information of the Q-function in each round of training;
[0027] S43. Update the network parameters using the gradient descent algorithm based on the current Q-function.
[0028] S44. Set a termination flag for each round of training;
[0029] S45. Use cgroups to limit the amount of resources that business applications can use, and simulate scenarios with different resource amounts for training.
[0030] Furthermore: In S41, the neural network includes an evaluation network and a target network. The evaluation network and the target network have the same structure, both including:
[0031] The input layer consists of 1000 input nodes and 64 output nodes. The number of input nodes is set according to the maximum concurrency. The input to the input layer is the state vector collected in step S2.
[0032] The hidden layer consists of 64 input nodes and 64 output nodes.
[0033] The hidden layer consists of 64 input nodes and 64 output nodes.
[0034] The output layer consists of 64 input nodes and 2 output nodes. The number of output nodes is set according to the action type, which includes allowing access and limiting the flow. That is, the output layer outputs the action that the last request in the same batch of inputs needs to perform.
[0035] S43 specifically involves updating the weight parameters of the evaluation network and the target network at different frequencies.
[0036] In this study, mean squared error is used as the loss function, and gradient descent algorithm is used to update the network parameters θ.
[0037]
[0038] In the formula, L(θ) is the loss function, the training goal is to find a set of optimal parameters θ* such that L(θ) reaches its minimum value, E represents the mathematical expectation, y is the target Q value, Q(S; A; θ) is the Q value estimate of the evaluation network for action A in state S, and θ is the parameter of the evaluation network.
[0039] The beneficial effects of this invention are as follows:
[0040] (1) This invention provides a global interface dynamic rate limiting method based on reinforcement learning. The interface rate limiting logic based on reinforcement learning can enable the agent to learn whether to rate limit or allow the current request under different loads, different concurrent requests, and when there is a potential statistical regularity in the access sequence of the interface.
[0041] (2) Traditional rule-based rate limiting methods can only be applied to a single interface. This method is applied to global rate limiting, which can maximize the improvement of global throughput.
[0042] (3) Traditional rule-based rate limiting methods cannot make dynamic decisions based on system load, while this method can make dynamic decisions based on resource load.
[0043] (4) Traditional rule-based rate limiting methods cannot dynamically limit rates based on interface characteristics. This method is an end-to-end method that can learn the characteristics of the interface to make decisions.
[0044] (5) This method can learn the potential statistical patterns of interface access sequences and can prevent overload from occurring in advance when the critical point is about to be reached. Attached Figure Description
[0045] Figure 1 This is a flowchart of a global interface dynamic rate limiting method based on reinforcement learning according to the present invention. Detailed Implementation
[0046] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0047] like Figure 1 As shown, in one embodiment of the present invention, a global interface dynamic rate limiting method based on reinforcement learning includes the following steps:
[0048] S1. Build a request simulator and send simulated requests to the agent through the request simulator;
[0049] S2. Collect state information, and make decisions based on the current state information through the intelligent agent to establish an action reward function;
[0050] S3. The agent randomly selects an action, and the action reward function obtains a reward value based on the selected action to establish the Q function;
[0051] S4. Train the Q-function using a deep neural network, and obtain the trained agent based on the optimal Q-function;
[0052] S5. Make decisions on simulated requests using a trained agent.
[0053] In this embodiment, the basic idea of the present invention is that user requests to the interface generally follow certain statistical patterns. For example, if a user first accesses the registration function, they are highly likely to access the login function next. Based on this phenomenon, if multiple users make requests simultaneously, the system can dynamically decide which requests to allow based on the access pressure on the interface.
[0054] Suppose at some point, the following access sequence is about to occur:
[0055] User 1's access sequence is Interface 1 -> Interface 2, with Interface 2 putting a lot of pressure on the service and taking a long time; User 2's access sequence is Interface 3 -> Interface 4, with Interfaces 3 and 4 having less service pressure; User 3's access sequence is Interface 4 -> Interface 5, with Interfaces 4 and 5 having less service pressure.
[0056] When the current server load is about to reach the threshold and is insufficient to support 3 requests, in order to improve the overall throughput, the rate limiting logic should limit the request for user 1's interface 1, while allowing the requests for users 2 and 3 to pass. In this way, user 1's interface 2 request will not occur, and subsequent requests from users 2 and 3 can pass normally.
[0057] This invention creates an intelligent agent that can make the initial optimal decision on whether to rate limit the current request based on environmental information. This decision needs to consider the current state and possible future states.
[0058] In S1, the specific method for requesting the simulator to generate a simulated request is as follows:
[0059] Capture HTTP traffic from the production environment using tcpdump, convert the HTTP traffic into a JSON file using Tshark, parse the JSON file, and create simulated requests.
[0060] In this embodiment, the creation of the agent takes a long time from initial setup to reaching a good decision-making state, and direct online training is costly. Therefore, this invention creates an offline request simulator to simulate online requests to train the agent.
[0061] In S2, the specific method for collecting status information is as follows:
[0062] Obtain status characteristic information, including CPU utilization, memory utilization, disk utilization, network bandwidth utilization, stack space utilization, the set of interfaces to be accessed, and the average response time in milliseconds for all interfaces within a preset time interval;
[0063] The state feature information is converted into a vector, and the values of the vector are regularized. For example, if the CPU utilization is 30%, all values are multiplied by 100 and the result is 30. For URL-type values, only the prefix part is taken. A dictionary mapping is established and the values are identified by numbers to obtain the state information.
[0064] In S2, the expression for establishing the action reward function R is as follows:
[0065]
[0066] In the formula, n is the time interval Δ between the last calculation and the current time. t The number of simulated requests within, r i Let be the reward value for the i-th simulated request;
[0067]
[0068] In the formula, res(i) is the response time of each interface, and i is the Δ t The sequence number of each request is avg(t-1), where t is the time. If the interface is allowed and the response time is shorter than the previous response time, the reward value is 1; if rate limiting is applied directly, the reward value is -1.
[0069] In this embodiment, the present invention establishes an action reward function based on the idea of reinforcement learning, which is used to obtain the reward value of each action taken in the current state. The global goal is to obtain the decision sequence with the maximum cumulative reward value.
[0070] In S3, the method for updating the Q function Q(S,A) is as follows:
[0071] Q(S,A)'←Q(S,A)+α[R+γmax Q(S',A')-Q(S,A)]
[0072] In the formula, Q(S,A)' is the updated Q function, α is the first fixed value, γ is the second fixed value, S' is the next state, A' is the next action, A is the action, S is the state, including allowing passage and limiting flow, and max Q(S',A') is the value with the largest expected reward among all possible actions taken in the next state S'.
[0073] In this embodiment, during the training phase, the agent selects different actions and calculates the reward value based on the action reward function, thereby establishing the Q function. The Q function is the result of the agent's training. That is, after obtaining the Q function, the agent can determine the maximum benefit that can be obtained after taking an action based on the current state, and decide which action to execute based on the benefit.
[0074] In S4, since the input vectors are mostly continuous non-discrete data, representing the Q function with a matrix would be too sparse. Therefore, the Q function is represented using a neural network. The specific definition and training process are as follows:
[0075] S41. Define a neural network;
[0076] The input layer consists of 1000 input nodes and 64 output nodes. The number of input nodes is set according to the maximum concurrency. The input to the input layer is the state vector collected in step S2.
[0077] The hidden layer consists of 64 input nodes and 64 output nodes.
[0078] The hidden layer consists of 64 input nodes and 64 output nodes.
[0079] The output layer consists of 64 input nodes and 2 output nodes. The number of output nodes is set according to the action type, which includes allowing access and limiting the flow. That is, the output layer outputs the action that the last request in the same batch of inputs needs to perform.
[0080] S42. Obtain the state information of the Q-function in each round of training;
[0081] Training data is generated by simulating random access requests from multiple users. Specifically, in each training round, access sequences of m different users are randomly selected, and a traffic generator is used to generate simulated requests using multiple threads according to the selected concurrency and access interval, thereby generating state information.
[0082] S43. Update the network parameters using the gradient descent algorithm based on the current Q-function.
[0083] To update the Q network using parameter weights, a target network approach is used to simulate the effect of temporal differencing, with mean squared error as the loss function and gradient descent algorithm used to update the network parameters θ.
[0084] L(θ) = E[(yQ(S;A;θ)) 2 ]
[0085] In the formula, L(θ) is the loss function, the training goal is to find a set of optimal parameters θ* such that L(θ) reaches its minimum value, E represents the mathematical expectation, y is the target Q value, Q(S; A; θ) is the Q value estimate of the evaluation network for action A in state S, and θ is the parameter of the evaluation network.
[0086] In this embodiment, two networks exist simultaneously, Q (evaluation network) and Q2. π (Target network), the two networks update the weight parameters at different frequencies, i.e., Q. π Updated in each batch iteration, while Q is updated once every 5 batches (a hyperparameter that can be adjusted). π Copy to Q).
[0087] To prevent the agent from getting trapped in local optima too early, an ε-greedy strategy is used to select exploration actions. That is, the action is determined by the Q function with a probability of 1-ε, and other actions are randomly selected with a probability of ε. ε gradually decreases as the iteration progresses, meaning that the later iterations tend to use the result of the Q function.
[0088] Each action will cause a change in the state of the business program, and the allowed interfaces will have different response times. The AI will monitor the response time of all requests to calculate the reward value.
[0089] S44. Set a termination flag for each round of training;
[0090] The termination criteria for each training round are: server resource utilization of 100% (CPU / mem / stack) exceeding a preset ratio (e.g., 5%), interface response time exceeding a certain threshold (e.g., 10 seconds), iteration count (e.g., 1000 iterations), and training time (e.g., 2 hours of training).
[0091] S45. Use cgroups to limit the amount of resources that business applications can use, and simulate scenarios with different resource amounts for training.
[0092] The beneficial effects of this invention are as follows: This invention provides a global interface dynamic rate limiting method based on reinforcement learning. The interface rate limiting logic based on reinforcement learning allows the agent to learn to make intelligent decisions on whether to rate limit or allow the current request under different loads, different concurrent requests, and when there is a potential statistical regularity in the access sequence of the interface.
[0093] Traditional rule-based rate limiting methods can only be applied to a single interface, while this method can be applied to global rate limiting, thereby maximizing the improvement of global throughput.
[0094] Traditional rule-based rate limiting methods cannot make dynamic decisions based on system load, while this method can make dynamic decisions based on resource load.
[0095] Traditional rule-based rate limiting methods cannot dynamically limit rates based on interface characteristics. This method is an end-to-end approach that can learn interface characteristics to make decisions.
[0096] This method can learn the potential statistical patterns of interface access sequences, and can prevent overload from occurring in advance when the critical point is about to be reached.
[0097] In the description of this invention, it should be understood that the terms "center," "thickness," "upper," "lower," "horizontal," "top," "bottom," "inner," "outer," and "radial," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, a feature defined by "first," "second," and "third" may explicitly or implicitly include one or more of that feature.
Claims
1. A global interface dynamic rate limiting method based on reinforcement learning, characterized in that, Includes the following steps: S1. Build a request simulator and send simulated requests to the agent through the request simulator; S2. Collect state information, and make decisions based on the current state information through the intelligent agent to establish an action reward function; S3. The agent randomly selects an action, and the action reward function obtains a reward value based on the selected action to establish the Q function; S4. Train the Q-function using a deep neural network, and obtain the trained agent based on the optimal Q-function; S5. Make decisions on simulated requests using a trained agent; In S2, the specific method for collecting status information is as follows: Obtain status characteristic information, including CPU utilization, memory utilization, disk utilization, network bandwidth utilization, stack space utilization, the set of interfaces to be accessed, and the average response time in milliseconds for all interfaces within a preset time interval; The state feature information is converted into a vector, and the values of the vector are regularized to obtain the state information. In S2, establish the action reward function. R The specific expression is: In the formula, n The time interval for calculating the period from the last time to this moment. The number of simulated requests within, For the first i The reward value of a simulated request; In the formula, For the response time of each interface, i for The sequence number of each request within. This is the average response time from the previous time. t For time.
2. The global interface dynamic rate limiting method based on reinforcement learning according to claim 1, characterized in that, In S1, the specific method for requesting the simulator to generate a simulated request is as follows: Capture HTTP traffic from the production environment using tcpdump, convert the HTTP traffic into a JSON file using Tshark, parse the JSON file, and create simulated requests.
3. The global interface dynamic rate limiting method based on reinforcement learning according to claim 1, characterized in that, In S3, the Q function Q(S, A) is updated using the following method: In the formula, For the updated Q function, α As the first fixed value, γ As the second fixed value, For the next state, For the next action, A For action, S Status indicators include allowing passage and limiting traffic. For the next state The value that yields the highest expected return among all possible actions taken.
4. The global interface dynamic rate limiting method based on reinforcement learning according to claim 1, characterized in that, In S4, the specific method for training the Q-function using a deep neural network is as follows: S41. Define a neural network; S42. Obtain the state information of the Q-function in each round of training; S43. Update the network parameters using the gradient descent algorithm based on the current Q-function; S44. Set a termination flag for each round of training; S45. Use cgroups to limit the amount of resources that business applications can use, and simulate scenarios with different resource amounts for training.
5. The global interface dynamic rate limiting method based on reinforcement learning according to claim 4, characterized in that, In S41, the neural network includes an evaluation network and a target network. The evaluation network and the target network have the same structure, both including: The input layer consists of 1000 input nodes and 64 output nodes. The number of input nodes is set according to the maximum concurrency. The input to the input layer is the state vector collected in step S2. The hidden layer consists of 64 input nodes and 64 output nodes. The output layer consists of 64 input nodes and 2 output nodes. The number of output nodes is set according to the action type, which includes allowing access and limiting the flow. That is, the output layer outputs the action that the last request in the same batch of inputs needs to perform. S43 specifically involves updating the weight parameters of the evaluation network and the target network at different frequencies. The mean squared error is used as the loss function, and the gradient descent algorithm is used to update the network parameters. ; In the formula, The loss function is used to find an optimal set of parameters during training. θ , making L ( θ To obtain the minimum value, Represents the mathematical expectation. For the target Q value, To evaluate the network in state S Next action A Q-value estimation, θ To evaluate the parameters of the network.