A data queue optimization method for high-concurrency platforms

Abstract

The present invention discloses a method for optimizing data queues of a high-concurrency platform, comprising the following steps: S1, based on the RED algorithm in the active queue management AQM, a forced learning controller is introduced to establish a RLIQM controller; S2, the RLIQM controller adaptively selects appropriate parameters for optimization settings according to different network scenarios; S3, through the RLIQM controller after the optimization settings, the optimal network strategy is obtained, and the data queues of the high-concurrency platform are optimized; based on the active queue management AQM, the invention introduces a forced learning controller to establish a RLIQM controller, which can adjust the maximum discard probability maxp according to changes in the network environment, thereby alleviating the problem of RED algorithm parameter sensitivity, can adapt to network environments with different loads, and avoid network congestion.

Description

Technical Field

[0001] The invention relates to the field of information technology, and in particular to a method for optimizing data queues on a high-concurrency platform. Background Art

[0002] High concurrency is one of the factors that must be considered in the design of Internet distributed system architecture. It usually means that the system is designed to be able to process many requests in parallel at the same time. High concurrency is a conceptual logic, which means that a large number of requests appear at the same time. Some commonly used indicators related to high concurrency include response time, throughput, query rate per second (QPS), number of concurrent users, etc. High concurrency systems are widely used, such as shared products, power monitoring systems, pipeline corridor monitoring systems, etc., but the rapid growth of network data volume makes the network more prone to congestion. How to improve the performance of high concurrency systems and optimize platform data queues is a problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the present invention is to propose a high-concurrency platform data queue optimization method. Based on active queue management AQM, the high-concurrency platform data queue optimization method introduces a forced learning controller to establish an RLIQM controller. The maximum discard probability maxp can be adjusted according to changes in the network environment, thereby alleviating the RED algorithm parameter sensitivity problem, adapting to network environments with different loads, and avoiding network congestion.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A high-concurrency platform data queue optimization method comprises the following steps:

[0006] S1. Based on the RED algorithm in active queue management AQM, a forced learning controller is introduced to establish the RLIQM controller;

[0007] S2,RLIQM controller adaptively selects appropriate parameters for optimization settings according to different network scenarios;

[0008] S3. Obtain the optimal network strategy through the optimized RLIQM controller and optimize the data queue of the high-concurrency platform.

[0009] Preferably, the RLIQM controller in step S1 includes a sender, a router queue, a forced learning controller and a receiver; the data processing process of the RLIQM controller is:

[0010] S11, the sender sends the data to the router queue;

[0011] S12, as a proxy, the forced learning controller collects the current environment state s in the router queue t∈S, selects action a∈A to execute according to strategy π, and the environment state changes to s'∈S, and generates a feedback signal at the current time t;

[0012] S13, feed back the reward value r to the forced learning controller, and the controller will update the action strategy π according to the reward value r and the environment state s';

[0013] S14, through continuous iteration, the forced learning controller obtains the optimal strategy π under each state s∈S * (s)∈A.

[0014] Preferably, the optimal strategy π in step S14 * The specific acquisition process of (s)∈A is:

[0015]

[0016] Among them, γ∈[0,1) is the discount factor, and the maximum value of formula (1) is solved according to the Bellman criterion:

[0017]

[0018] Where R(s,a) represents r(s t ,a t ) expectations; s,s' (a) is the state transition probability, which represents the probability that the environment state s reaches the next state s' under the action a;

[0019] Replace Q(s,a) with Q under strategy π π (s,a) is expressed as:

[0020]

[0021] Under the optimal decision conditions, we have:

[0022]

[0023] According to formula (2) and (3), we can get:

[0024]

[0025]

[0026] The reinforcement learning algorithm based on Q learning obtains the Q value at each moment through the following iterative method:

[0027]

[0028] Among them, α∈[0,1) is the learning rate. When t→∞, if α can gradually approach 0, then Qt (s,a) will converge to the optimal value Q * (s, a), using equation (4), the forced learning controller as the agent obtains the optimal strategy π * (s)∈A.

[0029] Preferably, the specific process of step S2 is:

[0030] S21. Define state S as the set of network states s(avg, △avg), where avg is the average queue length of the buffer, and △avg is the change in the average queue length of the buffer; define action A as the set of maximum drop probability maxp, and r is the immediate reward value;

[0031] S22. In each round of learning, the forced learning controller collects the current network state s, and then configures the maximum packet loss rate maxp in combination with the Q value table value and the corresponding action selection strategy. After the network state changes, the action a(maxp) at the current moment is calculated from s(avg,△avg)→s'(avg',△avg'), and the reward value r is obtained through the action a(maxp). The Q value corresponding to the current state-action pair (s,a) is updated accordingly. The above process is subjected to multiple rounds of learning and iteration until a perfect Q value table is obtained, and the perfect Q value table is stored as a reference for subsequent strategy adjustments.

[0032] S23, quantize the state s(avg,△avg) in the forced learning controller into 14×10 levels, that is, S={s i,j}={avg i ,△avg j}, i = 1, 2, 3 ... 14, j = 1, 2, 3 ... 10, similarly, the action a(maxp) is also divided into 14 levels, that is, A = {a n}, n = 1, 2, 3 ... 13, maxp0, where maxp0 represents the maximum drop probability value, and a specific parameter configuration is designed in a hierarchical manner;

[0033] S24. Define the utility function as follows:

[0034] Utility = δlogT ave -ηlogD ave (8)

[0035] Among them, δ is the network throughput; η is the network delay; T ave is the average throughput of the system; D ave is the average system delay;

[0036] In the learning process of the RLIQM controller, the execution effect is measured by the reward value r of the state-action pair (s, a), as follows:

[0037]

[0038] According to the reward value r, the parameters δ and η are optimally set.

[0039] Preferably, the RLIQM controller in step S3 starts learning iteration each time a new packet arrives, and the specific process is:

[0040] S31, parameter initialization: Initialize the Q value table, set the discount factor γ, the initial learning rate α 0 and the initial exploration probability ε 0 ;

[0041] S32, obtaining current network state parameters: when a new packet arrives, generating the current state parameters s(avg, △avg);

[0042] S33, select the execution action strategy: according to the current state s (avg, △avg) and the Q corresponding to each action t (s,a), select an action strategy to execute according to the ε greedy strategy;

[0043] S34, obtaining the current reward value r: the router performs packet discarding according to the latest discard probability function, and obtains the reward value r for the state-action pair (s, a) according to formula (9);

[0044] S35, obtain the parameters under the new network state and update the Q value table: according to the current network state, obtain the optimal value of s' Then use equation (7) to update Q t (s,a);

[0045] S36, update parameters: after each iteration, update the learning rate α and exploration probability ε;

[0046] S37. Determine convergence: Determine whether the Q function value has converged. If not, return to re-acquire the network state parameters and re-iterate learning. If converged, use the current Q value table as the basis for state selection actions to obtain the optimal network strategy and optimize the data queue of the high-concurrency platform.

[0047] After adopting the above technical scheme, the present invention has the following advantages compared with the background technology: based on the traditional RED (Random Early Detection) algorithm, the present invention introduces a forced learning controller to establish an RLIQM controller, adopts reinforcement learning theory, and adaptively adjusts the maximum drop probability to alleviate the defect that the parameters in the original RED algorithm are too sensitive. By dynamically collecting the changing network environment parameters, performing data analysis after multiple trials and errors, and then obtaining the optimal network strategy based on the network parameters, congestion can be avoided and performance can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the present invention;

[0049] Figure 2 It is a structural block diagram of the RLIQM controller of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] See also Figure 1 to Figure 2 , a high-concurrency platform data queue optimization method, comprising the following steps:

[0052] S1. Based on the RED algorithm in active queue management AQM, a forced learning controller is introduced to establish the RLIQM controller;

[0053] The RLIQM controller in step S1 includes a sender, a router queue, a forced learning controller and a receiver; the data processing process of the RLIQM controller is:

[0054] S11, the sender sends the data to the router queue;

[0055] S12, as a proxy, the forced learning controller collects the current environment state s in the router queue t ∈S, selects action a∈A to execute according to strategy π, and the environment state changes to s'∈S, and generates a feedback signal at the current time t;

[0056] S13, feed back the reward value r to the forced learning controller, and the controller will update the action strategy π according to the reward value r and the environment state s';

[0057] S14, through continuous iteration, the forced learning controller obtains the optimal strategy π under each state s∈S * (s)∈A;

[0058] The optimal strategy π described in step S14 * The specific acquisition process of (s)∈A is:

[0059]

[0060] Among them, γ∈[0,1) is the discount factor, and the maximum value of formula (1) is solved according to the Bellman criterion:

[0061]

[0062] Where R(s,a) represents r(s t ,a t ) expectations; s,s' (a) is the state transition probability, which represents the probability that the environment state s reaches the next state s' under the action a;

[0063] Replace Q(s,a) with Q under strategy π π (s,a) is expressed as:

[0064]

[0065] Under the optimal decision conditions, we have:

[0066]

[0067] According to formula (2) and (3), we can get:

[0068]

[0069]

[0070] The reinforcement learning algorithm based on Q learning obtains the Q value at each moment through the following iterative method:

[0071]

[0072] Among them, α∈[0,1) is the learning rate. When t→∞, if α can gradually approach 0, then Q t (s,a) will converge to the optimal value Q * (s, a), using equation (4), the forced learning controller as the agent obtains the optimal strategy π * (s)∈A;

[0073] S2,RLIQM controller adaptively selects appropriate parameters for optimization settings according to different network scenarios;

[0074] The specific process of step S2 is:

[0075] S21. Define state S as the set of network states s(avg, △avg), where avg is the average queue length of the buffer, and △avg is the change in the average queue length of the buffer; define action A as the set of maximum drop probability maxp, and r is the immediate reward value;

[0076] S22. In each round of learning, the forced learning controller collects the current network state s, and then configures the maximum packet loss rate maxp in combination with the Q value table value and the corresponding action selection strategy. After the network state changes, the action a(maxp) at the current moment is calculated from s(avg,△avg)→s'(avg',△avg'), and the reward value r is obtained through the action a(maxp). The Q value corresponding to the current state-action pair (s,a) is updated accordingly. The above process is subjected to multiple rounds of learning and iteration until a perfect Q value table is obtained, and the perfect Q value table is stored as a reference for subsequent strategy adjustments.

[0077] S23, quantize the state s(avg,△avg) in the forced learning controller into 14×10 levels, that is, S={s i,j}={avg i ,△avg j}, i = 1, 2, 3 ... 14, j = 1, 2, 3 ... 10, similarly, the action a(maxp) is also divided into 14 levels, that is, A = {a n}, n = 1, 2, 3 ... 13, maxp0, where maxp0 represents the maximum drop probability value, and a specific parameter configuration is designed in a hierarchical manner;

[0078] S24. Define the utility function as follows:

[0079] Utility = δlogT ave -ηlogD ave (8)

[0080] Among them, δ is the network throughput; η is the network delay; T ave is the average throughput of the system; D ave is the average system delay;

[0081] In the learning process of the RLIQM controller, the execution effect is measured by the reward value r of the state-action pair (s, a), as follows:

[0082]

[0083] According to the reward value r, optimize the setting parameters δ and η;

[0084] S3, obtain the best network strategy through the optimized RLIQM controller, and optimize the data queue of the high-concurrency platform;

[0085] The RLIQM controller in step S3 starts learning iteration every time a new packet arrives. The specific process is:

[0086] S31, parameter initialization: Initialize the Q value table, set the discount factor γ, the initial learning rate α 0 and the initial exploration probability ε 0 ;

[0087] S32, obtaining current network state parameters: when a new packet arrives, generating the current state parameters s(avg, △avg);

[0088] S33, select the execution action strategy: according to the current state s (avg, △avg) and the Q corresponding to each action t (s,a), select an action strategy to execute according to the ε greedy strategy;

[0089] S34, obtaining the current reward value r: the router performs packet discarding according to the latest discard probability function and obtains the reward value r for the state-action pair (s, a) according to formula (9);

[0090] S35, obtain the parameters under the new network state and update the Q value table: according to the current network state, obtain the optimal value of s' Then use equation (7) to update Q t (s,a);

[0091] S36, update parameters: after each iteration, update the learning rate α and exploration probability ε;

[0092] S37. Determine convergence: Determine whether the Q function value has converged. If not, return to re-acquire the network state parameters and re-iterate learning. If converged, use the current Q value table as the basis for state selection actions to obtain the optimal network strategy and optimize the data queue of the high-concurrency platform.

[0093] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A high-concurrency platform data queue optimization method, characterized in that: The following steps are involved: S1. Based on the RED algorithm in active queue management AQM, a forced learning controller is introduced to establish the RLIQM controller; The RLIQM controller in step S1 includes a sender, a router queue, a forced learning controller and a receiver; the data processing process of the RLIQM controller is: S11, the sender sends the data to the router queue; S12, as a proxy, the forced learning controller collects the current environment state s in the router queue t ∈S, select action a∈A to execute according to strategy π, and the environment state changes to s'∈S, and generates a feedback signal at the current time t; where S is the set of network states s(avg,△avg), and A is the set of maximum drop probability maxp; S13, feed back the reward value r to the forced learning controller, and the controller will update the action strategy π according to the reward value r and the environment state s'; S14, through continuous iteration, the forced learning controller obtains the optimal strategy π under each state s∈S * (s)∈A; The optimal strategy π described in step S14 * The specific acquisition process of (s)∈A is: Among them, γ∈[0,1) is the discount factor, and the maximum value of formula (1) is solved according to the Bellman criterion: Where R(s,a) represents r(s t ,a t ) expectations; s,s' (a) is the state transition probability, which represents the probability that the environment state s reaches the next state s' under the action a; Replace Q(s,a) with Q under strategy π π (s,a) is expressed as: Under the optimal decision conditions, we have: According to formula (2) and (3), we can get: The reinforcement learning algorithm based on Q learning obtains the Q value at each moment through the following iterative method: Among them, α∈[0,1) is the learning rate. When t→∞, if α can gradually approach 0, then Q t (s,a) will converge to the optimal value Q * (s, a), using equation (4), the forced learning controller as the agent obtains the optimal strategy π * (s)∈A; S2,RLIQM controller adaptively selects appropriate parameters for optimization settings according to different network scenarios; The specific process of step S2 is: S21. Define state S as the set of network states s(avg, △avg), where avg is the average queue length of the buffer, and △avg is the change in the average queue length of the buffer; define action A as the set of maximum drop probability maxp, and r is the immediate reward value; S22. In each round of learning, the forced learning controller collects the current network state s, and then configures the maximum packet loss rate maxp in combination with the Q value table value and the corresponding action selection strategy. After the network state changes, the action a(maxp) at the current moment is calculated from s(avg,△avg)→s'(avg',△avg'), and the reward value r is obtained through the action a(maxp). The Q value corresponding to the current state-action pair (s,a) is updated accordingly. The above process is subjected to multiple rounds of learning and iteration until a perfect Q value table is obtained, and the perfect Q value table is stored as a reference for subsequent strategy adjustments. S23, quantize the state s(avg,△avg) in the forced learning controller into 14×10 levels, that is, S={s i,j }={avg i ,△avg j }, i = 1, 2, 3 ... 14, j = 1, 2, 3 ... 10, similarly, the action a (maxp) is also divided into 14 levels, that is, A = {a n }, n = 1, 2, 3 ... 13, maxp0, where maxp0 represents the maximum drop probability value, and a specific parameter configuration is designed in a hierarchical manner; S24. Define the utility function as follows: Utility=δlogT ave -ηlogD ave (8) Among them, δ is the network throughput; η is the network delay; T ave is the average throughput of the system; D ave is the average system delay; In the learning process of the RLIQM controller, the execution effect is measured by the reward value r of the state-action pair (s, a), as follows: According to the reward value r, optimize the setting parameters δ and η; S3, obtain the best network strategy through the optimized RLIQM controller, and optimize the data queue of the high-concurrency platform; The RLIQM controller in step S3 starts learning iteration every time a new packet arrives. The specific process is: S31, parameter initialization: Initialize the Q value table, set the discount factor γ, the initial learning rate α0 and the initial exploration probability ε0; S32, obtaining current network state parameters: when a new packet arrives, generating the current state parameters s(avg, △avg); S33, select the execution action strategy: according to the current state s (avg, △avg) and the Q corresponding to each action t (s,a), select an action strategy to execute according to the ε greedy strategy; S34, obtaining the current reward value r: the router performs packet discarding according to the latest discard probability function, and obtains the reward value r for the state-action pair (s, a) according to formula (9); S35, obtain the parameters under the new network state and update the Q value table: according to the current network state, obtain the optimal value of s' Then use equation (7) to update Q t (s,a); S36, update parameters: after each iteration, update the learning rate α and exploration probability ε; S37. Determine convergence: Determine whether the Q function value has converged. If not, return to re-acquire the network state parameters and re-iterate learning. If converged, use the current Q value table as the basis for state selection actions to obtain the optimal network strategy and optimize the data queue of the high-concurrency platform.

Images