A queue intelligent management method based on traffic state prediction

Through the intelligent queue management method based on GRU and DRL, the network traffic status is predicted and the packet loss strategy model is constructed, which solves the problems of poor generalization ability and parameter setting in the existing technology in complex network environments, and realizes the problem of the correctness and sensitivity of the queue management algorithm in the existing technology. Dynamic adaptation of network environment and satisfaction of business needs.

CN118694717BActive Publication Date: 2024-11-08NANJING UNIV OF INFORMATION SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411164462.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-08
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

The existing active queue management algorithm has poor generalization capabilities in complex network environments, making it difficult to adapt to changes in network state, resulting in poor congestion control effect and lack of trade-offs on QoS requirements and throughput and latency performance in different service types.

Method used

Using the GRU algorithm-based traffic state prediction and deep reinforcement learning (DRL) method, an agent is formed through a policy neural network and a target neural network, monitor the network status, and use the ε-greedy strategy to make decisions and set reward functions to build a packet loss strategy model. Implement dynamic management of network traffic.

Benefits of technology

It improves the generalization and adaptability of the algorithm, can quickly respond to changes in the network environment, achieve reasonable control of burst traffic, meet business service quality requirements, and reduce packet loss rate and end-to-end delay.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118694717B_ABST
    Figure CN118694717B_ABST
Patent Text Reader

Abstract

The invention discloses a queue intelligent management method based on flow state prediction, comprising: predicting network flow state based on GRU algorithm; constructing network state tuple, forming an intelligent agent based on policy neural network and target neural network, the intelligent agent making decisions and outputting actions based on ε-greedy strategy according to the network state at the current moment; setting reward function, the intelligent agent obtaining corresponding rewards according to the actions taken, updating the network state, and forming experience samples; the intelligent agent selecting experience samples, training the policy neural network and the target neural network and updating weights, and completing queue management; the method designed by the invention realizes reasonable control of burst traffic by predicting the network flow state while improving the generalization of the algorithm.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information engineering, and in particular to a queue intelligent management method based on flow state prediction. Background Art

[0002] With the development of information technology, the number of terminals accessing the network has doubled, the network has become more heterogeneous and complex, and network traffic has shown the characteristics of instantaneous emergence. How to optimize the queue management algorithm in a complex network environment to solve the problem of overly sensitive parameter settings, adapt to the ever-changing network environment, reduce congestion, improve the generalization of the algorithm, and meet users' requirements for low network latency and high throughput is an urgent problem to be solved.

[0003] The existing queue management algorithms are mainly divided into three types: queue scale-based, link load-based, and hybrid scale-based. Among them, the queue scale-based queue management algorithm is based on the RED algorithm and measures the queue length, but it has defects such as poor stability, sensitivity to network parameter settings, and lack of theoretical basis for parameter adjustment, which leads to the problem of being unable to adapt to the time-varying network state; the queue management algorithm based on the link load scale is based on the BLUE algorithm and the AVQ algorithm to reduce the demand for cache space due to packet loss rate, but when a large number of heterogeneous business flows converge, the queue length cannot be well controlled, resulting in frequent queue overflow; the queue management algorithm based on the hybrid scale is based on fuzzy logic control. Although it has a certain degree of adaptability to the time-varying network state, it is still sensitive to parameter settings and the fixed fuzzy control rules are too simple, which makes it difficult to guarantee the differentiated service quality requirements for heterogeneous flows.

[0004] In recent years, reinforcement learning (RL) has been gradually applied to network congestion control. RL is a method of learning by trial and error. It attaches great importance to the interaction between the subject and the environment, and continuously adjusts the action selection strategy according to the feedback of the environment. It is very suitable for networks with significant time changes and emergent characteristics. However, the limitation of RL is that its model perception ability is weak and its learning ability is poor in dynamic network scenarios. Deep reinforcement learning (DRL) integrates deep learning (DL) and reinforcement learning technology, and shows significant advantages in solving complex high-dimensional problems. It has opened up a new way to solve complex network problems. However, the existing deep reinforcement learning methods only consider single or multiple link status indicators to make decisions, ignoring the differences in QoS requirements of different service types; and lack the trade-off between throughput and delay performance, and do not consider the impact of burst traffic on queue management. Summary of the invention

[0005] The purpose of the present invention is to provide a queue intelligent management method based on traffic state prediction to solve the problems that the existing active queue management algorithm has poor generalization ability, requires parameters to be configured in advance, is difficult to adapt to the constantly changing network state in a complex network environment, and leads to poor congestion control effect of the algorithm.

[0006] To achieve the above functions, the present invention designs a queue intelligent management method based on traffic state prediction. For a queue composed of data packets in a network environment, the following steps S1 to S5 are executed to complete the queue management:

[0007] Step S1: Based on the current network environment and the queue length at the current time L t Based on the GRU algorithm, after calculation by the update gate and reset gate, the predicted queue length at the next moment is output. L t+1 , complete the network traffic status prediction;

[0008] Step S2: The queue length at the next moment L t+1 , the packet loss rate at the current moment R deq and queue delay d t , which constitutes the current network status ;

[0009] Step S3: Based on the policy neural network and the target neural network, an intelligent agent is formed, and the intelligent agent monitors the current network status S t When there is a queue delay in the queue, the agent will S t , based on the ε-greedy strategy decision and output actions for the data packet a t ;

[0010] Step S4: Set the reward function according to the action output by the agent a t , the agent gets the corresponding reward r t , and update the network status S t+1 ; Based on network status S t ,action a t ,award r t , Network Status S t+1 , constitute the experience samples and store them in the experience replay pool;

[0011] Step S5: When the number of experience samples in the experience replay pool exceeds the preset minimum batch value, the agent randomly selects experience samples from the experience replay pool, trains the policy neural network and the target neural network, and updates the weights to complete the queue management.

[0012] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0013] The present invention first predicts the network traffic state through the gated recurrent unit (GRU), and uses the DQN model to build a packet loss strategy decision model, and further builds a reward function based on throughput and end-to-end delay, so that the algorithm can quickly adapt to the dynamic changes of the network environment and realize the optimal packet loss strategy that meets the service quality requirements of the business. While improving the generalization of the algorithm, this method further realizes the reasonable control of burst traffic by predicting the traffic state. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a structural diagram of a queue intelligent management method based on traffic state prediction provided according to an embodiment of the present invention;

[0015] Figure 2 is a schematic diagram of the internal structure of a GRU algorithm provided according to an embodiment of the present invention;

[0016] Figure 3 is a diagram of an action selection process of an intelligent agent provided according to an embodiment of the present invention;

[0017] Figure 4 is a network topology diagram provided according to an embodiment of the present invention;

[0018] Figure 5 is a queue length diagram of the RED algorithm under low load conditions provided by an embodiment of the present invention;

[0019] Figure 6 is a queue length diagram of the URED algorithm under low load conditions provided by an embodiment of the present invention;

[0020] Figure 7 is a queue length diagram of a TRED algorithm under a low load condition provided by an embodiment of the present invention;

[0021] Figure 8 is a queue length diagram of the HC-RED algorithm under low load conditions provided by an embodiment of the present invention;

[0022] Fig. 9 is a queue length diagram of the DRL-AQM algorithm under low load conditions provided by an embodiment of the present invention;

[0023] Fig.10 is a queue length diagram of the HC-AQM algorithm under high load conditions provided by an embodiment of the present invention;

[0024] Fig.11 is a queue length diagram of the DRL-AQM algorithm under high load conditions provided by an embodiment of the present invention;

[0025] Fig.12 is a queue length diagram of the HC-AQM algorithm under a strong burst condition provided by an embodiment of the present invention;

[0026] Fig.13 It is a queue length diagram of the DRL-AQM algorithm under a strong burst condition provided by an embodiment of the present invention;

[0027] Fig.14 is a comparison chart of packet loss rates of five algorithms provided according to an embodiment of the present invention;

[0028] Fig.15 Graphs of end-to-end average delays of five algorithms provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0030] An embodiment of the present invention provides a queue intelligent management method based on traffic state prediction ( Deep Reinforcement Learning – Active Queue Management , DRL-AQM), refer to Figure 1 , the method is divided into a traffic state prediction module based on the GRU algorithm and a deep reinforcement learning module, where the deep reinforcement learning module is divided into four parts: state space setting, action space setting, reward function setting and algorithm flow. The specific method is as follows:

[0031] For a queue composed of data packets in a network environment, the following steps S1 to S5 are executed to complete the management of the queue:

[0032] Step S1: Based on the current network environment and the queue length at the current time L t Based on the GRU algorithm, after calculation by the update gate and reset gate, the predicted queue length at the next moment is output. L t+1 , complete the network traffic status prediction;

[0033] like Figure 2 The figure shows the internal structure of the GRU algorithm. The GRU algorithm contains two important gating units: update gatez t (Update Gate) and Reset Gate R t (Reset Gate); These gating units control the flow of information and determine whether the hidden state of the current time step should be reset or updated when processing sequence data; Figure 2 middle σ represents the fully connected layer, y t Indicates the current time t Output of hidden nodes.

[0034] The specific steps of step S1 are as follows:

[0035] Step S1.1: Setting the update gate z t Controls whether new input information should be included in the hidden state at the current moment. Specifically, the update gate generates a value between 0 and 1 through a sigmoid activation function to determine how much information of the hidden state at the previous moment should be retained, and the degree to which new input information is accepted and combined with the new input information to update the hidden state. Update Gate z t As follows:

[0036] ;

[0037] in, is the weight matrix of the update gate, is the hidden state at the previous moment, To update the hidden state of the door at the last moment The weight matrix of For the current moment t The queue length is t Traffic characteristics at each moment;

[0038] Step S1.2: Set the reset gate R t The role of is to control the influence of the hidden state of the previous moment on the input information of the current moment, thereby determining whether to ignore the previous hidden state at the current moment and whether future input information will be affected by the previous hidden state. R t A sigmoid activation function generates a value between 0 and 1 to decide whether to keep or reset the previous hidden state information. R t As follows:

[0039] ;

[0040] in, To reset the gate weight matrix, To reset the last hidden state of the door The weight matrix of the GRU algorithm; if the reset gate is close to 0, it means ignoring the previous hidden state; if it is close to 1, it means retaining the previous hidden state information. This mechanism enables the GRU algorithm to better control the flow of information and the long-term and short-term weights of memory information when processing long sequence data, thereby improving the performance and effect of the GRU algorithm;

[0041] Step S1.3: The hidden state of the previous moment With the current input Connect them to form a comprehensive input vector and use the current moment to reset the gate R t The comprehensive input vector is weighted and the weighted comprehensive input vector is compared with the weight matrix Multiply, then add the bias vector , and then through the activation function The fully connected layer is calculated to obtain the candidate hidden state at the current moment ; Candidate hidden state at the current moment As follows:

[0042] ;

[0043] in, is the weight matrix, is the bias vector;

[0044] Step S1.4: Final hidden state The calculation formula is as follows:

[0045] ;

[0046] In the formula, is the candidate hidden state at the current moment, is the hidden state at the previous moment, For updating the door.

[0047] Step S2: The queue length at the next moment L t+1 , the packet loss rate at the current moment R deq and queue delay d t , which constitutes the current network status ;

[0048] Step S3: Based on the policy neural network and the target neural network, an intelligent agent is formed, and the intelligent agent monitors the current network status S tWhen there is a queue delay in the queue, the agent will S t , based on the ε-greedy strategy decision and output actions for the data packet a t ;

[0049] The policy neural network and the target neural network are multi-layer perceptrons with two hidden layers, each containing 64 neurons. ReLU activation function is deployed in each hidden layer, and the activation function is defined as , y Represents the input vector entering the neuron, and applies the softmax function in the output layer to convert the output into an action-value function, and uses the ε-greedy algorithm to select actions based on the action-value function.

[0050] In step S3, after the agent selects an action, it returns two probabilities (discard probability / retain probability) as output. When the data packet leaves the queue, the ε-greedy strategy of the DQN model is used to decide the action. The ε-greedy strategy is as follows:

[0051] ;

[0052] In the formula, a t Indicates the action for the data packet. x represents a random variable, ε represents the attenuation parameter, Indicates the current network status Q The value function is calculated by the policy neural network, where S t Indicates the current network status. a represents the actions that the agent can perform, θ represents the weight of the policy neural network; Represents actions that the agent can perform a The current network status Q Value Function The biggest move;

[0053] The actions for data packets are: a t Including discarding and retaining. If the action decided by the agent is to discard, then , if the action decided by the agent is to keep, then .

[0054] like Figure 3The figure shows the action selection process of the agent. In the ε-greedy algorithm, when the agent makes action decisions, its behavior strategy revolves around the game of exploration and development with a given probability. That is, when the agent chooses an action, it will choose between exploration and development with a given probability. In the simulation, the random variable is set x The value range of is [0,1], and the attenuation parameter ε It is used to adjust the frequency of random exploration. Its initial value is set to 0.9 and decreases as the number of simulation iterations increases. When , the agent tends to choose the action that maximizes the expected benefit; otherwise, it randomly selects the action. The significance of this strategy is to increase the probability of random exploration in the early stage to widely cover the possible state space, and then gradually increase the probability of selecting the action value maximization in the iterative process to optimize the model performance.

[0055] Attenuation Parameters ε As follows:

[0056] ;

[0057] In the formula, for ε The initial value of express ε The minimum value of is the attenuation factor, Steps is the number of iterations. Steps increase, ε The value of gradually decreases until it converges to , thus achieving a stable action selection strategy. Only when there is a queuing delay in the queue will the agent perform actions at periodic intervals. Execute actions in .

[0058] Step S4: Set the reward function according to the action output by the agent a t , the agent gets the corresponding reward r t , and update the network status S t+1 ; Based on network status S t ,action a t ,award r t , Network Status S t+1 , constitute the experience samples and store them in the experience replay pool;

[0059] After the agent takes an action, the agent Waiting for the next network statusS t+1 , and use the reward function to evaluate the selected action. The specific steps of setting the reward function in step S4 are as follows:

[0060] Step S4.1: Design a throughput reward function ; Design delay reward function , respectively as follows:

[0061] ;

[0062] ;

[0063] in, is the average network throughput in the current period, is the average network throughput in the previous period; is the expected queuing delay, is the current queuing delay, By queue length and dequeue rate Calculated; δ and μ is the scaling factor between the throughput reward function and the latency reward function; and The calculation of is as follows:

[0064] ;

[0065] ;

[0066] in, is the number of packets in the queue, is the number of dropped packets, is the queue length at the current moment. When it is 0, Also set to 0;

[0067] Step S4.2: In order to avoid the divergence of rewards and normalize them, set the clip function to reward r t Clipping to values ​​between -1 and 1. The clip function is a common mathematical function that limits the value to a specified range. The basic form is as follows:

[0068] ;

[0069] In the formula, u Represents the input value, lower is the lower limit of the clip function, upper It is the upper limit value of the clip function;

[0070] Rewards clipped by the clip function r t It is expressed as the following formula:

[0071] ;

[0072] when When r t Assign a value of -1; when When, reward r t is equal to itself; when When r t Assign a value of 1.

[0073] Step S5: When the number of experience samples in the experience replay pool exceeds the preset minimum batch value, the agent randomly selects experience samples from the experience replay pool, trains the policy neural network and the target neural network, and updates the weights to complete the queue management.

[0074] The specific steps of step S5 are as follows:

[0075] Step S5.1: Agent with network status S t ,action a t ,award r t , Network Status S t+1 Forming an experience sample , stored in the experience replay pool in the form of tuples;

[0076] Step S5.2: When the number of experience samples in the experience replay pool exceeds the preset minimum batch value, the agent randomly selects experience samples from the experience replay pool. i ;

[0077] Step S5.3: Based on the selected experience sample i , the policy neural network calculates the current experience sample i The corresponding network status Q Value Function ,in, S i , a i , θ The current experience samples i The corresponding network state, action, and weight of the policy neural network, the target neural network calculates the next experience sample i +1 corresponding to the network status of the target Q Value Function ,in,S i+1 , a , The next experience sample i +1 corresponds to the network state, the action that the agent can perform, and the weight of the target neural network;

[0078] Step S5.4: Use gradient descent to minimize the loss function to train the policy neural network and the target neural network, achieve the training goal of minimizing the temporal difference error, and update the weights of the policy neural network θ , the weights of the target neural network .

[0079] The loss function is as follows:

[0080] ;

[0081] In the formula, loss represents the loss function, r i Based on the selected experience sample i , the reward obtained by the agent, γ is the reward discount factor, E represents mathematical expectation.

[0082] When training the policy neural network and the target neural network, the Xavier initializer is used to initialize the weights of the policy neural network and the target neural network, and the Adam optimizer is used for training optimization.

[0083] The following is an application example of the present invention:

[0084] like Figure 4 The network topology diagram is shown. n represents n transmission sources, R1 and R2 represent two sink nodes, D1~D m represents m receiving sources, S n with R1, R1 with R2 and R2 with D mThe links formed between them are access links, bottleneck links and output links. In order to show the improvement of the generalization of this algorithm, a total of three topological conditions are set, namely low load, high load and strong burst. Low load refers to the low load condition of 40 sending sources; high load refers to the high load condition of 75 sending sources; strong burst refers to the traffic burst condition of [75,115] sending sources. The traffic burst condition is to start 75 sending sources at the beginning of the simulation; 40 sending sources are added at 35s, and the sending sources become 115 at this time; 40 sending sources are reduced at 65s, and the sending sources become 75 at this time. The simulation process lasts for 100s. After the network environment is built, the RED algorithm, URED algorithm, TRED algorithm, HC-AQM algorithm and DRL-AQM algorithm are implemented on the R1 queue in turn, and the performance of the algorithms is compared one by one, and the generalization of this algorithm and the processing effect of burst traffic are analyzed. The parameter settings of the RED algorithm are shown in Table 1, and the parameter settings of the DRL-AQM algorithm are shown in Table 2:

[0085]

[0086]

[0087] like Figures 5 to 9The figure shows the overall queue length of the RED algorithm, URED algorithm, TRED algorithm, HC-AQM algorithm and DRL-AQM algorithm in a low-load environment, with the background of heterogeneous link aggregation in a tactical communication network. It can be seen that the average queue length of the five algorithms is the worst under low load conditions, because it introduces a new upper threshold, allowing the buffer to receive more data packets. However, since the URED algorithm does not optimize the discard policy function, the algorithm shows a large volatility when facing dynamic changes in the network, which is not conducive to the adaptability and stable control of the network environment. Although the TRED algorithm has improved in performance compared with the URED algorithm, it still has obvious shortcomings, because its adherence to the linear packet loss strategy fails to effectively capture and respond to subtle changes in network congestion conditions, resulting in the continued instability of the average queue length. In contrast, the HC-AQM algorithm has achieved more significant results in controlling the queue length and its fluctuation range. By dividing the queue into 6 states according to the link state and proposing a reasonable discard strategy, the stability of the average queue length is improved. However, the fluctuation of the average queue length is still more obvious than that of the DRL-AQM algorithm. This is because the DRL-AQM algorithm uses the next-moment average queue length, current-moment packet loss rate, and queue delay in the queue as the state space of the agent, trains it, and selects the optimal packet loss strategy through cyclic iteration. This process not only does not require parameter configuration, but can also sensitively perceive changes in network congestion, thereby achieving rapid response and stabilizing the queue length within a low range.

[0088] like Figures 10-11 The figure shows the overall queue length of the HC-AQM algorithm and the DRL-AQM algorithm under a high-load environment. At this time, the sending source is set to 75 nodes, the network traffic scale is large, and severe congestion is prone to occur. The HC-AQM algorithm takes into account the relationship between network traffic characteristics and queue length, and controls the fluctuation of queue length to a certain extent. Based on the HC-AQM algorithm, the DRL-AQM algorithm continues to consider network traffic, uses the GRU algorithm to directly predict the average queue length, that is, the traffic state, and inputs all states into the neural network. Through the DQN algorithm, an optimal strategy is finally learned, and the optimal packet loss strategy in the current environment is continuously selected by interacting with the environment, which improves the generalization of the algorithm, so that the DRL-AQM algorithm can also make accurate decisions in time according to the network situation when facing high-load conditions.

[0089] like Figures 12-13The figure shows the overall queue length of the HC-AQM algorithm and the DRL-AQM algorithm in a strong burst environment. Under the conditions of drastic and dynamic changes in the network, especially in the case of strong burst traffic, the rapid influx of massive burst data flows further hinders the router's instant processing capabilities. The instantaneous change of the topology structure and the adjustment of link parameters make the data convergence large-scale and time-varying, making it difficult to accurately predict the direction of the network. This causes the average queue length of each comparison algorithm to fluctuate in different ways under strong burst conditions. However, it can be seen from the figure that the average queue length controlled by the DRL-AQM algorithm shows a minimization trend, and its fluctuation amplitude is also the smallest. This is because compared with the above four algorithms, the DRL-AQM algorithm can monitor the convergence state of network traffic in real time, and predict the traffic state at the next moment in real time based on the current traffic state, take actions (discard / retain) based on the current link environment state, update the state at the next moment, and feed back the reward corresponding to the action to the intelligent agent, iterate in a loop, and then obtain the optimal packet loss strategy. This optimization not only fundamentally solves the problem of sensitive parameter configuration and improves the generalization of the algorithm, but also effectively reduces the impact of burst traffic on network transmission performance, allowing data packets to reach the router stably. This demonstrates the ability of this algorithm to optimize the stability of queue length, whether in low-load, high-load or strong burst network environments.

[0090] like Fig.14 The figure shows the packet loss rates of the five algorithms under three loads. Under low load conditions, the packet loss rate of the DRL-AQM algorithm is reduced by 21.17%, 20.78%, 27.52% and 12.96% compared with the RED, URED, TRED and HC-AQM algorithms, respectively; under high load conditions, the packet loss rate of the DRL-AQM algorithm is reduced by 23.17%, 20.02%, 27.66% and 5.00% compared with the above four algorithms, respectively; under strong burst conditions, the packet loss rate of the DRL-AQM algorithm is reduced by 20.56%, 18.70%, 24.33% and 9.47% compared with the above four algorithms, respectively. From the above data, we can see that no matter in which environment or under which load condition, the packet loss rate of the DRL-AQM algorithm is the lowest among the five algorithms. This is because the algorithm dynamically adjusts the packet loss strategy according to the environmental status to improve the algorithm's response speed and accuracy to congestion, ensuring a low probability of packet loss, relatively increasing the algorithm's throughput, and reducing the probability of data being forcibly discarded when queuing in the buffer. Even in a strong burst network environment, by predicting the queue length, the algorithm can also make corresponding processing for burst traffic in advance and obtain the optimal packet loss strategy through cyclic iteration.

[0091] like Fig.15The figure shows the end-to-end average delay of the five algorithms under three load conditions. Under low load conditions, the end-to-end average delay of the DRL-AQM algorithm is reduced by 3.44%, 3.56%, 3.93% and 1.44% compared with the RED, URED, TRED and HC-AQM algorithms, respectively; under high load conditions, the end-to-end average delay of the DRL-AQM algorithm is reduced by 6.48%, 6.72%, 6.55% and 1.56% compared with the above four algorithms, respectively; under strong burst conditions, the end-to-end average delay of the DRL-AQM algorithm is reduced by 6.00%, 6.18%, 5.95% and 1.76% compared with the above four algorithms, respectively. Therefore, no matter in which environment or under which parameter conditions, the delay of the DRL-AQM algorithm is the smallest among the five algorithms. This is because the DRL-AQM algorithm can process burst traffic in a timely manner, and constantly train and interact with the link environment and traffic status. By adopting a reasonable packet discarding strategy, it can achieve precise control of the source end sending rate, thereby effectively keeping the data backlog in the buffer at a reasonable level. Based on the graphical data and its analysis, it can be clearly seen that the performance of the DRL-AQM algorithm in delay optimization has been significantly improved.

[0092] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A queue intelligent management method based on traffic state prediction, characterized in that: For a queue composed of data packets in a network environment, the following steps S1 to S5 are executed to complete the management of the queue: Step S1: Based on the current network environment and the queue length at the current time L t Based on the GRU algorithm, after calculation by the update gate and reset gate, the predicted queue length at the next moment is output. L t+1 , complete the network traffic status prediction; The specific steps of step S1 are as follows: Step S1.1: Setting the update gate z t As follows: ; in, is the weight matrix of the update gate, is the hidden state at the previous moment, To update the hidden state of the door at the last moment The weight matrix of For the current moment t The queue length; Step S1.2: Set the reset gate R t As follows: ; in, To reset the gate weight matrix, To reset the last hidden state of the door The weight matrix of Step S1.3: Calculate the candidate hidden state at the current moment As follows: ; in, is the weight matrix, is the bias vector; Step S1.4: Final hidden state The calculation formula is as follows: ; In the formula, is the candidate hidden state at the current moment, is the hidden state at the previous moment, To update the door; Step S2: The queue length at the next moment L t+1 , the packet loss rate at the current moment R deq and queue delay d t , which constitutes the current network status ; Step S3: Based on the policy neural network and the target neural network, an intelligent agent is formed, and the intelligent agent monitors the current network status S t When there is a queue delay in the queue, the agent will S t , based on the ε-greedy strategy decision and output actions for the data packet a t ; The ε-greedy strategy in step S3 is as follows: ; In the formula, a t Indicates the action for the data packet. x represents a random variable, ε represents the attenuation parameter, Indicates the current network status Q The value function is calculated by the policy neural network, where S t Indicates the current network status. a represents the actions that the agent can perform, θ represents the weight of the policy neural network; Represents actions that the agent can perform a The current network status Q Value Function The biggest move; Attenuation Parameters ε As follows: ; In the formula, for ε The initial value of express ε The minimum value of is the attenuation factor, Steps is the number of iterations; Step S4: Set the reward function according to the action output by the agent a t , the agent gets the corresponding reward r t , and update the network status S t+1 ; Based on network status S t ,action a t ,award r t , Network Status S t+1 , constitute the experience samples and store them in the experience replay pool; The specific steps for setting the reward function in step S4 are as follows: Step S4.1: Design a throughput reward function ; Design delay reward function , respectively as follows: ; ; in, is the average network throughput in the current period, is the average network throughput in the previous period; is the expected queuing delay, is the current queuing delay, By queue length and dequeue rate Calculated; δ and μ is the scaling factor between the throughput reward function and the latency reward function; and The calculation of is as follows: ; ; in, is the number of packets in the queue, is the number of dropped packets, L t is the queue length at the current moment. When it is 0, Also set to 0; Step S4.2: Set the clip function to reward r t Clipped to values ​​between -1 and 1, expressed as: ; when When r t Assign a value of -1; when When, reward r t is equal to itself; when When r t Assign a value of 1; Step S5: When the number of experience samples in the experience replay pool exceeds the preset minimum batch value, the agent randomly selects experience samples from the experience replay pool, trains the policy neural network and the target neural network, and updates the weights to complete the queue management.

2. According to claim 1, a queue intelligent management method based on traffic state prediction is characterized in that: Action a for the data packet in step S3 t Including discarding and retaining. If the action decided by the agent is to discard, then , if the action decided by the agent is to keep, then .

3. The method for intelligent queue management based on traffic state prediction according to claim 1, characterized in that: The specific steps of step S5 are as follows: Step S5.1: Agent with network status S t ,action a t ,award r t , Network Status S t+1 Forming an experience sample , stored in the experience replay pool in the form of tuples; Step S5.2: When the number of experience samples in the experience replay pool exceeds the preset minimum batch value, the agent randomly selects experience samples from the experience replay pool. i ; Step S5.3: Based on the selected experience sample i , the policy neural network calculates the current experience sample i The corresponding network status Q Value Function ,in, S i , a i , θ The current experience samples i The corresponding network state, action, and weight of the policy neural network, the target neural network calculates the next experience sample i +1 corresponding to the network status of the target Q Value Function ,in, S i+1 , a , The next experience sample i +1 corresponds to the network state, the action that the agent can perform, and the weight of the target neural network; Step S5.4: Train the policy neural network and target neural network by minimizing the loss function using gradient descent, and update the weights of the policy neural network θ , the weights of the target neural network .

4. The method for intelligent queue management based on traffic state prediction according to claim 3 is characterized in that: The loss function in step S5.4 is as follows: ; In the formula, loss represents the loss function, r i Based on the selected experience sample i , the reward obtained by the agent, γ is the reward discount factor, E represents mathematical expectation.

5. The method for intelligent queue management based on traffic state prediction according to claim 3 is characterized in that: When training the policy neural network and the target neural network in step S5.4, the weights of the policy neural network and the target neural network are initialized using the Xavier initializer, and the Adam optimizer is used for training optimization.

6. The method for intelligent queue management based on traffic state prediction according to claim 1, characterized in that: The policy neural network and the target neural network are multi-layer perceptrons with two hidden layers, each containing 64 neurons. ReLU activation function is deployed in each hidden layer, and the activation function is defined as , y represents the input vector entering the neuron, and the softmax function is applied in the output layer to convert the output into an action value function, and the ε-greedy algorithm is used to select actions based on the action value function.