Data center adaptive flow control method based on deep reinforcement learning

Abstract

The invention belongs to the field of data center network flow control, and discloses a data center adaptive flow control method based on deep reinforcement learning, which comprises the following steps: S1, selecting action space and state space parameters; s2, designing a reward function according to the action space and the state space; s3, managing switch port queues in a unified manner; s4, obtaining a state, calculating an award, updating the # imgabs0 # network, and training and optimizing a network environment; s5, if the switch port threshold value does not generate rewards in multiple time steps, it is indicated that the threshold value is matched with the current network environment, and training is paused, and S6, when the switch port data accumulation of which training is paused exceeds a limit, it is represented that flow control is possibly triggered at the moment, and training of the port threshold value is recovered. According to the invention, each port of the switch realizes adaptive flow regulation and control, the network transmission speed can be effectively improved, and the method has the advantages of fast convergence, easy deployment, strong adaptability and the like.

Description

Technical Field

[0001] The present invention belongs to the field of traffic control in data centers, and specifically relates to a data center adaptive traffic control method based on deep reinforcement learning. Background Art

[0002] Current data centers use Remote Direct Memory Access (RDMA) technology to ensure high throughput and low latency in data centers. RDMA allows computers to directly access the memory of remote systems without the involvement of the operating system, thereby significantly improving the transmission performance of data center networks. The RDMA transmission protocol relies on a lossless network environment, that is, data packets will not be lost due to network congestion, thereby avoiding additional transmission delays caused by data retransmission. In order to achieve a lossless network, the data center network uses Priority-based Flow Control (PFC) technology. PFC avoids buffer overflow by pausing data transmission on the switch port or port priority queue, thereby ensuring lossless data transmission. Specifically, when the switch queue length exceeds the PFC pause threshold, the switch sends a PFC pause frame to its upstream switch. After receiving the PFC pause frame, the upstream switch suspends the transmission of the data flow. When the switch queue length is lower than the PFC resume threshold, it sends a PFC resume frame to the upstream switch. After receiving the PFC resume frame, the upstream switch resumes the transmission of the data flow. The transmission efficiency of data in the network is closely related to the setting of the PFC pause threshold. If the PFC pause threshold is too low, data pause operations will be triggered frequently, thus affecting traffic transmission efficiency; if the PFC pause threshold is too high, the switch will be slow to respond to network status, resulting in serious queue accumulation. Existing traffic control solutions still require manual configuration of PFC thresholds. However, the real network environment is complex and changeable. Manual configuration of PFC thresholds cannot respond to emerging network applications and their sudden traffic in a timely manner, resulting in a decrease in network performance. Manual configuration is difficult to adapt to frequent changes in the network environment in a timely manner, and may lead to threshold configuration errors. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a data center adaptive traffic control method based on deep reinforcement learning, designs an adaptive dynamic traffic control threshold configuration scheme, improves network transmission performance, and ensures the reliability of threshold configuration.

[0004] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0005] The present invention is a data center adaptive flow control method based on deep reinforcement learning, and the adaptive flow control implementation specifically includes the following steps:

[0006] Step 1: Select the threshold for sending a pause frame when triggering flow control as the action space, and select the switch data transmission and reception flow and the upstream switch port pause time as the state space parameters;

[0007] Step 2: According to the action space and state space, determine the changes in the network environment before and after the action is executed, and design an effective reward function;

[0008] Step 3: Design and deploy an intelligent agent to uniformly manage the switch port queues;

[0009] Step 4: The agent interacts with the switch, calculates rewards, and updates Network, training and optimization of network environment;

[0010] Step 5: If the trained switch port threshold does not generate rewards in multiple time steps, it means that this threshold is adapted to the current network environment, and the training is suspended to reduce the switch performance overhead;

[0011] Step 6: When the accumulated data of the switch that has suspended training exceeds a certain limit, it indicates that flow control may be triggered at this time, so the training of the port threshold is resumed.

[0012] A further improvement of the present invention is that in step 1, the process of selecting action space and state space parameters includes:

[0013] Step 1.1: Calculate the remaining space in the switch buffer in real time based on the traffic flowing through the switch :

[0014]

[0015] in, is the total buffer size, is the total headroom, For the total reserved space, The number of bytes used for the shared buffer;

[0016] Step 1.2: Calculate the PFC pause frame threshold based on the remaining space in the switch buffer

[0017]

[0018] in, A shift factor for adjusting the priority-based flow control (PFC) pause threshold, whose value is ;

[0019] Step 1.3: To ensure throughput, timely recovery is required, so the PFC recovery threshold is set to

[0020]

[0021] in, is the maximum transmission unit;

[0022] Step 1.4, select As an action to dynamically adjust the PFC trigger sensitivity, set its action space to [1,2,4,8,16,32,64]. This action space can be adjusted according to the different switch buffers;

[0023] Step 1.5: Select the data flow rate of sent and received data and the number of received data packets within a time step as state parameters. The data flow rate of sent and received data reflects the traffic transmission status of the port, and the number of received data packets reflects the traffic density of the port. Select the pause time of the upstream switch within a time step as the state parameter to reflect the impact of the current switch PFC triggering. If there is no PFC triggering, the value of this state is 0.

[0024] A further improvement of the present invention is that in step 2, the process of determining the change of the network environment before and after the action is executed based on the action space and the state space and designing an effective reward function includes:

[0025] Step 2.1: If the amount of received data in a time step after an action is executed is equal to the amount of sent data or the number of received data packets is 0, and there is no PFC trigger, it means that this action has no effect on the network, the total reward is set to 0, and the next training is directly entered;

[0026] Step 2.2, set the receiving reward. When the received data packets are mostly large or the number of received data packets is small, the PFC threshold can be appropriately increased to avoid triggering PFC; when the received data packets are large and mostly small, the PFC threshold can be appropriately reduced to trigger PFC in advance to avoid causing more serious congestion. The above situation can be solved by the average size of the received data packets. reflect, The amount of data received, The number of received data packets is a large average data packet, which corresponds to a large flow and a small number of data packets; the small average data packet corresponds to a small flow and a large number of data packets. By comparing the state changes before and after the action is executed, the receiving reward function is designed:

[0027]

[0028] in, and The amount of data received before and after the action is executed. and The number of packets received before and after the action is executed. and It is two actions, front and back;

[0029] Step 2.3, set the pause reward. When there are many data packets, appropriately extending the pause time of the upstream switch is helpful to alleviate network congestion, but too long a pause time will also affect the traffic transmission efficiency of the switch port. Therefore, we use To weigh the rationality of this pause length, and combine it with the before and after action execution The change of reflects the pros and cons of the current action, and the pause reward function is designed as follows:

[0030]

[0031] in, and is the pause duration in the time step before and after the action is executed, is the pause duration within the time step;

[0032] Step 2.4, set the total reward. In order to fully balance the receiving reward and the pausing reward, set the total reward as a weighted function of the receiving reward and the pausing reward:

[0033]

[0034] in, and The weights of receiving rewards and pausing rewards in the total rewards respectively.

[0035] A further improvement of the present invention is that in step 3, the process of designing and deploying an intelligent agent that can acquire states and calculate rewards by executing actions on the switch to learn and help make the best decision to uniformly manage the switch port queue includes:

[0036] Step 3.1: Select depth The learning algorithm trains the intelligent agent and adaptively adjusts the PFC pause threshold through the intelligent agent;

[0037] Step 3.2, initialize a deep neural network as Network, used to approximate Value function, in the continuous updating and learning process, the Q-value function is used to evaluate the value of the current action and guide the selection of actions to help the agent make the best decision when adjusting the PFC pause threshold of the switch port;

[0038] Step 3.3: Initialize a target The network, its structure and The network is the same, but the parameters are updated independently to calculate the target Q value;

[0039] Step 3.4: Create an experience replay buffer to store the experience gained during the interaction between the agent and the switch, namely the state, action, reward, and next state;

[0040] Step 3.5: Set learning rate and discount factor , used for Hyperparameters of the policy’s random probability epsilon, batch size, and update frequency.

[0041] A further improvement of the present invention is that in step 4, the agent interacts with the switch, calculates the reward, and updates The process of network,training and optimizing the network environment includes:

[0042] Step 4.1: According to Strategy selection action: by probability Choose random actions to explore new states with probability Select Current Network output The action with the largest value;

[0043] Step 4.2: The agent determines the action and deploys it to the switch queue, adjusts the sensitivity of the priority-based flow control trigger, then obtains the next time step state from the switch, calculates the reward based on the action and state, and stores the current state, action, reward, and next state in the experience replay buffer;

[0044] Step 4.3: Repeat steps 4.1 and 4.2. When the experience in the experience replay buffer reaches the set number, sample a batch of experience from the experience replay buffer for training. network;

[0045] Step 4.4: Use the target Network to calculate the target The value is updated by minimizing the loss function The parameters of the network, the loss function is the target Value and current Mean square error of the values:

[0046]

[0047] Every certain number of steps, The parameters of the network are copied to the target Keep the target in the network The stability of the network, For the goal value, is the expected value, which represents the average value of all sampled state and action pairs. is the current time step state, is the action at the current time step, is the weight of the neural network, For the current Network Estimation value;

[0048] Step 4.5: As the training progresses, Gradually decay to reduce the selection ratio of random actions and gradually make the agent transition from the exploration stage to the exploitation stage.

[0049] A further improvement of the present invention is that in step 5, the process of suspending training to reduce switch performance overhead when a specific condition is met includes:

[0050] Step 5.1, set a counter to track the number of consecutive rounds with a reward of 0, so as to determine whether the current threshold has an impact on the traffic transmission of the switch port;

[0051] Step 5.2: If the reward is 0 for multiple consecutive rounds, for example, 3-5 rounds, it means that the current action does not affect the traffic transmission of the switch port. The agent is marked as idle and the training of this switch port is suspended.

[0052] Further, in step 6, the process of resuming the training of the port threshold when flow control may be triggered includes:

[0053] Step 6.1: In the idle state, the shared buffer size is checked at each time step to see if it exceeds 1 / 2 of the current threshold, which can be adjusted according to the network bandwidth;

[0054] Step 6.2: If the shared buffer size exceeds 1 / 2 of the current threshold, the training of this port is resumed.

[0055] The beneficial effects achieved by the present invention are as follows: the present invention combines flow control with deep reinforcement learning, can respond to changes in the network environment in a timely manner, and adaptively adjust the PFC pause threshold according to the network environment to reduce the flow completion time, thereby avoiding problems such as head-of-line blocking that may arise as a result, and also avoiding problems such as configuration errors caused by manual modification of the PFC threshold, thereby ensuring the efficiency and stability of data transmission in the data center network. The present invention is an adaptive flow control method based on deep reinforcement learning, which has the advantages of fast convergence, easy deployment, and strong adaptability. Therefore, the adaptive flow control method is more suitable for data center networks that are always in a changing state. Compared with the prior art, it has the following advantages:

[0056] (1) The present invention can achieve more flexible regulation of the sensitivity of the PFC pause threshold trigger to adapt to complex and changing network environments.

[0057] (2) The present invention combines deep reinforcement learning in a programmable switch, which can automatically adapt to dynamic changes in network traffic based on real-time network load conditions without any device-level modification and without human intervention, thereby simplifying the network operation of the data center.

[0058] (3) During the online training process, the present invention has a fast convergence speed and only requires a very short training time to quickly adapt to the current network traffic conditions in order to cope with various sudden traffic conditions.

[0059] (4) The present invention adopts distributed agent deployment and controls the pause and resume of the intelligent agent to alleviate the huge pressure on the switch caused by excessive training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flow chart of a data center adaptive flow control method based on deep reinforcement learning of the present invention.

[0061] Figure 2 Schematic diagram of flow completion time before and after training in an embodiment of the present invention.

[0062] Figure 3 This is a comparison chart of the PFC triggering times of each switch when the present application and the existing algorithm are congested.

[0063] Figure 4 It is a comparison diagram of the queue length changes of the congested flow when the present invention and the existing algorithm are congested.

[0064] Figure 5 It is a comparison chart of the maximum / average queue lengths during the operation of the present invention and the existing algorithm. DETAILED DESCRIPTION

[0065] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0066] like Figure 1 As shown, the present invention is a data center adaptive flow control method based on deep reinforcement learning, and the method specifically includes the following steps:

[0067] Step 1: Select the threshold for sending a pause frame when triggering flow control as the action space, and select the switch data transmission and reception flow and the upstream switch port pause time as the state space parameters.

[0068] Calculate the remaining space in the switch buffer in real time based on the traffic flowing through the switch :

[0069]

[0070] in, is the total buffer size, is the total headroom, For the total reserved space, The number of bytes used for the shared buffer.

[0071] Then, the remaining space in the switch buffer is used to calculate the threshold for triggering the PFC pause frame.

[0072]

[0073] in, A shift factor to adjust the PFC pause threshold, its value is ;

[0074] To ensure throughput, timely recovery is required. Set the PFC recovery threshold to

[0075]

[0076] in, is the maximum transmission unit.

[0077] choose As an action to dynamically adjust the sensitivity of PFC triggering, set its action space to [1,2,4,8,16,32,64]. This action space can be adjusted according to the different switch buffers. The data flow rate and the number of received packets in a time step are selected as state parameters. The data flow rate reflects the traffic transmission of the port, and the number of received packets reflects the traffic density of the port; the pause time of the upstream switch in a time step is selected as the state parameter to reflect the impact of the current switch PFC triggering. If there is no PFC triggering, the value of this state is 0.

[0078] Step 2: Based on the action space and state space, determine the changes in the network environment before and after the action is executed and design an effective reward function.

[0079] If the amount of received data within a time step after an action is executed is equal to the amount of sent data or the number of received data packets is 0, and there is no PFC triggered, it means that this action has no effect on the network, the total reward is set to 0, and the next training is directly entered. At the same time, the reward function consists of two parts: the reward for receiving data packets and the reward for pauses. The receiving reward is set as follows. When the received data packets are large or the number of received data packets is small, the PFC threshold can be appropriately increased to avoid the triggering of PFC; when the received data packets are large and mostly small, the PFC threshold can be appropriately reduced to trigger PFC in advance to avoid causing more serious congestion. The above situation can be explained by the average size of the received data packets. reflect, The amount of data received, The number of received data packets is a large average data packet, which corresponds to a large flow and a small number of data packets; the small average data packet corresponds to a small flow and a large number of data packets. By comparing the state changes before and after the action is executed, the receiving reward function is designed:

[0080]

[0081] in, and The amount of data received before and after the action is executed. and The number of packets received before and after the action is executed. and It is two actions, front and back;

[0082] The pause reward is set as follows: When there are many data packets, appropriately extending the pause time of the upstream switch is helpful to alleviate network congestion, but too long a pause time will also affect the traffic transmission efficiency of the switch port. To weigh the rationality of this pause length, and combine it with the before and after action execution The change of reflects the pros and cons of the current action, and the pause reward function is designed as follows:

[0083]

[0084] in, and The pause duration is the time step before and after the action is executed. is the pause duration within the time step;

[0085] The total reward is composed of the receiving reward and the pausing reward. In order to fully weigh the receiving reward and the pausing reward, the total reward is set as a weighted function of the receiving reward and the pausing reward:

[0086] .

[0087] in, and The weights of receiving rewards and pausing rewards in the total rewards respectively.

[0088] Step 3: Design and deploy an intelligent agent to uniformly manage the switch port queues, and help the switch achieve adaptive traffic control by building an intelligent agent.

[0089] The present invention selects depth The learning algorithm trains the agent and adaptively adjusts the PFC threshold through the agent; initializes a deep neural network as Network, used to approximate The value function initializes a target The network, its structure and The network is the same, but the parameters are updated independently to calculate the target value; create an experience replay buffer to store the agent's experience, namely state, action, reward, and next state; finally set the learning rate for the optimizer's parameter update rate, such as 0.001, discount factor , used to measure the discount rate of future rewards, such as 0.95, The epsilon of the strategy is used to control the trade-off between exploration and exploitation, such as the initial value is 1 and the minimum value is 0.01. The batch size is used to determine the number of samples sampled from the experience replay buffer each time, such as 32. The update frequency is used to specify how many time steps are used to update the target network, such as 10 steps. These hyperparameters help the agent explore and optimize the neural network in the interaction with the switch. Value Function Used to indicate a given state Take an action After that, the expected total return that may be obtained in the future. The goal of learning is to find a policy that maximizes the cumulative reward in the future by taking the action in each state. In the DQN network, a deep neural network is used to approximate Value function. During training, the neural network receives the state As input, and output for each possible action value, and then update the weights of the neural network to approximate This approximation allows the agent to learn effectively in large environments even when the state and action spaces are large.

[0090] Step 4: The agent interacts with the switch, calculates the reward, and updates Network,training optimizes the network environment.

[0091] Agents based on Strategy selection action: by probability Choose a random action to explore a new state with probability 1- Select Current Network output The agent determines the action and deploys it to the switch queue, adjusts the sensitivity of the PFC trigger, moves to the next state, observes the reward, and stores the current state, action, reward, and next state in the experience replay buffer; randomly samples a small batch of experience from the experience replay buffer for training Network; Use Target Network to calculate the target The value is updated by minimizing the loss function The parameters of the network, the loss function is the target Value and current Mean squared error (MSE) of the values:

[0092]

[0093] in, For the goal value, is the expected value, which represents the average value of all sampled state and action pairs. is the current time step state, is the action at the current time step, is the weight of the neural network, For the current Network Estimation value;

[0094] Every certain number of steps, The parameters of the network are copied to the target Keep the target in the network The stability of the network; as the training progresses, the epsilon is gradually decayed to reduce the proportion of random action selections, gradually allowing the agent to transition from the exploration stage to the exploitation stage. value is through -learning Bellman equation calculated: ,in, is the reward of the current time step, is the discount factor, is the state of the next time step, is the action for the next time step, are the old neural network weights, The maximum next step calculated for the target network value. (epsilon) is a value between 0 and 1. Larger, such as 0.9, means that the agent will have a 90% chance of randomly choosing actions and conduct more exploration; as training progresses, As it decreases, the agent will rely more on existing knowledge to choose the best action. The decrease satisfies the decay function , which is executed before each time step ends. is the minimum value, for example 0.01, is the attenuation rate, for example 0.995, The initial value can be customized.

[0095] Step 5: If the trained switch port threshold does not generate rewards in multiple time steps, it means that this threshold is adapted to the current network environment, and the training is suspended to reduce the switch performance overhead.

[0096] The agent will set a counter to track the number of consecutive rounds with a reward of 0, so as to judge the impact of the current threshold on the network environment: if the reward is 0 for 3-5 consecutive rounds, it means that the current action does not affect the traffic transmission of the switch port, and the agent will be marked as idle, and the training of this switch port will be suspended. By pausing under this condition, the training burden of the switch can be reduced, and the performance overhead of the switch can be reduced.

[0097] Step 6: When the accumulated switch data of the suspended training exceeds a certain limit, it indicates that flow control may be triggered at this time, so the training of the port threshold is resumed.

[0098] In the idle state, the shared buffer size is checked at each time step to see if it exceeds 1 / 2 of the current threshold. It can be adjusted according to the network bandwidth. If the shared buffer size exceeds 1 / 2 of the current threshold, the training of this port is resumed. This method can adjust the PFC threshold in time when the network environment changes, thereby improving network performance.

[0099] Figure 2 It is the effect diagram of the present invention, which shows that the network data transmission rate based on the adaptive PFC pause threshold designed by the present invention is significantly improved. Specifically, the topology environment used in this method is a fat tree topology, including 4 core switches, 8 aggregation layer switches, 8 access layer switches and 16 servers. The link bandwidth between switches is 400Gbps, and the link bandwidth between servers and switches is 100Gbps. The data set used for training is AliStorage, the network load is 90%, and the training traffic size is the traffic transmitted at a host bandwidth of 100Gbps for 0.1 seconds. Figure 2As shown, compared with the network using the traditional PFC threshold, the completion time of the small flow, i.e., the flow <8Kb, is reduced by 83.26%, the completion time of the medium flow, i.e., the flow between 8Kb and 125Kb, is reduced by 82.11%, and the completion time of the large flow, i.e., the flow >125Kb, is reduced by 72.94%, indicating that the use of the present invention significantly improves the overall network traffic transmission efficiency. This shows that the present invention has obvious advantages in improving network performance.

[0100] Figure 3 This is a comparison chart of the PFC triggering times of each switch when the present application and the existing algorithm are congested. Figure 3 As shown, using the traditional PFC threshold control method, the PFC trigger times fluctuate greatly, reaching a maximum of 4000-5000 times, showing obvious network congestion problems. Compared with the network using the traditional PFC threshold control, the present invention maintains the PFC trigger times of the switch at about 1000-2000 times, effectively reducing the PFC trigger times of the switch, while making the PFC trigger times more evenly distributed and less volatile, reducing congestion events in the network and improving the overall performance of the switch.

[0101] Figure 4 : is a comparison chart of the queue length changes of the congested flow of the present invention and the existing algorithm when congested. Figure 4 As shown in the figure, the queue length fluctuates violently when the traditional PFC threshold control method is used, especially when the event is near the end, the queue rises sharply and reaches a peak value of about 2.385MB, which shows that in the traditional PFC threshold control method, the network is prone to large congestion, resulting in a large number of data packets piled up in the queue, forming a network bottleneck. Compared with the traditional PFC threshold control method, the curve of this method is smoother, and the queue length is maintained at a low level for most of the time, with almost no obvious sharp rise in the queue, indicating that this method can help the switch better predict and allocate network resources, avoid the accumulation of a large number of data packets and network congestion, thereby improving the stability and performance of the switch.

[0102] Figure 5 This is a comparison chart of the maximum / average queue lengths during the operation of the present invention and the existing algorithm. Figure 5As shown in the figure, in the AliStorage dataset, the average queue length of the traditional PFC threshold control method is close to 158KB, and the maximum queue length is close to 2.385MB. The average queue length of this method is close to 69KB, and the maximum queue length is close to 1.051MB. Compared with the traditional method, the average queue length is reduced by 56%, and the maximum queue length is reduced by 56%; in the WebSearch dataset, the average queue length of the traditional PFC threshold control method is close to 641KB, and the maximum queue length is close to 2.407MB. The average queue length of this method is close to 60KB, and the maximum queue length is close to 0.809MB. Compared with the traditional method, the average queue length is reduced by 90%, and the maximum queue length is reduced by 66%. Whether it is the AliStorage dataset or the WebSearch dataset, this method significantly reduces the average and maximum lengths of the queue, effectively optimizes the queue management of different applications, reduces congestion and improves network performance.

[0103] The present invention uses deep reinforcement learning to adaptively select a suitable PFC pause threshold interval and reasonably adjust the timing of PFC triggering, thereby ensuring the network throughput and reducing the flow completion time, while also eliminating cumbersome manual control that causes lag and unnecessary errors. Compared with manually adjusting the PFC threshold, the adaptive flow control technology used in the present invention can respond to different network flows and automatically adjust the PFC threshold to ensure the network throughput and flow completion time. The solution proposed by the present invention may be more effective and practical.

[0104] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A data center adaptive traffic control method based on deep reinforcement learning, characterized by: The data center adaptive flow control method specifically comprises the following steps: Step 1: Select the threshold for sending a pause frame when triggering flow control as the action space, and select the switch data transmission and reception flow and the upstream switch port pause time as the state space parameters; Step 2: According to the action space and state space, determine the changes in the network environment before and after the action is executed, and design the reward function; Step 3: Design and deploy an intelligent agent on the switch to obtain the state and calculate the reward by executing actions to learn and help make the best decision to uniformly manage the switch port queue; Step 4: The agent interacts with the switch, calculates rewards, and updates Network, training and optimization of network environment; Step 5: If the trained switch port threshold does not generate rewards in multiple time steps, it means that this threshold is adapted to the current network environment, and the training is suspended to reduce the switch performance overhead; Step 6: When the accumulated data of the switch that has suspended training exceeds the set limit, flow control is triggered and the training of the port threshold is resumed.

2. The method for adaptive traffic control in a data center based on deep reinforcement learning according to claim 1, characterized in that: In step 1, selecting the threshold for sending a pause frame when triggering flow control as the action space, and selecting the switch data transmission and reception flow and the upstream switch port pause time as the state space parameters specifically includes the following steps: Step 1.1: Calculate the remaining space in the switch buffer in real time based on the traffic flowing through the switch : ， in, is the total buffer size, is the total headroom, For the total reserved space, The number of bytes used for the shared buffer; Step 1.2: Calculate the pause frame threshold for triggering priority-based flow control based on the remaining space in the switch buffer. : ， in, A shift factor for adjusting the priority-based flow control pause threshold; Step 1.3: Set the priority-based flow control recovery threshold to : , in, is the maximum transmission unit; Step 1.4, select as an action to dynamically adjust the sensitivity of priority-based flow control triggers; Step 1.5, select the data flow rate of transceiver and sender, the number of received data packets and the pause time of the upstream switch within a time step as the state parameters. The data flow rate of transceiver and sender reflects the traffic transmission situation of the port, the number of received data packets reflects the traffic density of the port, and the pause time of the upstream switch reflects the impact of the current switch's priority-based flow control trigger. If there is no priority-based flow control trigger, the value of this state is 0.

3. The data center adaptive traffic control method based on deep reinforcement learning according to claim 1 is characterized in that: In step 2, the changes in the network environment before and after the action is executed are determined according to the action space and the state space, and a reward function is designed, which specifically includes the following steps: Step 2.1: If the amount of received data in a time step after an action is executed is equal to the amount of sent data or the number of received data packets is 0, and there is no priority-based flow control trigger, it means that this action has no effect on the network, set the total reward to 0, and directly enter the next training; Step 2.2, set the receiving reward: when the received data packets are large and the number of received data packets is small, increase the priority-based flow control threshold to avoid the triggering of priority-based flow control; when the received data packets are large and the flow is small, reduce the priority-based flow threshold to trigger the priority-based flow control in advance to avoid causing more serious congestion. reflect, The amount of data received, For the number of received data packets, the average data packet is large, corresponding to the situation of large traffic and small data packets, and the average data packet is small, corresponding to the situation of small traffic and large data packets. Compare the state changes before and after the action is executed, and design the receiving reward function: ， in, and The amount of data received before and after the action is executed. and The number of packets received before and after the action is executed. and It is two actions, front and back; Step 2.3, set pause reward: When there are many packets, extend the pause time of the upstream switch to help alleviate network congestion. To weigh the duration of this pause, and combine it with the before and after action execution The change of reflects the pros and cons of the current action and designs the pause reward function: ， in, and The duration of the pause in the time step before and after the action is executed; is the pause duration within the time step; Step 2.

4. Set the total reward: Set the total reward as a weighted function of the received reward and the pause reward: ， in, and The weights of receiving rewards and pausing rewards in the total rewards respectively.

4. The data center adaptive traffic control method based on deep reinforcement learning according to claim 1 is characterized in that: In step 3, the intelligent agent designed and deployed on the switch acquires the state and calculates the reward by executing actions to learn, and helps make the best decision to uniformly manage the switch port queue, which specifically includes the following steps: Step 3.1: Select depth The learning algorithm trains the agent and uses the agent to adaptively adjust the priority-based flow control pause threshold; Step 3.2, initialize a deep neural network as Network, used for approximation The value function uses approximation in the continuous updating and learning process The value function evaluates the value of the current action and guides the selection of actions to help the agent make optimal decisions when adjusting the priority-based flow control pause threshold of the switch port; Step 3.3: Initialize a target Network, Target The structure of the network The network is the same, but the parameters are updated independently to calculate the target value; Step 3.4: Create an experience replay buffer to store the experience gained during the interaction between the agent and the switch, namely the state, action, reward, and next state; Step 3.5: Set learning rate and discount factor , used for Hyperparameters of the policy’s random probability, batch size, and update frequency.

5. The data center adaptive traffic control method based on deep reinforcement learning according to claim 1 is characterized in that: In step 4, the agent interacts with the switch, calculates the reward, and updates Network, training and optimizing the network environment specifically includes the following steps: Step 4.1: According to Strategy selection action: by probability Choose random actions to explore new states with probability Select Current Network output The action with the largest value; Step 4.2: The agent determines the action and deploys it to the switch queue, adjusts the sensitivity of the priority-based flow control trigger, then obtains the next time step state from the switch, calculates the reward based on the action and state, and stores the current state, action, reward, and next state in the experience replay buffer; Step 4.3: Repeat steps 4.1 and 4.

2. When the experience in the experience replay buffer reaches the set number, sample a batch of experience from the experience replay buffer for training. network; Step 4.4: Use the target Network to calculate the target The value is updated by minimizing the loss function The parameters of the network, the loss function is the target Value and current Mean square error of the values: ， Every fixed number of steps, The parameters of the network are copied to the target Keep the target in the network The stability of the network, For the goal value, is the expected value, which represents the average value of all sampled state and action pairs. is the current time step state, is the action at the current time step, is the weight of the neural network, For the current Network Estimation value; Step 4.5: As training progresses, adjust the parameters Gradually decay to reduce the selection ratio of random actions and gradually make the agent transition from the exploration stage to the exploitation stage.

6. The data center adaptive traffic control method based on deep reinforcement learning according to claim 1 is characterized in that: In step 5, if the trained switch port threshold does not generate rewards in multiple time steps, it means that the threshold is adapted to the current network environment, and the training is suspended to reduce the switch performance overhead, which specifically includes the following steps: Step 5.1, set a counter to track the number of consecutive rounds with a reward of 0, so as to determine whether the current threshold has an impact on the traffic transmission of the switch port; Step 5.2: If the reward is 0 for multiple consecutive rounds, it means that the current action does not affect the traffic transmission of the switch port. The agent is marked as idle and the training of this switch port is suspended.

7. The data center adaptive traffic control method based on deep reinforcement learning according to claim 1 is characterized in that: The step 6 specifically includes the following steps: Step 6.1: In the idle state, the shared buffer size is checked at each time step to see if it exceeds 1 / 2 of the current threshold. Step 6.2: If the shared buffer size exceeds 1 / 2 of the current threshold, the training of this port is resumed.

Images