A microburst mitigation method based on in-network reinforcement learning

Abstract

The present application relates to a kind of microburst mitigation methods based on in-network reinforcement learning, which is by deploying reinforcement learning algorithm in programmable data plane, make switch have the microburst event processing capability without the assistance of controller, compared with traditional scheme, reduce data-control plane delay, meet the microsecond level cache solution demand required by microburst.In addition, reinforcement learning model makes network element can adapt to changing network environment and new traffic pattern, so that switch can make the best microburst mitigation strategy according to real-time network state, reduce the processing burden of host end and the influence to service traffic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network, in particular to a micro-burst mitigation method based on in-network reinforcement learning. BACKGROUND

[0002] Micro-burst refers to the reception of a relatively large number of data packets within milliseconds or microseconds. Its typical characteristics are short duration (1 to 100 ms) and high instantaneous burst data rate (several tens or hundreds of times the average data rate, or even exceeding the port bandwidth). Micro-bursts are common in modern data center networks (DCN). In DCN, the network often operates at a high rate (>10 Gbps) with extremely low end-to-end delay (about 10 microseconds). Under such low latency requirements, even a small amount of data packet queue is considered unacceptable in terms of queue delay, so micro-bursts can have a significant impact. In addition, micro-bursts also frequently occur in Internet of Things, stock market, and large-scale transactions on e-commerce promotion day. For example, a large number of instantaneous trading businesses in the stock market or the inevitable multi-flow traffic in the actual network will form micro-burst traffic in a very short time.

[0003] When the micro-burst exceeds the forwarding capacity of the switch, the switch buffers the burst data for subsequent transmission. If the switch does not have enough buffer space, the excess data will be discarded, resulting in high congestion and a large number of packet losses. In addition, as the network equipment continues to run, it is inevitable that the ability to cope with burst flow will decline due to network changes, device aging, software bugs, etc., and the number of potential micro-burst events will increase, making it more likely to cause service interruption, downtime, and other serious situations.

[0004] Due to the unique characteristics of data center applications, such as high capacity and short traffic dynamic traffic patterns, micro-bursts can significantly affect application performance. Traditional congestion control algorithms are too slow to respond, making them ineffective against micro-bursts. While existing micro-burst mitigation methods have achieved some results, they still have significant limitations: First, fixed strategy and parameter configuration. The response strategy of existing work to micro-burst events is predefined, that is, relatively fixed, which makes it difficult for existing work to adapt to real-time network changes. Any modification of the strategy and parameter configuration requires redeployment. Second, lack of fine-grained analysis of network state. Existing work lacks network environment information collection, relying on local information to infer surrounding network information or collecting only some coarse-grained network information, such as directly connected queue length. Third, some work requires terminal information assistance. Some related work requires terminal hosts to monitor their transmitted traffic and embed information related to traffic burstiness and priority information into data packets, thereby assisting the host end with information. However, this assistance increases the additional burden on the host end and affects business flow.

[0005] With the development of programmable network devices, the flexibility and programmability of network devices have been greatly improved. In particular, programmable switches and routers based on P4 language provide support for deep control and customized processing capabilities of data plane, making it possible to directly deploy reinforcement learning models in network devices. Therefore, using programmable switches and combining related functions designed for data plane provides a new idea for solving the problem of micro-burst flow detection that traditional switches and traditional software solutions have not considered and are difficult to support. SUMMARY

[0006] The purpose of the present application is to provide a micro-burst mitigation method based on in-network reinforcement learning, which can optimize the forwarding decision in the data plane of a programmable switch, thereby achieving effective mitigation of micro-burst traffic in a complex and changing network.

[0007] To achieve the above purpose, the technical solution adopted by the present application is: a micro-burst mitigation method based on in-network reinforcement learning, which monitors traffic in real time, realizes rapid response through an in-network reinforcement learning model, and can select the best forwarding strategy to mitigate micro-bursts according to the network environment; specifically including the following steps:

[0008] (1) Data collection: collect network state information and adjacent switch information; the agent obtains adjacent switch information through probes sent by adjacent switches;

[0009] (2) Micro-burst detection and judgment: set queue threshold according to micro-burst definition; monitor the queue length of each port of the switch to determine the start and end of the micro-burst; if the port queue length exceeds the set threshold within the set time, it is considered that the micro-burst has started; if the port queue length is lower than the set threshold within the set time, it is considered that the micro-burst has ended;

[0010] (3) State update: when a micro-burst occurs, input the collected data into the reinforcement learning model to obtain the corresponding state information; if the state does not exist, initialize the state in the corresponding register and initialize the probability of the corresponding mitigation strategy;

[0011] (4) Action selection: according to the strategy with the highest probability given by the reinforcement learning inference, find the selection probability of each port under the current state; select the port with the highest probability as the action for this round of mitigation, set the egress port of the data packet to the corresponding port, and record the current state and action selection in the register;

[0012] (5) Reward feedback: after the micro-burst ends, calculate the reward function; obtain the feedback of this micro-burst and update the selection probability of each port under the previously recorded state; update the historical average packet loss data under the corresponding state.

[0013] Further, the in-network reinforcement learning model is deployed on the data plane to have the ability to learn the optimal mitigation strategy according to the real-time network state, so as to achieve the effect of responding to micro-burst traffic and making decisions within microseconds.

[0014] Further, by deploying the method in the network, the micro-burst mitigation strategy without the assistance of terminal hosts is realized, and the processing burden of the host end and the influence on service traffic are minimized.

[0015] Further, in step (1), the network state information includes queue packet loss, data packet byte number within 10 microseconds of each port, micro-burst size and long / short flow ratio, and the adjacent switch information includes adjacent switch port queues and load conditions.

[0016] Further, in step (1), an information collection method based on in-band network telemetry is constructed, real-time traffic characteristics are collected and analyzed, and a response strategy is made according to the traffic pattern, micro-burst size and switch port load information in the network.

[0017] Further, in step (4), three micro-burst mitigation strategies of port steering, buffer management and congestion control are adopted, different strategies can better adapt to different network environments and network demands, and the network performance is optimized to the maximum extent.

[0018] Further, the in-network reinforcement learning model implements the micro-burst mitigation strategy on the network element, relies on the characteristics of wide deployment range and large deployment scale of the network element, realizes large-scale scalability of the micro-burst mitigation system, and thus adapts to complex network environments.

[0019] Compared with the prior art, the present application has the following beneficial effects: the present application provides a micro-burst mitigation method based on in-network reinforcement learning, which deploys the reinforcement learning algorithm on the programmable data plane, enables the switch to have the micro-burst event processing capability without the assistance of the controller, reduces the data-control plane delay compared with the traditional scheme, meets the microsecond-level mitigation strategy requirement of micro-bursts. In addition, the reinforcement learning model enables the network element to adapt to the changing network environment and new traffic pattern, so that the switch can make the best micro-burst mitigation strategy according to the real-time network state, and the processing burden of the host end and the influence on service traffic are reduced. The present application has scalability and can effectively improve the micro-burst traffic monitoring and mitigation capability in large-scale network environments. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is the overall architecture diagram of the embodiment of the present application;

[0021] Figure 2 is the theoretical architecture of the micro-burst mitigation method of the embodiment of the present application;

[0022] Figure 3 The microburst mitigation method reinforcement learning model is an embodiment of the present application. DETAILED DESCRIPTION

[0023] The present application is further described below in conjunction with the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed description is merely exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0025] It is to be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments consistent with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0026] The embodiment provides a microburst mitigation method based on in-network reinforcement learning. The method constructs an information collection method based on in-band network telemetry (INT) to monitor traffic in real time, realizes rapid response through an in-network reinforcement learning (RL) model, can select a suitable switch port forwarding strategy according to a network environment, and then realizes microsecond-level microburst traffic mitigation. The method is a microburst adaptive mitigation system realized based on P4 and reinforcement learning, solves network problems caused by microbursts, takes advantage of the programmable data plane, and realizes microsecond-level decision making. First, by deploying reinforcement learning, dynamic network adaptation is realized, microburst traffic mitigation strategies are optimized in real time, and congestion, delay and packet loss are reduced. With the help of the P4 programmable data plane, the programmable switch supports Gbps to Tbps level traffic, solves the delay problem of the traditional control plane, realizes millisecond-level microburst rapid response, and reduces the delay. At the same time, the method collects information of adjacent network entities in fine granularity by deploying in-band network telemetry, and supports dynamic adaptation of decision making schemes. In addition, the method deploys the scheme in the network, autonomously realizes microburst traffic mitigation, and reduces the burden of the host and the impact of business traffic.

[0027] The specific implementation steps of the method are as follows.

[0028] (1) Data collection: Collect network state information and adjacent switch information. Network state information includes queue packet loss, number of bytes of data packets in each port within 10 microseconds, microburst size, long / short flow ratio, etc. Adjacent switch information includes adjacent switch queue, load, etc. The agent obtains adjacent switch information through probes sent by adjacent switches.

[0029] In this embodiment, an in-band network telemetry-based information collection method is constructed to collect and analyze real-time traffic characteristics and make response strategies based on traffic patterns, microburst sizes, and switch port load information in the network.

[0030] (2) Microburst detection and judgment: According to the definition of microburst, set appropriate queue threshold. Monitor the length of each port queue of the switch to determine the start and end of the microburst. If the port queue length exceeds the set threshold within the set time (100ns), it is considered that the microburst starts; if the port queue length is lower than the set threshold within the set time (100ns), it is considered that the microburst ends.

[0031] (3) State update: When a microburst occurs, input the collected data into the reinforcement learning model to obtain the corresponding state information. If the state does not exist, initialize the state in the corresponding register and initialize the probability of the corresponding mitigation strategy (such as the probability of selecting each steering port).

[0032] (4) Action selection: According to the probability of the strategy given by the reinforcement learning inference, find the selection probability of each port under the current state. Select the port with the highest selection probability as the action for this round of mitigation, set the egress port of the data packet to the corresponding port, and record the current state and action selection in the register.

[0033] In this embodiment, three microburst mitigation strategies are used: port steering, buffer management, and congestion control. Different strategies can better adapt to different network environments and network demands, and maximize network performance optimization.

[0034] (5) Reward feedback: After this microburst ends, calculate the reward function; obtain the feedback of this microburst and update the selection probability of each port under the previously recorded state; update the historical average packet loss data under the corresponding state.

[0035] The in-network reinforcement learning model is deployed on the data plane to have the ability to learn the best mitigation strategy according to real-time network state, achieving the effect of quickly responding to microburst traffic and making decisions within microseconds.

[0036] The in-network reinforcement learning model implements microburst mitigation strategies on network elements, relying on the wide deployment range and large deployment scale of network elements to achieve large-scale scalability of the microburst mitigation system, thereby adapting to complex network environments.

[0037] As Figure 1 shown, the present application proposes a micro-burst mitigation scheme with high real-time performance and high responsiveness by combining reinforcement learning model with programmable network devices. By deploying in-band network telemetry technology, fine-grained collection of information of adjacent network entities is realized, providing information support for rapid response of micro-burst traffic. By innovatively combining in-band telemetry technology with data plane reinforcement learning, the present application supports micro-burst traffic mitigation in high-throughput and high-concurrency networks, significantly improves network utilization, minimizes the processing burden of the host end and the impact on service traffic. The present application can realize real-time monitoring and microsecond-level response decision of micro-burst traffic, and provides an effective solution for large-scale expansion of micro-burst mitigation system, as follows:

[0038] 1. Data board and data plane deployment: as Figure 1 shown, the overall architecture of the present application includes two parts of data board and data plane. In the data plane, fine-grained traffic monitoring is performed through in-band telemetry technology, and a reinforcement learning model is deployed to have the ability to learn the best mitigation strategy according to the real-time network state, so as to achieve the effect of making decisions within microseconds in response to micro-burst traffic. Programmable switches execute different forwarding strategies according to different decisions, achieving the effect of micro-burst traffic mitigation.

[0039] 2. Reinforcement learning model design and training: most micro-burst scenarios can be modeled as Markov decision processes, and the reward-based learning framework and value function optimization method of reinforcement learning perform excellently in solving Markov problems. The packet loss rate in the network is used as the reward function, and the packet loss rate calculation method is: packet loss rate = packet loss amount / micro-burst size. Packet loss rate can better reflect the damage of traffic than packet loss amount, and can also reflect the mitigation effect of micro-burst traffic.

[0040] 3. Micro-burst mitigation method theoretical architecture: as Figure 2 shown, the theoretical architecture of the present application is divided into application layer, decision layer, strategy layer and data layer. The application layer explains that micro-burst mitigation is mainly applied to scenarios where multiple ends send a large amount of traffic to one end, such as distributed machine learning, network caching and other application scenarios; the decision layer explains that the system uses a reinforcement learning model to make decisions on micro-burst mitigation methods under real-time network state; the strategy layer explains that the system provides three mitigation schemes of port steering, buffer management and congestion control; the data layer explains that the system is implemented based on programmable data plane.

[0041] 4. Reinforcement learning model decision execution framework: as Figure 3As shown, the reinforcement learning agent obtains information about neighboring switches through probes sent by these switches, monitors the queue lengths on each switch port, and determines the start and end of a microburst. When a microburst occurs, the collected data is fed into the reinforcement learning model to obtain the corresponding state information. The model's strategy is used to determine the selection probability of each port in the current state and select the port with the highest probability as the mitigation action for the current round. After each microburst round, the reward function is updated.

[0042] To address the problems of frequent microbursts, difficulty in monitoring, and delayed mitigation strategies in existing large-scale networks, the present invention provides a microburst mitigation method based on in-network reinforcement learning. This method can optimize forwarding decisions in the data plane of programmable switches, achieve real-time monitoring of microbursts and microsecond-level response decisions, and effectively mitigate microbursts in complex and changing networks, thereby reducing the processing burden on the host side and the impact on service traffic.

[0043] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0045] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0046] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0047] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A microburst mitigation method based on in-network reinforcement learning, characterized in that, Real-time monitoring of traffic, through the network reinforcement learning model to achieve rapid response, can choose the best forwarding strategy to relieve micro-burst according to the network environment; Specifically, including the following steps: (1) Data collection: Collect network state information and adjacent switch information; The agent obtains the adjacent switch information through the probe sent by the adjacent switch; The adjacent switch information includes the queue and load of each port of the adjacent switch; (2) Micro-burst detection and judgment: According to the definition of micro-burst, set the queue threshold; Monitor the queue length of each port of the switch to determine the start and end of the micro-burst; If the queue length of the port exceeds the queue threshold within a certain time, it is considered that the micro-burst starts; If the queue length of the port is lower than the queue threshold within a certain time, it is considered that the micro-burst ends; (3) State update: When a micro-burst occurs, input the collected data into the reinforcement learning model to obtain the corresponding state; If the state does not exist, initialize the state in the corresponding register and initialize the probability of the corresponding relief strategy; (4) Action selection: According to the strategy with the highest probability given by the reinforcement learning model inference, find the selection probability of each port under the current state; Select the port with the highest probability as the action for this round of relief, and set the output port of the data packet to the corresponding port, and record the current state and action selection in the register; (5) Reward feedback: After the micro-burst ends, calculate the reward function; Obtain the feedback of this micro-burst and update the selection probability of each port under the previously recorded state; Update the historical average packet loss data under the corresponding state; The reinforcement learning model is deployed on the data plane, and the micro-burst relief strategy does not need to be assisted by the terminal host to execute.

2. The microburst mitigation method based on in-network reinforcement learning according to claim 1, wherein, In step (1), the network state information includes queue packet loss, data packet byte number within 10μs of each port, micro-burst size, and long / short flow ratio.

3. The microburst mitigation method based on in-network reinforcement learning according to claim 1, wherein, In step (4), three micro-burst relief strategies of port steering, buffer management, and congestion control are adopted, and different strategies are suitable for different network environments and network requirements.

Images