Micro burst relieving method based on in-network reinforcement learning

Abstract

The invention relates to a micro-emergency mitigation method based on in-network reinforcement learning, and the method enables a switch to have a micro-emergency processing capability without the assistance of a controller through deploying a reinforcement learning algorithm in a programmable data plane, reduces the time delay of a data-control plane compared with a conventional scheme, and improves the reliability of the data-control plane. And the microsecond-level remission decision-making requirement required by the microburst is met. Besides, the reinforcement learning model enables the network element to be adaptive to a continuously changing network environment and a new flow mode, so that the switch can make an optimal micro-burst mitigation strategy according to a real-time network state, and the processing burden of a host end and the influence on service flow are reduced.

Description

Technical Field

[0001] The present invention relates to the field of network technology, and in particular to a microburst mitigation method based on in-network reinforcement learning. Background Art

[0002] Microbursts refer to the receipt of a relatively large number of datagrams within milliseconds or microseconds. The typical characteristics are short duration (1 to 100ms) and high instantaneous burst data rate (tens or hundreds of times higher than the average data rate, or even higher than the port bandwidth). Microbursts are common in modern data center networks (DCNs). In DCNs, the network often runs at high rates (>10Gbps) with extremely low end-to-end latency (about 10 microseconds). Under such extremely low latency requirements, even a small number of datagram queues have queue latency that is considered unacceptable, so microbursts often have an impact that cannot be underestimated. In addition, microbursts also frequently occur in large-scale transactions on the Internet of Things, stock markets, and e-commerce promotion days. For example, a large number of instantaneous transactions in the stock market or the inevitable consolidation of multiple flows in the actual network will form microbursts in a very short time.

[0003] When a microburst exceeds the forwarding capacity of the switch, the switch buffers the burst data for subsequent transmission. If the switch does not have enough buffer space, the excess data will be discarded, resulting in high congestion and a large number of data packets lost. In addition, as network equipment continues to operate, it is inevitable that the ability to cope with burst flows will decrease due to network changes, device aging, software bugs, etc., and the number of potential microburst events will increase, making it more likely to cause serious situations such as service interruptions and downtime.

[0004] Due to the unique characteristics of data center applications, such as high capacity and dynamic traffic patterns of short flows, microbursts can significantly affect the performance of applications. The slow response of traditional congestion control algorithms makes them ineffective against microbursts. Although existing microburst mitigation methods have achieved certain results, they still have significant limitations: First, fixed policies and parameter configurations. The response strategies of existing works to microburst events are pre-defined, that is, relatively fixed, which makes it difficult for existing works to achieve real-time dynamic adaptation to network changes. Any changes in policies and parameter configurations require redeployment. Second, there is a lack of fine-grained analysis of network status. Existing works lack the collection of network environment information, either relying only on local information to infer surrounding network information, or only collecting some coarse-grained network information, such as the length of directly connected queues. Third, some works require terminal information assistance. Some related works require the terminal host to monitor the traffic it sends and embed information related to traffic burstiness and priority information into the data packet, thereby achieving host-side information assistance, but this assistance adds additional host-side burden and affects the service flow.

[0005] With the development of programmable network devices, the flexibility and programmability of network devices have been greatly improved. In particular, programmable switches and routers based on the P4 language provide support for deep control of the data plane and customized processing capabilities, making it possible to directly deploy reinforcement learning models in network devices. Therefore, the use of programmable switches combined with the relevant functions of data plane design provides a new idea for solving the problem of microburst flow detection that traditional switches and traditional software solutions do not consider and are difficult to support. Summary of the invention

[0006] The object of the present invention is to provide a microburst mitigation method based on in-network reinforcement learning, which can optimize the forwarding decision in the data plane of a programmable switch, thereby achieving effective mitigation of microburst traffic in a complex and changing network.

[0007] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a microburst mitigation method based on in-network reinforcement learning, which monitors the traffic in real time, achieves rapid response through the in-network reinforcement learning model, and can select the best forwarding strategy to mitigate microbursts according to the network environment; specifically, it includes the following steps: (1) Data collection: Collect network status information and adjacent switch information; the agent obtains adjacent switch information through probes sent by adjacent switches; (2) Microburst detection and judgment: According to the definition of microburst, set the queue threshold; monitor the queue length of each port of the switch to determine the start and end of the microburst; if the port queue length exceeds the set threshold within the set time, it is considered that the microburst has started; if the port queue length is lower than the set threshold within the set time, it is considered that the microburst has ended; (3) State update: When a microburst occurs, the collected data is input into the reinforcement learning model to obtain the corresponding state information; if the state does not exist, the state is initialized in the corresponding register and the probability of the corresponding mitigation strategy is initialized; (4) Action selection: Based on the strategy with the highest probability given by reinforcement learning inference, find the selection probability of each port in the current state; select the port with the highest probability as the action for this round of mitigation, set the egress port of the data packet to the corresponding port, and use a register to record the current state and action selection; (5) Reward feedback: After the microburst ends, the reward function is calculated; the feedback of the microburst is obtained and the probability of selecting each port under the previously recorded state is updated; the historical average packet loss data under the corresponding state is updated.

[0008] Furthermore, the in-network reinforcement learning model is deployed on the data plane so as to have the ability to learn the best mitigation strategy according to the real-time network status, so as to achieve the effect of quickly responding to microburst traffic and making decisions within microseconds.

[0009] Furthermore, by deploying the method in the network, a microburst mitigation decision is implemented without the assistance of a terminal host, minimizing the processing burden on the host side and the impact on service traffic.

[0010] Furthermore, in step (1), the network status information includes queue packet loss, the number of data packet bytes within 10 μs of each port, microburst size and long flow / short flow ratio, and the adjacent switch information includes the queue and load status of each port of the adjacent switch.

[0011] Furthermore, in step (1), an information collection method based on in-band network telemetry is constructed to collect and analyze real-time traffic characteristics, and a response strategy is made according to the traffic pattern, microburst size and switch port load information in the network.

[0012] Furthermore, in step (4), three microburst mitigation strategies, namely port steering, buffer management, and congestion control, are adopted. Different strategies can better adapt to different network environments and network requirements and optimize network performance to the greatest extent.

[0013] Furthermore, the in-network reinforcement learning model implements the microburst mitigation strategy on the network element, and relies on the wide deployment range and large deployment scale of the network element to achieve large-scale scalability of the microburst mitigation system, thereby adapting to the complex network environment.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a microburst mitigation method based on in-network reinforcement learning. This method enables the switch to handle microburst events without the assistance of a controller by deploying a reinforcement learning algorithm on a programmable data plane. Compared with traditional solutions, this method reduces the data-control plane latency and meets the microsecond-level mitigation decision requirements required for microbursts. In addition, the reinforcement learning model enables network elements to adapt to the ever-changing network environment and new traffic patterns, so that the switch can make the best microburst mitigation strategy based on the real-time network status, reducing the processing burden on the host side and the impact on business traffic. The present invention is scalable and can effectively improve the microburst traffic monitoring and mitigation capabilities in large-scale network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is an overall architecture diagram of an embodiment of the present invention; Figure 2 It is the theoretical framework of the microburst mitigation method of the embodiment of the present invention; Figure 3 This is a reinforcement learning model of the microburst mitigation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0017] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0018] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0019] This embodiment provides a microburst mitigation method based on in-band network telemetry (INT), which builds an information collection method based on in-band network telemetry (INT) to monitor traffic in real time, achieves rapid response through an in-band reinforcement learning (RL) model, and can select a suitable switch port forwarding strategy according to the network environment, thereby achieving microburst traffic mitigation at the microsecond level. The method is based on a microburst adaptive mitigation system implemented by P4 and reinforcement learning, solves the network problems caused by microbursts, and uses the advantages of programmable data planes to achieve microsecond-level decision-making. First, by deploying reinforcement learning, dynamic network adaptation is achieved, microburst traffic mitigation strategies are optimized in real time, and congestion, delay and packet loss are reduced. With the help of the P4 programmable data plane, the support of programmable switches for Gbps to Tbps-level traffic is used to solve the delay problem of the traditional control plane, achieve millisecond-level microburst rapid response, and reduce delay. At the same time, the method deploys in-band network telemetry to collect adjacent network entity information in a fine-grained manner to support dynamic adaptation of decision-making schemes. In addition, the method deploys the scheme within the network, autonomously achieves microburst traffic mitigation, and reduces the host burden and business traffic impact.

[0020] The specific implementation steps of this method are as follows.

[0021] (1) Data collection: Collect network status information and adjacent switch information. Network status information includes queue packet loss, number of data packet bytes within 10μs on each port, microburst size, long flow / short flow ratio, etc. Adjacent switch information includes queues and load conditions of each port of the adjacent switch. The agent obtains adjacent switch information through probes sent by the adjacent switch.

[0022] In this embodiment, an information collection method based on in-band network telemetry is constructed to collect and analyze real-time traffic characteristics, and a response strategy is made according to the traffic pattern, microburst scale and switch port load information in the network.

[0023] (2) Microburst detection and judgment: According to the definition of microburst, set the appropriate queue threshold. Monitor the queue length of each port of the switch to determine the start and end of the microburst. If the port queue length exceeds the set threshold within the set time (100ns), it is considered that the microburst has started; if the port queue length is lower than the set threshold within the set time (100ns), it is considered that the microburst has ended.

[0024] (3) State update: When a microburst occurs, the collected data is input into the reinforcement learning model to obtain the corresponding state information. If the state does not exist, the state is initialized in the corresponding register, and the probability of the corresponding mitigation strategy is initialized (such as the probability of selecting each steering port).

[0025] (4) Action selection: According to the strategy with the highest probability given by reinforcement learning inference, find the selection probability of each port in the current state. Select the port with the highest probability as the action for this round of mitigation, set the egress port of the data packet to the corresponding port, and use a register to record the current state and action selection.

[0026] In this embodiment, three microburst mitigation strategies, namely, port steering, buffer management, and congestion control, are adopted. Different strategies can better adapt to different network environments and network requirements and optimize network performance to the greatest extent.

[0027] (5) Reward feedback: After the microburst ends, the reward function is calculated; the feedback of the microburst is obtained and the probability of selecting each port under the previously recorded state is updated; the historical average packet loss data under the corresponding state is updated.

[0028] The in-network reinforcement learning model is deployed on the data plane to have the ability to learn the best mitigation strategy according to the real-time network status, so as to achieve the effect of quickly responding to microburst traffic and making decisions within microseconds.

[0029] The in-network reinforcement learning model implements the microburst mitigation strategy on the network element, and relies on the wide deployment range and large deployment scale of the network element to achieve large-scale scalability of the microburst mitigation system, thereby adapting to the complex network environment.

[0030] like Figure 1As shown, the present invention proposes a highly real-time and highly responsive microburst mitigation solution by combining a reinforcement learning model with a programmable network device. By deploying in-band network telemetry technology, fine-grained collection of adjacent network entity information is achieved, providing information support for rapid response to microburst traffic. By innovatively combining in-band telemetry technology with data plane reinforcement learning, the present invention supports microburst traffic mitigation in high-throughput, high-concurrency networks, significantly improves network utilization, and minimizes the processing burden on the host side and the impact on business traffic. The present invention can achieve real-time monitoring of microburst traffic and microsecond-level response decisions, providing an effective solution for the large-scale scalability of microburst mitigation systems, as follows: 1. Data dashboard and data plane deployment: Figure 1 As shown, the overall architecture of the present invention includes two parts: data dashboard and data plane. In the data plane, fine-grained traffic monitoring is performed through in-band telemetry technology, and a reinforcement learning model is deployed to have the ability to learn the best mitigation strategy based on the real-time network status, so as to achieve the effect of quickly responding to microburst traffic and making decisions within microseconds. The programmable switch executes different forwarding strategies according to different decisions to achieve the effect of mitigating microburst traffic.

[0031] 2. Reinforcement learning model design and training: Most microburst scenarios can be modeled as Markov decision processes, and the reward-based learning framework and value function optimization method of reinforcement learning are excellent in solving Markov problems. The packet loss rate in the network is used as the reward function. The packet loss rate calculation method is: packet loss rate = packet loss amount / microburst scale. The packet loss rate can better reflect the damage of traffic than the packet loss amount, and it can also reflect the mitigation effect of microburst traffic.

[0032] 3. Theoretical framework of microburst mitigation method: Figure 2 As shown, the theoretical architecture of the present invention is divided into application layer, decision layer, policy layer and data layer. The application layer explains that microburst mitigation is mainly used in scenarios where multiple terminals send a large amount of traffic to one terminal, such as distributed machine learning, network caching and other application scenarios; the decision layer explains that this system uses a reinforcement learning model to make decisions on microburst mitigation methods under real-time network conditions; the policy layer explains that this system provides three mitigation solutions, including port steering, buffer management and congestion control; the data layer explains that this system is implemented based on a programmable data plane.

[0033] 4. Reinforcement learning model decision execution framework: Figure 3As shown in the figure, the reinforcement learning agent obtains the information of adjacent switches through the probes sent by the adjacent switches, monitors the queue length of each port of the switch, and determines the start and end of the microburst. When a microburst occurs, the collected data is input into the reinforcement learning model to obtain the corresponding state information. According to the model strategy, the selection probability of each port in the current state is found, and the port with the highest probability is selected as the action for this round of mitigation. After each round of microburst ends, the reward function is updated.

[0034] In order to solve the problems of frequent microburst traffic generation, difficulty in monitoring and lagging mitigation strategies in existing large-scale networks, the present invention provides a microburst mitigation method based on in-network reinforcement learning, which can optimize the forwarding decisions in the data plane of programmable switches, realize real-time monitoring of microburst traffic and microsecond-level response decisions, and effectively alleviate microburst traffic in complex and changing networks, thereby reducing the processing burden on the host side and the impact on business traffic.

[0035] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0036] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0037] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0038] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0039] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A microburst mitigation method based on in-network reinforcement learning, characterized in that: Monitor traffic in real time and achieve rapid response through the in-network reinforcement learning model. It can select the best forwarding strategy to alleviate microbursts according to the network environment. The specific steps include: (1) Data collection: Collect network status information and adjacent switch information; the agent obtains adjacent switch information through probes sent by adjacent switches; (2) Microburst detection and judgment: According to the definition of microburst, set the queue threshold; monitor the queue length of each port of the switch to determine the start and end of the microburst; if the port queue length exceeds the set threshold within the set time, it is considered that the microburst has started; if the port queue length is lower than the set threshold within the set time, it is considered that the microburst has ended; (3) State update: When a microburst occurs, the collected data is input into the reinforcement learning model to obtain the corresponding state information; if the state does not exist, the state is initialized in the corresponding register and the probability of the corresponding mitigation strategy is initialized; (4) Action selection: Based on the strategy with the highest probability given by reinforcement learning inference, find the selection probability of each port in the current state; select the port with the highest probability as the action for this round of mitigation, set the egress port of the data packet to the corresponding port, and use a register to record the current state and action selection; (5) Reward feedback: After the microburst ends, the reward function is calculated; the feedback of the microburst is obtained and the probability of selecting each port under the previously recorded state is updated; the historical average packet loss data under the corresponding state is updated.

2. According to claim 1, a microburst mitigation method based on in-network reinforcement learning is characterized in that: The in-network reinforcement learning model is deployed on the data plane to have the ability to learn the best mitigation strategy according to the real-time network status, so as to achieve the effect of quickly responding to microburst traffic and making decisions within microseconds.

3. The microburst mitigation method based on in-network reinforcement learning according to claim 1, characterized in that: By deploying the method within the network, microburst mitigation decisions can be made without the assistance of terminal hosts, minimizing the processing burden on the host side and the impact on business traffic.

4. The microburst mitigation method based on in-network reinforcement learning according to claim 1, characterized in that: In step (1), the network status information includes queue packet loss, the number of data packet bytes within 10 μs of each port, the microburst scale and the ratio of long flow to short flow, and the adjacent switch information includes the queue and load status of each port of the adjacent switch.

5. The microburst mitigation method based on in-network reinforcement learning according to claim 1, characterized in that: In step (4), three microburst mitigation strategies are adopted: port steering, buffer management, and congestion control. Different strategies can better adapt to different network environments and network requirements and optimize network performance to the greatest extent.

6. The microburst mitigation method based on in-network reinforcement learning according to claim 1, characterized in that: The in-network reinforcement learning model implements the microburst mitigation strategy on the network element, and relies on the wide deployment range and large deployment scale of the network element to achieve large-scale scalability of the microburst mitigation system, thereby adapting to the complex network environment.

Images