Traffic control device and traffic control method

Abstract

This traffic control device (1) comprises: a CF (110) that distributes input packets to policers (131, 132) on the basis of a predetermined distribution setting; the policers (131, 132) which mark the input packets according to CIR and PIR; queues (251, 252) that store the marked packets; and a monitoring processing unit (410) that monitors the marked packets, selects a flow to be filtered from among flows determined to be discarded on the basis of congestion avoidance control based on the remaining capacity of the queues, extracts a 5-tuple of the selected flow, and additionally sets the 5-tuple as filter information in the CF (110).

Description

Traffic control device and traffic control method

[0001] The present invention relates to a traffic control device and a traffic control method, and more particularly to a traffic control device and a traffic control method for controlling congestion.

[0002] In recent years, with the development of AI (Artificial Intelligence) and LLM (Large Language Models), RDMA (Remote Direct Memory Access) has attracted attention (see Non-Patent Literature 1). RDMA has the following characteristics: The local host sends data directly to the memory of the GPU (Graphics Processing Unit) of the remote host over the network. Communication standards include Infiniband provided by NVIDIA® and RoCE (RDMA over Converged Ethernet), which implements Infiniband over Ethernet®. RDMA assumes that there is no packet loss, i.e., lossless transmission, where packets are not lost along the transmission path. In RDMA, communication can occur in bursts.

[0003] In RDMA, TCP (Transmission Control Protocol) is often not used to reduce the complexity of communication processing. In such cases, if a packet is lost, subsequent packets within the same communication become wasted (Go-Back-N). For example, as shown in Figure 6, suppose server 901 sends packets with sequence numbers 001 to 005 within a predetermined communication. Here, the sequence number is indicated as "SEQ" in the figure. When server 902 detects the loss of packet with sequence number 003, it notifies server 901 of this fact with a NAK (Negative Acknowledgment) packet. Upon receiving the NAK packet, server 901 retransmits packets with sequence numbers 003 to 005. In other words, all packets after the dropped packet must be retransmitted.

[0004] On the other hand, in the prior art, technologies such as PFC (Priority-Based Flow Control) and ECN (Explicit Congestion Notification) that suppress packets before packet loss occurs have also been proposed (see Non-Patent Documents 2 to 6).

[0005] For example, PFC (Non-Patent Document 2) is a congestion control algorithm per queue. In PFC, traffic is classified by CoS (class of service) and DSCP (Differentiated Services Code Point). Then, when the usage amount of the target queue, for example, the No. 3 queue in FIG. 7, exceeds a predetermined threshold TH, a request to "stop transmission for a certain period" is sent from the switch side to the adjacent node. Thereby, in PFC, packet loss caused by queue overflow can be prevented.

[0006] For example, ECN (Non-Patent Documents 3 and Non-Patent Document 6) realizes fairness between flows by probabilistic marking. In ECN, the IP packet header has an ECN-capable Transport (ECT) bit and a CE (Congestion Experienced) bit. FIG. 8A shows four combinations of "00", "01", "10", and "11" using the ECT bit and the CE bit, and their explanations. Also, when the queue usage amount exceeds the threshold, the switch probabilistically flags the packet header and notifies the host and network devices that the congestion state has occurred.

[0007] For example, as shown in Figure 8B, server 903 sends a packet with the ECN field in the header set to "10" to server 906 via switches 904 and 905. At this time, switch 904 detects queue congestion and drops the packet through random early detection (RED DROP). In this state, switch 904 changes the ECN field in the header to "11" and sends the packet to server 906. As a result, server 906 confirms that congestion is occurring and notifies the sender of a CNP (Congestion Notification Packets). Upon receiving this notification, server 906 adjusts the transmission speed to suppress congestion.

[0008] The Enhanced Transmission Selection (ETS) described in Non-Patent Document 4 is a technology that separates queues by traffic type, assigns different weights to each queue, and determines the transmission schedule. In the example shown in Figure 9, for queues "0" to "2" of the eight priorities, the traffic type is defined as "Other" and the transmission algorithm as "WRR (Weighted Round Robin): 30%". For queue "3", the traffic type is defined as "RoCEv2: data" and the transmission algorithm as "WRR (Weighted Round Robin): 70%". RoCEv2 is an abbreviation for RDMA over Converged Ethernet version 2. For queue "6", the traffic type is defined as "RoCEv2: CNP" and the transmission algorithm as "Strict Priority". Strict Priority indicates that transmission is performed in order of priority.

[0009] Non-patent document 5 describes trTCM (Two Rate Three Color Marker) as measuring an IP packet stream and marking the packet in red, yellow, or green based on two rates, PIR (Peak Information Rate) and CIR (Committed Information Rate), and the associated burst size. Non-patent document 6 describes WRED (Weighted Random Early Detection) as enabling early detection of congestion and providing a means for handling multiple traffic classes.

[0010] NVIDIA, “Developing a Linux Kernel Module using GPUDirect RDMA”, [online], [Retrieved September 20, 2024], Internet <URL: https: / / docs.nvidia.com / cuda / gpudirect-rdma / index.html> IEEE 802.1Qbb, “Priority-based Flow Control”, [online], [Retrieved September 20, 2024], Internet <URL: https: / / 1.ieee802.org / dcb / 802-1qbb / > RFC9331, “The Explicit Congestion Notification (ECN) Protocol for Low Latency, Low Loss, and Scalable Throughput (L4S)”, [online], [Retrieved September 20, 2024], Internet <URL: https: / / datatracker.ietf.org / doc / rfc9331 / > IEEE 802.1Qaz, “Enhanced Transmission Selection”, [online], [Retrieved September 20, 2020], Internet <URL: https: / / 1.ieee802.org / dcb / 802-1qaz / >RFC2698, “A Two Rate Three Color Marker”, [online], [Retrieved September 20, 2020], Internet <URL: https: / / www.rfc-editor.org / rfc / rfc2698> Cisco, “Chapter: WRED Explicit Congestion Notification”, [online], [searched on September 20, 2020], Internet <URL: https: / / www.cisco.com / c / en / us / td / docs / ios-xml / ios / qos_conavd / configuration / xe-16 / qos-conavd-xe-16-book / qos-conavd-wred-ecn.html>

[0011] Conventional technologies such as PFC and ECN apply the same rules to each server or to all packets, causing ongoing processing to be suspended for a certain period of time. Therefore, in view of the above situation, the present invention aims to reduce the range of packet retransmissions in situations where congestion is unavoidable.

[0012] The traffic control device according to the present invention is characterized by comprising: an inlet classifier that distributes input packets to a policer based on predetermined distribution settings; the policer that marks input packets according to CIR and PIR; a queue that stores the marked packets; and a monitoring processing unit that monitors the marked packets, selects flows to be filtered from among flows that are determined to be discarded based on congestion avoidance control according to the remaining capacity of the queue, extracts a 5-tuple of the selected flow, and adds the 5-tuple as filter information to the inlet classifier.

[0013] According to the present invention, the range of packet retransmissions can be reduced in situations where congestion is unavoidable.

[0014] This is a schematic diagram of a traffic control device according to the first embodiment of the present invention. This is a schematic diagram showing an example of the monitoring results for each flow. This is a schematic diagram showing each processing step of the traffic control device. This is a schematic diagram of a traffic control device according to the second embodiment of the present invention. This is a schematic diagram of a traffic control device according to the third embodiment of the present invention. This is a schematic diagram showing inter-server communication using RDMA. This is a schematic diagram showing an example of congestion control by PFC. This is a diagram showing the bit setting of ECN. This is a schematic diagram showing an example of congestion control by ECN. This is a schematic diagram showing an example of scheduling by ETS.

[0015] The traffic control device according to this embodiment will be described in detail below with reference to the drawings. As shown in Figure 1, the traffic control device 1 includes an ASIC (Application Specific Integrated Circuit) 100, an ASIC 200, a memory 300, a CPU (Central Processing Unit) 400, IFs (Interfaces) 510, 520, 530, and an internal bus 540. The traffic control device 1 is configured as a network device such as a router. The traffic control device 1 controls traffic between, for example, a server 610 connected via IF 510 and a server 620 connected via IF 530.

[0016] The ASIC 100 is an Ingress processing unit that processes packets received from the outside. The ASIC 100 includes a CF (Classifier) ​​110, policers 131 and 132, queues (Ingress Queues) 151 and 152, and a scheduler 170. The CF 110 distributes packets that enter via IF 510 or IF 520 to policers 131 and 132 based on predetermined distribution settings. The CF 110 distributes packets by referring to, for example, a distribution setting table.

[0017] Policers 131 and 132 monitor incoming packets and discard them according to their settings. Policers 131 and 132 mark incoming packets according to the CIR and PIR using a three-color marker. As the three-color marker, for example, trTCM (Two Rate Three Color Marker) described in Non-Patent Literature 5 can be used. In trTCM, packets exceeding the PIR are marked in red. Otherwise, packets exceeding the CIR are marked in yellow, while packets not exceeding the CIR are marked in green. Note that srTCM (Single Rate Three Color Maker) may be used instead of trTCM as the three-color marker. Policers 131 and 132 discard packets according to their filter settings.

[0018] Queues 151 and 152 are individual queues for each policer and store marked packets. The scheduler 170 determines the transmission schedule for packets in the individual queues (queues 151 and 152). The scheduler 170 may employ a transmission algorithm such as WRR.

[0019] ASIC200 is connected to ASIC100 by an internal bus 540. ASIC200 is an Egress processing unit that handles the processing of sending packets to the outside. ASIC200 includes a CF (exit classifier) ​​210, Egress queues 251 and 252, and a scheduler 270.

[0020] CF210 distributes packets output from individual queues (queues 151, 152) to queues 251, 252 based on the settings. Queues 251 and 252 are priority class (QoS class) queues and store packets. Scale 261 schematically shows the marking threshold (CIR), and scale 262 schematically shows the marking threshold (PIR). The scheduler 270 determines the transmission schedule for packets output from priority class queues (queues 251, 252). Packets stored in queues 251 and 252 are output via IF530. The scheduler 270 may employ, for example, a Strict Priority transmission algorithm.

[0021] The memory 300 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive). The memory 300 stores the results of the CPU 400's processing, such as the monitoring results 310.

[0022] The CPU 400 is equipped with a monitoring process (monitoring processing unit) 410. The monitoring process 410 monitors packets passing through the Ingress processing unit (ASIC 100) and Egress processing unit (ASIC 200), as well as specific queues. The monitoring process 410 determines whether or not to discard packets based on congestion avoidance control according to the remaining capacity of the queue. The monitoring process 410 monitors marked packets and selects flows to be filtered from among the flows that it has determined to discard based on congestion avoidance control. The monitoring process 410 extracts a 5-tuple from the selected flow and adds this 5-tuple as filter information to the CF 110. Here, other selection methods such as WRED (Weighted Random Early Detection) or RED (Random Early Detection) can be used as the congestion avoidance control. In the following, the congestion avoidance control will be assumed to be WRED as an example. Therefore, the monitoring process 410 determines whether or not to discard packets based on WRED, which is based on the queue length of packets accumulated in the priority class queues (queues 251, 252).

[0023] Flow information is forwarded to the monitoring process 410 via the router's mirroring function or NetFlow, etc. Flow information consists of packet or header information. For example, in the traffic control device 1 acting as a router, a copy of the packet is sent to the monitoring process 410 via mirroring.

[0024] The monitoring process 410 extracts a 5-tuple from the packets colored red and yellow that were determined to be discarded by WRED. The 5-tuple consists of the source IP address, source port number, destination IP address, destination port number, and protocol number.

[0025] The monitoring process 410 monitors the queue length of each priority class queue (queues 251, 252) and selects a flow when the queue length exceeds a preset threshold. The time (hereinafter referred to as unit time) for the monitoring process 410 to repeat the process of monitoring each queue length is preset. When the queue length exceeds the threshold, the monitoring process 410 selects a flow to be filtered from among the marked flows based on the monitoring result 310 stored as data for the most recent unit time.

[0026] In this embodiment, the monitoring process 410 monitors the number of marked packets per unit time for each flow and selects flows to be filtered based on the number of marked packets per unit time. The monitoring process 410 monitors the number of incoming packets per unit time for each marked flow.

[0027] As a selection logic for selecting a flow, for example, a weighted probability selection based on incoming packets can be employed. This weighted probability selection will be explained with reference to Figure 2. Figure 2 shows an example of monitoring results stored for the most recent unit period. In this example, for flow A, the number of yellow packets was "40", the number of red packets was "20", and the total number of incoming packets was "120". For flow B, the number of yellow packets was "12", the number of red packets was "0", and the total number of incoming packets was "72".

[0028] Here, for example, if we assign a weight of 12 to red and a weight of 6 to yellow, the score and probability of each flow are calculated as follows: Score of flow A: 6 × 40 + 12 × 20 = 480 … (1) Score of flow B: 6 × 12 + 12 × 0 = 72 … (2) Total score: 480 + 72 = 552 … (3) Probability of flow A being selected: 480 / 552 … (4) Probability of flow B being selected: 72 / 552 … (5) Therefore, according to equations (4) and (5), flow A, which has a higher probability, can be selected as the flow to be filtered. Alternatively, selection could be based on the magnitude of the score. Furthermore, the method of selecting the flow to be filtered from the marked flows is not limited to these methods; random selection is also acceptable.

[0029] [Operation of the Traffic Control Device] Next, the operation of the traffic control device 1 will be explained with reference to Figure 3 (and Figure 1 as appropriate). First, the monitoring process 410 is started in the traffic control device 1 (step S1). Then, the CF 110 of the traffic control device 1 distributes the input packets to the policers 131 and 132 based on predetermined distribution settings.

[0030] Then, the policers 131 and 132 of the traffic control device 1 mark the input packets according to the CIR and PIR (step S2). Then, for example, the policers 131 and 132 transfer the flow information by mirroring (step S3).

[0031] The monitoring process 410 then monitors the number of marked packets per unit time (step S4). The monitoring process 410 also determines whether or not to discard a flow based on WRED. Specifically, the monitoring process 410 monitors the queue length and determines whether or not the queue length exceeds a threshold (step S5).

[0032] For example, if it is determined that a flow should be discarded, that is, if the queue length exceeds a threshold, the monitoring process 410 selects the flows to be filtered from among the flows that it has determined to discard (step S6).

[0033] Then, the monitoring process 410 extracts the 5-tuple of the selected flow and adds a filter setting that uses this 5-tuple as filter information (step S7). In other words, the monitoring process 410 modifies the ACL (Access Control List) setting of the Ingress Interface (IF 510) by designating the 5-tuple of the selected flow as a target for dropping. Note that the above step numbers are for convenience only and do not necessarily accurately represent the processing order. For example, packet marking (S2) and counting the number of marked packets per unit time (S4) operate in parallel as separate processes from the queue monitoring process (S5). Also, queue monitoring (S5) is a loop process and continues even after flow selection (S6) and filter addition (S7).

[0034] (Second Embodiment) Next, with reference to Figure 4, the traffic control device 1B according to the second embodiment will be described. Components similar to those in the first embodiment are denoted by the same reference numerals and their descriptions are omitted. As shown in Figure 4, the traffic control device 1B is connected to node 630 via IF 530. Server 620, which is connected to the traffic control device 1B via IF 530, is connected to node 640. Nodes 630 and 640 are adjacent nodes of their own node (traffic control device 1B). Here, adjacent nodes mean nodes that are subject to load balancing using ECMP (Equal Cost Multi-Path).

[0035] In this traffic control device 1B, the monitoring process 410B has additional functionality for nodes 630 and 640. The monitoring process 410B not only selects flows to be filtered from the marked flows and uses them at its own node, but also notifies the neighboring nodes (nodes 630 and 640) of the filter information for the selected flows. This allows nodes 630 and 640 to perform the same filtering.

[0036] (Third Embodiment) Next, with reference to Figure 5, a traffic control device 1C according to the third embodiment will be described. Components similar to those in the first embodiment are denoted by the same reference numerals and their descriptions are omitted. As shown in Figure 5, in the traffic control device 1C, the CPU 400C includes a monitoring process 410, a determination unit 420, and a congestion suppression unit 450.

[0037] The determination unit 430 performs a two-stage threshold determination on the queue length of the priority class queues (queues 251, 252) using a first threshold and a second threshold that is greater than the first threshold. The determination unit 430 determines whether the queue length of the priority class queues (queues 251, 252) exceeds the first threshold, and also determines whether the queue length exceeds the second threshold.

[0038] The congestion suppression unit 450 performs existing congestion control. When the queue length exceeds a first threshold, the congestion suppression unit 450 suppresses congestion using methods such as PFC or ECN. The monitoring process 410 performs additional filter information setting when the queue length exceeds a second threshold before the congestion suppression unit 450 operates. The traffic control device 1C is expected to avoid significant performance degradation by using existing methods (PFC, ECN, etc.) in combination depending on the conditions.

[0039] [Effects] As described above, the traffic control device 1 is characterized by comprising: a CF 110 that distributes input packets to policers 131 and 132 based on predetermined distribution settings; policers 131 and 132 that mark input packets according to CIR and PIR; queues 251 and 252 that store the marked packets; and a monitoring process 410 that monitors the marked packets, selects flows to be filtered from among flows that are determined to be discarded based on congestion avoidance control (e.g., WRED) according to the remaining capacity of the queue, extracts a 5tuple of the selected flow, and adds the 5tuple as filter information to the CF 110.

[0040] In this way, the traffic control device 1 marks incoming packets according to the CIR and PIR as ingress control. srTCM or trTCM can be used for marking. For example, trTCM measures the IP packet stream and marks the packet as green, yellow, or red based on two rates: the peak information rate (PIR) and the commit information rate (CIR), along with the associated burst size. This efficiently selects candidate packets to be discarded in situations where congestion is unavoidable, based on the information of the packets flowing into the device. In addition, as outgress control, the traffic control device 1 selects flows with candidate packets to be discarded based, for example, a threshold for the queue length inside the device using WRED. This allows the traffic control device 1 to filter the selected flows, thereby blocking communication for a certain period before queue overflow occurs. Therefore, because the traffic control device 1 can discard packets for specific flows, it can reduce the packet retransmission range and suppress the scale of congestion.

[0041] In the traffic control device 1, the monitoring process 410 monitors the number of marked packets per unit time for each flow and selects flows to be filtered based on the number of marked packets per unit time.

[0042] By doing so, the traffic control device 1 aggregates the selected marked packets for each flow, allowing it to employ, for example, weighted probability selection based on incoming packets as the logic for selecting flows to be filtered. This makes it possible to prioritize filtering flows with a higher proportion of incoming packets when selecting flows to be filtered. As a result, the probability of suppressing congestion earlier is higher than when flows are selected randomly.

[0043] In the traffic control device 1B, the monitoring process 410 is characterized by notifying adjacent nodes of filter information in order to filter the selected flow at adjacent nodes of its own node.

[0044] By doing so, the traffic control device 1B can not only utilize the filter information regarding the flow to be filtered in its own node, but also utilize it in adjacent nodes. For example, in nodes that are targets of load distribution by ECMP, etc., by utilizing the filter information, the influence of congestion can be effectively suppressed. In particular, in a multi-tenant environment, by suppressing the occurrence of congestion across multiple tenants, an improvement in the learning efficiency of the LLM in a congested environment can be expected.

[0045] Note that the present invention is not limited to the embodiments described above, and many modifications are possible by those with ordinary knowledge in the art within the technical idea of the present invention. For example, although the traffic control device 1 is assumed to be a router, it may be configured as a network switch. Although the traffic control device 1 is provided with two ASICs, it may be integrated into one ASIC. In that case, the scheduler 170 may also perform the function of the CF210.

[0046] 1, 1B, 1C Traffic control device 100 ASIC 110 CF (input classifier) 131, 132 Policer 151, 152 Queue 170 Scheduler 200 ASIC 210 CF (output classifier) 251, 252 Queue 270 Scheduler 261, 262 Scale 300 Memory 310 Monitoring result 400, 400C CPU 410, 410B Monitoring process (monitoring processing unit) 430 Determination unit 450 Congestion suppression unit 510, 520, 530 IF 540 Internal bus 610, 620 Server 630, 640 Node

Claims

1. A traffic control device comprising: an inlet classifier that distributes input packets to a policer based on predetermined distribution settings; the policer that marks input packets according to CIR and PIR; a queue for storing marked packets; and a monitoring processing unit that monitors marked packets, selects flows to be filtered from among flows that are determined to be discarded based on congestion avoidance control according to the remaining capacity of the queue, extracts a 5-tuple from the selected flows, and adds the 5-tuple as filter information to the inlet classifier.

2. The traffic control device according to claim 1, characterized in that the monitoring processing unit monitors the number of marked packets per unit time for each flow and selects flows to be filtered based on the number of marked packets per unit time.

3. The traffic control device according to claim 1 or 2, characterized in that the monitoring processing unit notifies the adjacent node of the filter information in order to filter the selected flow at the adjacent node of the node.

4. A traffic control method by a traffic control device, the traffic control device comprising the steps of: marking input packets distributed to a policer based on predetermined distribution settings according to CIR and PIR; monitoring the marked packets and determining whether or not to discard the flow based on congestion avoidance control according to the remaining capacity of the queue; selecting a flow to be filtered from among the flows determined to be discarded; and extracting a 5tuple of the selected flow and adding the 5tuple as filter information to the distribution settings.

Images