Network flow size measurement scheme based on ROCE protocol

Abstract

The invention relates to the technical field of high-performance networks and network telemetering technologies, in particular to a network flow size measurement scheme based on an ROCE protocol, which comprises the following steps of: unloading tasks of data packet processing and counter position mapping on an end side to a P4 switch, and then sending flow information to a queriable storage structure of a collector through the ROCE protocol, so that the flow size of a network flow is measured. The process does not need the participation of a collector CPU (Central Processing Unit), so that the network time delay is greatly reduced; the method comprises the following steps: S1, establishing a Sketch data structure in a collector, and recording the Sketch data structure as RT Sketch; performing RDMA configuration initialization; s2, constructing and sending a data packet; s3, the switch processes the network data packet; s4, calculating a target position of the ROCE data packet; s5, calculating an incremental value of the Fetch and Add; s6, the copied message is converted into an ROCE message; s7, the collector receives the RDMA message; and S8, inquiring or printing the key value information by the collector.

Description

Technical Field

[0001] The present invention relates to the technical field of high-performance networks and network telemetry technology, and in particular to a network flow size measurement solution based on the ROCE protocol. Background Art

[0002] In a network environment dominated by the traditional TCP / IP five-layer protocol, most upper-layer applications implement data interaction through transport layer sessions. Since the transport layer depends on the network layer and the data link layer, slight changes in delay or packet loss during network transmission may affect latency-sensitive upper-layer applications such as real-time audio and video calls and online games, resulting in poor user experience. Therefore, the importance of network measurement has become increasingly prominent. Network measurement provides key information for congestion control, network operation, and anomaly detection in data centers and backbone networks. Sketch, as a data structure based on hash tables, is highly favored in network measurement. Compared with other network measurement methods, Sketch not only has higher accuracy and speed, but also can effectively compress large amounts of data. This method can store huge amounts of traffic feature information in a smaller memory space while maintaining high estimation accuracy. Sketch has attracted much attention due to its excellent memory efficiency and accurate estimation of traffic measurements. However, the Sketch data structures studied in existing studies are mainly divided into two types. One is the simple and easy-to-use Sketch represented by Count-Min Sketch, but their accuracy is poor because they do not match the actual highly skewed network traffic: most of them are small flows, but a small number of large flows occupy 80% of the bandwidth in the network. The other is the complex Sketch represented by Elastic Sketch and ASketch, which improve the accuracy, but at the cost of using additional data structures to record additional information, such as flow id and flags. Therefore, it is important to implement a simple and accurate sketch.

[0003] Additionally, as telemetry requirements become more granular, the amount of data sent to collectors is increasing, and scaling data collection systems is becoming increasingly difficult. In practice, a switch can generate up to millions of telemetry reports per second, and a data center network can contain thousands of switches. Furthermore, the amount of data continues to grow with larger networks and higher line rates. Existing research has improved the scalability of data collection by improving the collector's network stack, by aggregating and filtering data on switches, or by reducing the amount of information output through collaboration between switches. However, collectors can easily become CPU or memory bound due to the need to perform data processing (i.e., I / O, parsing, and data insertion) for each incoming report.

[0004] The host CPU where Sketch is deployed needs to process a large number of data packets before compressing the message data into Sketch. The application needs to be copied through the kernel and context switched between user mode and kernel mode. Therefore, when processing network data packets, the host side consumes a lot of CPU resources and has low performance. In addition, for the sketch data structure of the telemetry collector, it is difficult to strike a balance between simple structure, accurate measurement, and memory utilization. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a network flow size measurement solution based on the ROCE protocol. The solution offloads the tasks of packet processing and counter position mapping on the end side to the P4 switch, and then sends the flow information to the queryable storage structure of the collector through the ROCE protocol. This process does not require the participation of the collector CPU, which greatly reduces the network latency.

[0007] (II) Technical solution

[0008] To achieve the above object, the present invention provides the following technical solution: a network flow size measurement solution based on the ROCE protocol, comprising the following steps:

[0009] S1. Create a Sketch data structure in the collector, denoted as RT Sketch; initialize RDMA configuration;

[0010] S2. Construct and send data packets;

[0011] S3. The switch processes the network data packet;

[0012] S4. Calculate the target location of the ROCE data packet;

[0013] S5. Calculate the incremental value of Fetch and Add;

[0014] S6. Convert the replicated message into a ROCE message;

[0015] S7. The collector receives the RDMA message;

[0016] S8. The collector queries or prints key value information.

[0017] Furthermore, the present invention has improved the RT Sketch data structure in S1: the experimental setting has 64 8-byte slots per layer, with a total of three layers; each counter of the first layer Sketch is 4 bytes; each counter of the second layer Sketch is 2 bytes; each counter of the third layer Sketch is 1 byte.

[0018] Furthermore, the present invention has the following improvements: the contents initialized in S1 include: creating an RDMA event channel, binding an RDMAID and an address, configuring a completion queue of an RDMA connection, and creating a queue pair.

[0019] Furthermore, the present invention has improved methods of constructing and sending data packets in S2, including: the sending end server constructs and sends data packets using the Scapy network tool library, constructs data packets of different protocols by writing a python program, and changes the five-tuple information of each generated data packet to achieve the purpose of emulating different data flows in the network.

[0020] Furthermore, the present invention improves the method for the S3 switch to process network data packets, including: at Ingress, for TCP or UDP data packets entering the switch, the switch forwards them to corresponding ports according to the flow table, and replicates the data packets through multicast operations.

[0021] Further, the present invention is improved in that the method of calculating the target position of the ROCE data packet in S4 includes: after the data packet enters the Egress of the switch, the five-tuple information of the data packet and the multicast packet sequence number are combined by the CRC32 algorithm to perform hash calculation to obtain a hash value;

[0022] The calculated hash value is modulo the number of counting slots in each layer of RT Sketch to obtain the offset of the data stream in each layer in RT Sketch. The specific storage location of the stream size information is obtained through the memory starting address and position offset.

[0023] The five-tuple information includes: source IP, destination IP, protocol number, source port, and destination port.

[0024] Further, the present invention is improved in that the method of converting the replicated message into the ROCE message in S6 includes: constructing a global routing header, a basic transmission header and an atomic extended transmission header;

[0025] Then the target address value and the increment value are assigned to the atomic extended transmission header, and finally the ROCE message is forwarded from the specified port to the collector end.

[0026] Furthermore, the present invention has improved the method of the S7 collector receiving RDMA messages, comprising: the switch sends the RDMA message to the collector, the collector adopts a direct asynchronous I / O method, and after the RDMA message is collected by the network card on the collector side, the DMA controller extracts the valid data of the RDMA message and transmits it to the DMA memory storage area of ​​the collector according to the target address field.

[0027] (III) Beneficial effects

[0028] Compared with the prior art, the present invention provides a network flow size measurement scheme based on the ROCE protocol, which has the following beneficial effects: (1) RDMA communication reduces the burden on the CPU of the network telemetry collector for processing packets and write operations, and these aspects greatly reduce the latency overhead and CPU load of data processing.

[0029] (2) Design the RT Sketch data structure, change the original fixed 8-byte counter size to a variable-length counter size, maximize the use of the memory storage space of the collector, and ensure the accuracy of counting.

[0030] (3) Offload most of the work of the collector to the P4 programmable switch, such as multicasting and replicating data packets, calculating the accurate storage location of data through hashing, and converting TCP / UDP packets into ROCE packets.

[0031] (4) Impose the minimum hardware resource overhead on the switch. In an environment such as a data center, it is not necessary for each switch to maintain an expensive RDMA connection with the collector, and only the edge switch needs to establish an RDMA connection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the RT Sketch slot of the present invention;

[0033] Figure 2 Processing flow of the data packet of the present invention in the switch;

[0034] Figure 3 Flowchart of the RT Sketch algorithm of the present invention;

[0035] Figure 4 Schematic diagram of the ROCE packet format of the present invention;

[0036] Figure 5 Schematic diagram of the collector slot information printing of the present invention;

[0037] Figure 6 Packet sending situation of the sending server of the present invention;

[0038] Figure 7 Demonstration of the collector flow size information query function of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In order to solve the above technical problems, the present invention designs a network flow size calculation solution based on the ROCE protocol, and establishes a Sketch data structure in the collector, namely Remote TowerSketch (hereinafter referred to as RT Sketch), which is used to collect network-wide flow size information. The technical solution of the present invention includes the following steps:

[0041] S1: RDMA configuration initialization

[0042] Before RDMA communication, the collector needs to do some initialization work to establish an RDMA connection with the switch, such as creating an RDMA event channel, RDMA ID and binding address, configuring the completion queue of the RDMA connection, and creating queue pairs. Then the collector will send the metadata information required for RDMA communication to the P4 switch, including the starting address of the data storage area, queue pair number, rkey, global and local ID, etc. Through these metadata, the switch can hash and calculate the accurate data storage location and send the RDMA information to the collector.

[0043] S2: Construct and send data packets

[0044] The sending server uses the Scapy network tool library to construct and send data packets. By writing a Python program, data packets of different protocols (UDP / TCP) can be constructed. By changing the five-tuple information of each generated data packet, the purpose of imitating different data flows in the network can be achieved.

[0045] S3: The switch processes the network data packets

[0046] In Ingress, for TCP or UDP data packets entering the switch, the switch will forward them to the corresponding port according to the flow table, and copy the data packets through multicast operations. The number of data packets copied is related to the number of layers of the RT Sketch data structure. The experiment of the present invention is a three-layer Sketch, and three data packets are copied based on the original message to perform incremental operations on the corresponding data of different layers of the RT Sketch.

[0047] S4: Calculate the target location of the ROCE packet

[0048] After the data packet enters the Egress of the switch, the CRC32 algorithm is used to combine the five-tuple information of the data packet (source IP, destination IP, protocol number, source port, destination port) and the multicast packet sequence number to perform hash calculation to obtain a hash value. The calculated hash value is modulo the number of counting slots at each layer of the RT Sketch to obtain the offset of the data flow at each layer in the RT Sketch. The specific storage location of the flow size information can be obtained through the memory start address and position offset.

[0049] S5: Calculate the incremental value of Fetch and Add

[0050] Since the Fetch and Add primitive of RDMA can only operate 64-bit data, the low latency of RDMA communication is guaranteed. However, the distribution of network flows in real traffic is extremely uneven. In a certain measurement cycle, most flows are small flows, which may only have a few data packets, while the largest network flow may contain tens of thousands of data packets. If the number of bits of each counter is 64 bits, the memory utilization will be extremely low. In order to solve this problem, the storage area occupied by each layer of counters in RT Sketch is the same, but the number of counters is different. Therefore, the present invention changes the increment value of the Fetch and Add primitive to distinguish different counters belonging to the same 64-bit slot.

[0051] S6: Convert the replicated message to a ROCE message

[0052] After completing the calculation of the target location and the increment value, the replicated message copy needs to be converted into a ROCE message to complete the RDMA communication and achieve the purpose of reducing latency. First, it is necessary to construct the global routing header (GRH), basic transmission header (BTH) and atomic extended transmission header (AETH), and then assign the target address value and the increment value to the corresponding fields of the atomic extended transmission header. Finally, the ROCE message is forwarded from the specified port to the collector.

[0053] S7: The collector receives the RDMA message

[0054] The switch sends the RDMA message to the collector, which uses direct asynchronous I / O. After the RDMA message is collected by the network card on the collector side, the DMA controller extracts the valid data of the RDMA message and transmits it to the DMA memory storage area of ​​the collector according to the target address field. This process does not require CPU participation and reduces the kernel copy process, greatly reducing the transmission delay of the data packet.

[0055] S8: Collector queries or prints key value information

[0056] The user enters the value of the quintuple in the terminal, and the CRC32 algorithm is used to calculate the memory storage location of the corresponding stream based on the combination of the quintuple key and the multicast packet sequence number. Since RT Sketch has three layers of counters, the location of each layer of hash calculation is different, so it is necessary to calculate the hash value three times, take out the stream size information corresponding to the three locations, take the minimum value of the three as the final query value, and print the value. The multi-layer counter method can minimize the impact of hash conflicts. In addition, the user can also enter "print" to let the system print the data information of all counters.

[0057] Example:

[0058] Figure 1 The figure is a schematic diagram of the processing flow of user data messages passing through the P4 switch in the present invention. Figure 2 : is a schematic diagram of the RT Sketch data structure of the collector in the present invention, Figure 3 This is the RT Sketch algorithm flow chart of the present invention. Figure 1 , Figure 2 and Figure 3 The present invention is further described, and the specific steps are as follows:

[0059] Step 1: Before RDMA communication, the collector needs to do some initialization work to establish an RDMA connection with the switch, such as creating an RDMA event channel, RDMA ID and binding address, configuring the completion queue of the RDMA connection, and creating queue pairs. Then the collector will send the metadata information required for RDMA communication to the P4 switch, including the starting address of the data storage area, queue pair number, rkey, global and local ID, etc. Through these metadata, the switch can hash and calculate the accurate data storage location and send the RDMA information to the collector.

[0060] Step 2: The sending server uses the Scapy network tool library to construct and send data packets. By writing a python program, data packets of different protocols (UDP / TCP) can be constructed, and the source port number in the five-tuple can be changed randomly within the range of 1530-1540 to achieve the purpose of simulating different data flows in the network. Of course, the present invention supports the measurement of flow size information of all different five-tuples, and can change the source IP, destination IP, destination port and other information of each generated data message, but in order to reduce the complexity of the experiment, the experiment of the present invention only changes the source port number of the five-tuple.

[0061] Step 3: If Figure 1As shown, in Ingress, for TCP or UDP data packets entering the switch, the switch will forward them to the corresponding port according to the flow table, and copy the data packets through multicast operations. The number of data packets copied is related to the number of layers of the RT Sketch data structure. The experiment of the present invention is a three-layer Sketch, and three data packets are copied based on the original message to perform incremental operations on the corresponding data of different layers of the RT Sketch.

[0062] Step 4: After the data packet enters the Egress of the switch, the switch will calculate the destination slot address of the packet. The algorithm flow chart is as follows: Figure 3 As shown. First, the CRC32 algorithm is used to hash the five-tuple information of the data packet (source IP, destination IP, protocol number, source port, destination port) and the multicast packet sequence number pkt_num to obtain a hash value. The calculated hash value is modulo the number of counting slots in each layer of RT Sketch to obtain the offset slot_offset of each layer of the data stream in RTSketch. Then the low bit of slot_offset is taken as the counter sequence number of the same target slot, and the high bit is taken as the slot sequence number. The number of slots in each layer is the same, but the number of counters is different, such as Figure 2 As shown. An example can be given to illustrate: Assuming that the number of slots in each layer is 64, the total number of counters in the second layer is 64*4=256. When calculating the position of the second layer counter, assume that the value of slot_offset after modulus is 101, corresponding to 0b1100101 in binary. Since each slot in the second layer contains 4 counters, the lower two bits 01 are taken as the counter sequence number skecthid, and 11001 obtained by right shifting is taken as the slot index, that is, the 25th slot in the second layer. Finally, the specific storage location of the flow size information can be obtained through the memory start address and slot offset. For the P4 switch, in order to ensure the fast processing and forwarding of data packets, it adopts a pipeline structure inside. Such a structure only supports simple data processing in the switch, such as addition, shift, bit AND or operation, etc., and does not support division. In addition, for displacement, the displacement amount can only be a constant but not a variable. Therefore, the P4 programming of this algorithm has a certain complexity and needs to be completed by table entry matching.

[0063] Step 5: In order to maximize the use of the collector's memory area, the present invention proposes an RT Sketch data structure, such as Figure 2. Since the Fetch and Add primitive of RDMA can only operate 64-bit data, the low latency of RDMA communication is guaranteed. However, the distribution of network flows in real traffic is extremely uneven. In a certain measurement cycle, most flows are small flows. A flow may have only a few data packets, while the largest network flow may contain tens of thousands of data packets. If the number of bits of each counter is 64 bits, the memory utilization will be extremely low. In order to solve this problem, the number of bits of each layer of counters in RT Sketch is different. The first layer counter is 32 bits, the second layer is 16 bits, and the third layer is 8 bits. The storage area occupied by each layer of counters is the same, but the number of counters is different. Therefore, the present invention changes the increment value of the Fetch and Add primitive to distinguish different counters belonging to the same 64-bit slot. In the previous calculation, the present invention takes the low bit of slot_offset as the counter sequence number skecthid, and then selects the increment value based on the sequence number value. Here we take the example mentioned in step 4. If the sketchid is 01, then the increment value of the Fetch and Add primitive should be 1<<16. Because the same slot in the second layer contains 4 counters, each counter occupies 16 bits, the increment value can be expressed by the formula data=1<<(sketchid*weight[i]), where weight[i] represents the number of bits of each layer counter. In this way, Figure 2 The 16-bit counter corresponding to the data value of 37 is incremented by 1. Similarly, since there is no array in the P4 language, the above operation can only be completed through table entry matching.

[0064] Step 6: After completing the calculation of the target position and the increment value, the copied message copy needs to be converted into a ROCE message to complete the RDMA communication and achieve the purpose of reducing latency. The format of the ROCE message is as follows: Figure 4 As shown, it is first necessary to construct a global routing header (GRH), a basic transmission header (BTH) and an atomic extended transmission header (AETH) after the Ethernet header, and then assign the target address value and the increment value stored in the metadata to the corresponding fields of the atomic extended transmission header, and finally forward the ROCE message from the designated port to the collector end. The present invention uses the Fetch and Add primitive of RDMA to perform remote increment operations on the data of RT Sketch. In the opcode field of BTH, the operation type of RDMA can be specified, such as when opcode is 0x07, it indicates a Fetch and Add atomic operation. And the atomic operation needs to attach the AtomicETH header as part of the Payload, and AtomicETH carries the additional information required for the atomic operation, such as the operation type, the target address, and the operand.

[0065] Step 7: The switch sends the RDMA message to the collector. The collector uses direct asynchronous I / O. After the RDMA message is collected by the network card on the collector side, the DMA controller extracts the valid data of the RDMA message and transmits it to the DMA memory storage area of ​​the collector. This process does not require CPU participation and reduces the kernel copy process, greatly reducing the transmission delay of the data packet.

[0066] Step 8: The user can enter "query" in the terminal to query the flow size information, and then enter the quintuple to be queried, such as "192.168.1.1192.168.1.21540545117". The background will perform CRC32 calculation hash value based on the combination of the quintuple and the multicast packet sequence number (0-2), process the hash value and finally obtain the memory storage location of the flow information in each layer of Sketch. The algorithm process is the same as Figure 3 The data insertion process shown is similar. The collector is programmed in C++, which is more convenient and simpler than P4 programming. Since RT Sketch has three layers of counters, the location of each layer of hash calculation is different, so it is necessary to calculate the hash value three times, take out the flow size information corresponding to the three locations, take the minimum value of the three as the final query value, and print the value, such as Figure 7 As shown in the figure. Hash collisions may cause the corresponding values ​​to be too large in a certain layer, so the multi-layer Sketch method can minimize the impact of hash collisions. In addition, users can also enter "print" in the terminal to let the system display the data information of all counters, such as Figure 5 .

[0067] The present invention realizes the measurement of network flow size information by the collector based on the ROCE protocol. ROCE is an Ethernet-based RDMA technology. The RDMA technology can bypass the kernel and directly access the remote memory without the participation of the remote CPU, reducing the CPU intervention and context switching overhead, realizing CPU unloading, kernel bypass and zero-copy data transmission, and eliminating the computing task bottleneck caused by traditional TCP / IP network communication.

[0068] The present invention verifies the feasibility of the scheme and the accuracy of the measurement through experiments. Figure 2 The RT Sketch data structure of the collector is set in the experiment. Each layer has 64 8-byte slots, with a total of three layers. Each counter of the first layer Sketch is 4 bytes, that is, one slot can store two 4-byte data; each counter of the second layer Sketch is 2 bytes; each counter of the third layer Sketch is 1 byte. This method can maximize the use of the collector's memory storage area.

[0069] like Figure 6, the sender constructs and sends different data packets through SCAPY, changes the source port number to simulate different flows, and finally sends out 666 data packets, and counts the number of data packets of different flows (source port numbers). On the collector side, the flow size information of all counters can be displayed through "print", such as Figure 5 At the same time, it also supports querying the flow size information of a certain flow by entering "query". Figure 7 In the command output, enter the five-tuple information "192.168.1.1192.168.1.21537545117" to view the flow size with source port number 1537. The output result is 55, which is consistent with the statistical data of the sender, and the accuracy is verified.

[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A network flow size measurement solution based on the ROCE protocol, characterized in that: The following steps are involved: S1. Create a Sketch data structure in the collector, denoted as RT Sketch; initialize RDMA configuration; S2. Construct and send data packets; S3. The switch processes the network data packet; S4. Calculate the target location of the ROCE data packet; S5. Calculate the incremental value of Fetch and Add; S6. Convert the replicated message into a ROCE message; S7. The collector receives the RDMA message; S8. The collector queries or prints key value information.

2. According to a network flow size measurement solution based on ROCE protocol according to claim 1, it is characterized in that: The RT Sketch data structure in S1: the experimental setting has 64 8-byte slots per layer, with a total of three layers; each counter of the first layer Sketch is 4 bytes; each counter of the second layer Sketch is 2 bytes; each counter of the third layer Sketch is 1 byte.

3. According to a network flow size measurement solution based on ROCE protocol according to claim 1, it is characterized in that: The contents initialized in S1 include: creating an RDMA event channel, an RDMA ID and binding an address, configuring a completion queue of an RDMA connection, and creating a queue pair.

4. The network flow size measurement solution based on the ROCE protocol according to claim 1 is characterized in that: The method for constructing and sending data packets in S2 comprises: a sending end server constructs and sends data packets using a Scapy network tool library, constructs data packets of different protocols by writing a python program, and changes the five-tuple information of each generated data packet to achieve the purpose of emulating different data flows in the network.

5. The network flow size measurement solution based on the ROCE protocol according to claim 1 is characterized in that: The method for the S3 switch to process a network data packet includes: at Ingress, for a TCP or UDP data packet entering the switch, the switch forwards it to a corresponding port according to a flow table, and replicates the data packet through a multicast operation.

6. A network flow size measurement solution based on ROCE protocol according to claim 2, characterized in that: The method of calculating the target position of the ROCE data packet in S4 includes: after the data packet enters the Egress of the switch, the five-tuple information of the data packet and the multicast packet sequence number are combined by the CRC32 algorithm to perform hash calculation to obtain a hash value; The calculated hash value is modulo the number of counting slots in each layer of RT Sketch to obtain the offset of the data stream in each layer in RT Sketch. The specific storage location of the stream size information is obtained through the memory starting address and position offset. The five-tuple information includes: source IP, destination IP, protocol number, source port, and destination port.

7. A network flow size measurement solution based on ROCE protocol according to claim 6, characterized in that: The method of converting the replicated message into the ROCE message in S6 includes: constructing a global routing header, a basic transmission header and an atomic extended transmission header; Then the target address value and the increment value are assigned to the atomic extended transmission header, and finally the ROCE message is forwarded from the specified port to the collector end.

8. The network flow size measurement solution based on the ROCE protocol according to claim 1 is characterized in that: The method for the S7 collector to receive RDMA messages includes: the switch sends the RDMA message to the collector, the collector adopts a direct asynchronous I / O mode, and after the RDMA message is collected by the network card of the collector, the DMA controller extracts the valid data of the RDMA message and transmits it to the DMA memory storage area of ​​the collector according to the target address field.

Images