Container network flow control system based on XDP

Abstract

The invention discloses a container network flow control system based on an XDP, which relates to the field of flow control and comprises an analysis bandwidth limit high and low module, a bandwidth dynamic proportion adjustment module and a data packet filtering module. The bandwidth limit high-guarantee-low module is used for distinguishing a source container of a network data packet, associating an IP address with a retention value and a limit value provided by a user, and transmitting data to the bandwidth dynamic proportion adjustment module; the bandwidth dynamic proportion adjusting module is used for dynamically adjusting the bandwidth distributed to each container and transmitting the data to the data packet filtering module; the data packet filtering module is used for carrying out rule matching and network data packet processing on a network driving layer through an XDP program, an XDP context comprises all information needing to access a data packet in an original form, a data packet source is obtained by analyzing the context, and compared with an iptables / netfilter, messages can be recognized and discarded earlier, and therefore higher processing performance is achieved.

Description

Technical Field

[0001] The present invention relates to the field of flow control, and in particular to a container network flow control system based on XDP. Background Art

[0002] With the development of cloud computing and related virtualization technologies, virtualized data centers have gradually replaced traditional data centers. Virtualization technology pools the physical resources of data centers, allowing users to flexibly obtain resources from the resource pool. Containers and virtual machines are two different virtualization technologies that are used to isolate applications and their required operating environments to build a set of independently running units. However, these two virtualization technologies are significantly different.

[0003] Virtual machines are a hardware layer abstraction technology. Each virtual machine has its own separate virtual hardware, kernel, and user space. It uses the "virtual machine manager" Hypervisor to run directly on the host physical device and use Hypervisor to centrally and efficiently manage all virtual environments. These virtual environments are completely independent of each other and have strong isolation. However, each virtual machine will occupy a large amount of host resources. For example, a virtual machine often occupies several GB of memory space. To run an application on a virtual machine, the Guest OS of the virtual machine and all the virtual hardware required to support the operation of the Guest OS must also be run at the same time, which greatly increases the RAM and CPU overhead. Containers, as another virtualization technology, perform resource abstraction on top of the host operating system, and all containerized applications share the host operating system.

[0004] Compared with virtual machines, containers share the kernel of the host machine, abstract at a higher level, are more lightweight, and reduce the overhead of the virtual machine running GuestOS; however, since containers share the operating system and kernel with the host, they cannot provide strong hardware isolation and resource management; Docker, as the most popular engine that supports the creation and use of Linux containers, uses the CGroup mechanism to provide memory, storage and other resource allocation for containers; CGroup manages and controls the behavior of processes using system resources in the form of groups, and the specific strategies are completed through various subsystem modules, each subsystem is responsible for the management and control of one type of resource; unfortunately, the current framework lacks management and control of network resources; therefore, it is extremely necessary to develop a network traffic control system for containers.

[0005] Most of the current mainstream traffic control research is based on the original network traffic control framework in the operating system, Linux Traffic Control (TC). TC implements traffic control by setting queue rules for data packets at input and output. However, TC operations are performed in the kernel protocol stack, resulting in relatively high CPU usage. In addition, traditional TC is static in terms of network resource configuration.

[0006] However, containers have the characteristics of short life cycle, large number, frequent changes, and fierce resource competition. The static configuration of TC cannot be well applied to the dynamically changing container environment. XDP (eXpress Data Path), that is, the fast data path, is located in the network card driver layer and can process network data packets before they reach the protocol stack. The location of XDP and TC in the kernel is as follows: Figure 2 As shown in the figure, compared with TC, XDP has higher processing performance and greater flexibility.

[0007] Aiming at the shortcomings of the existing network traffic control process, a solution is proposed. Summary of the invention

[0008] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art. The present invention aims to explore an efficient and flexible container network traffic control mechanism based on XDP technology. Specifically, the present invention dynamically adjusts the bandwidth allocated to each container based on historical data transmission characteristics. On the one hand, it meets the minimum demand of each container for network resources. On the other hand, in order to prevent network IO intensive containers from excessively occupying network resources, the present invention will also limit the upper limit of network resource occupation by each container. In addition, within the upper and lower limits, the present invention achieves fair allocation of bandwidth. Experimental results show that compared with the traditional TC-based solution, when network IO intensive container applications and non-network IO intensive container applications coexist, this work greatly optimizes the average completion time of non-network IO intensive container applications. Finally, in a containerized environment, a large number of containers with different security levels or different uses coexist on the same host. In order to avoid malicious traffic in some containers from occupying network bandwidth, this paper implements a high-performance packet filtering mechanism based on XDP, and compares it with the mainstream i ptab l es / netfilter mechanism to verify the efficiency of the filtering mechanism implemented in this paper.

[0009] The object of the present invention can be achieved by the following technical solutions: a container network traffic control system based on XDP, including a bandwidth high limit and low guarantee module, a bandwidth dynamic ratio adjustment module and a data packet filtering module;

[0010] The bandwidth high limit and low limit module is used to distinguish the source container of the network data packet, associate the IP address with the reserved value and limit value provided by the user, and transmit the data to the bandwidth dynamic ratio adjustment module;

[0011] The bandwidth dynamic ratio adjustment module is used to dynamically adjust the bandwidth allocated to each container and transmit the data to the data packet filtering module;

[0012] The packet filtering module is used to perform rule matching and network packet processing at the network driver layer through the XDP program.

[0013] Preferably, the specific contents of the bandwidth limit and low-limit module are as follows:

[0014] The XDP context contains all the information needed to access the data packet in its original form. The source of the data packet is obtained by analyzing its context. The context of the XDP program is a structure named xdp_md. In this structure, data and data_end are the addresses of the packet header and tail, respectively. In the packet header, each field is of fixed length. By parsing the data field in the context, the location of the field storing the source IP address can be inferred.

[0015] After obtaining the IP address, the IP address is associated with the reserved value and limit value provided by the user. The information in the BPF MAP is shared by the kernel-mode eBPF program and the user-mode program of the loader. The application and the eBPF hook program communicate through the sharing feature of the BPF MAP, so that the user-space program calls and creates the BPF MAP through the 'bpf()' system call, where the MAP uses the IP address as the key and the reserved value and limit value as the value, and stores the mapping relationship between the IP, reserved value and limit value, so that the eBPF program obtains the relevant mapping relationship by reading the BPFMAP. After allocating the reserved bandwidth of each container, the algorithm is adjusted in proportion to allocate the remaining bandwidth; when allocating the remaining bandwidth, it is compared with the limit value entered by the user. If it exceeds the limit value, the excess bandwidth is recorded; the excess bandwidth is proportionally distributed to the containers that have not reached the upper limit, and the operation is repeated until all the excess bandwidth is allocated or each container reaches the upper limit.

[0016] Preferably, the adjustment process of the bandwidth dynamic ratio adjustment module is as follows:

[0017] When it changes over time, the minimum bandwidth requirement of the container is guaranteed, and the maximum bandwidth usage is limited. The bandwidth allocated to each container is still dynamically adjusted according to the container demand. The "heat value" of each container is calculated based on the historical transmission volume and the distance from the current time by adopting the heat sorting algorithm. The container with low heat indicates that it has a small demand for network IO resources. Bandwidth is allocated to containers with small resource requirements first to avoid such containers being blocked by network IO intensive, that is, containers with high heat, to ensure their service quality QoS. When the QoS of the network IO intensive container remains basically unchanged, the QoS of the non-network IO intensive container is maximized, and the number of reserved tokens that can be allocated to the container is calculated when the total number of tokens is N.

[0018] Preferably, the filtering process of the data packet filtering module is as follows:

[0019] Network data packets are processed at the network driver layer processing stage through XDP. The packet filtering solution based on XDP reduces the CPU overhead occupied when processing data packets, and uses the XDP program to implement rule matching and data packet processing. By parsing the IP address and data packet processing rules in the user input command, the IP and processing rules are stored in BPFMAPi p_map. Whenever a data packet is received, the data packet IP is parsed from the XDP context, and the corresponding processing rules are found through i p_map to accept, reject or forward it. While maintaining the QoS of network IO-intensive container applications basically unchanged, the QoS of non-network IO-intensive containers is greatly optimized.

[0020] The beneficial effects of the present invention are as follows:

[0021] The capacity network traffic control system designed by the present invention greatly optimizes the QoS of non-network IO intensive container applications in multi-container scenarios. In addition, compared with the traditional i ptab l es / netfilter framework, the packet filtering mechanism designed by the present method based on XDP technology shows a higher data packet processing speed and lower single packet processing overhead.

[0022] In this experiment, two containers C1 and C2 are deployed on a virtual machine, running 500 IO-intensive connections and 500 non-IO-intensive connections respectively (both based on the TCP protocol). Figure 6 The average latency and tail latency of the two connections are shown (the 90th percentile latency is used as the tail latency in this experiment). It can be seen that after using this flow control algorithm, the average latency / tail latency of IO-intensive connections only increased by 18% / 10%, while the average latency / tail latency of non-IO-intensive connections only decreased by 75% / 89%. In other words, the user experience of non-IO-intensive applications is greatly optimized while the experience of IO-intensive applications remains almost unchanged.

[0023] In addition, this solution tests the processing performance of the IPTables / Netfilter framework and the XDP-based packet filtering module mechanism developed in this project. Specifically, the remote virtual machine runs a container application that continuously sends packets to the local machine, and the packet processing speed and CPU usage of the two filtering tools are tested. Figure 7 The data packet processing rates of the two tools are shown: it can be seen that compared with the IPtabs / Netfilter framework, the solution in this paper improves the processing speed by 38%, verifying the efficiency of XDP technology.

[0024] Finally, this solution also tests the CPU usage under the two mechanisms. Figure 8 The results show that the CPU usage of this solution is 6% to 30% lower than that of the iPtab les / netfilter framework. This is because the iPtab les / netfilter framework is located in the kernel network protocol stack. Even if the message is discarded at the earliest hook point PREROUTING in the packet receiving path, it has already experienced the expensive Linux network protocol stack processing overhead. XDP is located in the network card driver layer. The data packet can be processed by the XDP program before the socket buffer is allocated. Compared with iPtab les / netfilter, it can identify and discard the message earlier, so it has higher processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below in conjunction with the accompanying drawings;

[0026] Figure 1 It is a flowchart of the system of the present invention;

[0027] Figure 2 It is a schematic diagram of the system architecture of this method;

[0028] Figure 3 It is the XDP context content description diagram;

[0029] Figure 4 It is the BPF MAP function diagram;

[0030] Figure 5 It is a rule matching flowchart;

[0031] Figure 6 Optimize graphs for average and tail latency;

[0032] Figure 7 Rate optimization diagram for the packet filtering module;

[0033] Figure 8 Optimize graph for CPU usage. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Example:

[0036] See also Figures 1 to 8 As shown, the present invention is a container network traffic control system based on XDP, including a bandwidth high limit and low guarantee module, a bandwidth dynamic ratio adjustment module and a data packet filtering module; the bandwidth high limit and low guarantee module is used to distinguish the source container of the network data packet, and associate the IP address with the reserved value and limit value provided by the user, and transmit the data to the bandwidth dynamic ratio adjustment module; the bandwidth dynamic ratio adjustment module is used to dynamically adjust the bandwidth allocated to each container, and transmit the data to the data packet filtering module; the data packet filtering module is used to perform rule matching and network data packet processing at the network driver layer through the XDP program;

[0037] The specific contents of the bandwidth limit and low-limit module are as follows:

[0038] The XDP context contains all the information needed to access the data packet in its original form. The source of the data packet is obtained by analyzing its context. The context of the XDP program is a structure named xdp_md. In the structure, data and data_end are the addresses of the data packet header and tail, respectively. In the data packet header, each field is of fixed length. By parsing the data field in the context, the location of the field storing the source IP address is inferred. To ensure that the bandwidth of each container is within the range of the reserved value and the limit value set by the user, the method uses a token bucket algorithm to perform network traffic shaping and rate limiting. At the beginning, the total number of available tokens N is set, and one token is consumed for processing one data packet. After each second, the number of tokens is reset to N to limit the maximum processing rate to N pps (in order to effectively utilize the remaining bandwidth, the present invention adopts a "moderate over-issuance of tokens" strategy, that is, the number of tokens issued N is slightly greater than the processing capacity of the network card). First, define reserv_map to store the reserved value information. Specifically, the MAP uses the IP address as the key and the reserved value as the value. When the program is running, first traverse the MAP reserv_map storing the reserved value to store the total of all reserved values. R, based on which the proportion of each container's retention value to the total retention value is calculated:

[0039] P i =R i / total R

[0040] Wherein, Ri represents the retention value of the i-th container, and Pi represents the proportion of the retention value of the i-th container;

[0041] After obtaining the IP address, the IP address is associated with the reserved value and limit value provided by the user. The information in the BPF MAP is shared by the kernel-mode eBPF program and the user-mode program of the loader. The application and the eBPF hook program communicate through the sharing feature of BPFMAP, so that the user-space program can call and create BPFMAP through the 'bpf()' system call, where MAP uses the IP address as the key and the reserved value and limit value as the value, and stores the mapping relationship between the IP, reserved value and limit value, so that the eBPF program can obtain the relevant mapping relationship by reading BPFMAP;

[0042] Token i =N*P i

[0043] If Token i (the number of reserved tokens allocated to the i-th container) is greater than the reserved value stored in reserv_map, the reserved value in reserv_map is updated to Token i, otherwise, the reserved value in reserv_map remains unchanged; after allocating the reserved bandwidth of each container, the algorithm is adjusted in proportion to allocate the remaining bandwidth; when allocating the remaining bandwidth, it is compared with the limit value entered by the user. If it exceeds the limit value, the excess bandwidth is recorded; the excess bandwidth is proportionally distributed to the containers that have not reached the upper limit, and the operation is repeated until all the excess bandwidth is allocated or each container reaches the upper limit value;

[0044] The adjustment process of the bandwidth dynamic ratio adjustment module is as follows: when it changes over time, the minimum bandwidth requirement of the container is guaranteed, and after limiting the maximum bandwidth usage, the bandwidth allocated to each container is still adjusted dynamically according to the container demand. By adopting the heat sorting algorithm, the "heat value" of each container is calculated based on the historical transmission volume and the distance from the current time. The container with low heat indicates that it has a small demand for network IO resources, and bandwidth is allocated to containers with small resource requirements first. When the QoS of the network IO intensive container remains basically unchanged, the QoS of the non-network IO intensive container is maximized; then, the number of reserved tokens that can be allocated to the container is calculated when the total number of tokens is N:

[0045] Secondly, this solution dynamically adjusts the bandwidth occupied by each container based on the Hacker News popularity ranking algorithm; HackerNews is a news submission community under Y Combinator, and the news ranking algorithm it uses is called the Hacker News ranking algorithm; the algorithm formula is as follows:

[0046] Score=P / (T+2)G

[0047] Score represents the calculated heat value, P represents the number of data packets sent by the container in one cycle; T represents the index of the cycle. This project stores the data packet transmission statistics of the past 10 cycles, from recent to far, with indexes of 1, 2, 3...10, and adding 2 is to prevent the denominator from being too small, resulting in a large calculation result; G represents the "gravity factor", that is, how fast the heat decreases over time; the present invention uses 10s as a cycle, analyzes the data transmission characteristics of the past 10 cycles, and takes the sum of the heat values ​​of the 10 cycles as the current heat value of the container;

[0048] The filtering process of the packet filtering module is as follows: the network data packets are processed by XDP in the network driver layer processing stage. The packet filtering solution based on XDP reduces the CPU overhead occupied when processing data packets, and uses the XDP program to implement rule matching and data packet processing. By parsing the IP address and data packet processing rules in the user input command, the IP and processing rules are stored in BPFMAPip_map. Whenever a data packet is received, the data packet IP is parsed from the XDP context, and the corresponding processing rules are searched through i p_map to accept, reject or forward it; based on the obtained heat value, the bandwidth allocated to each container is dynamically adjusted; the ratio calculation formula is as follows:

[0049]

[0050] Pi represents the proportion of the i-th container, and Score i represents the heat of the i-th container. By taking the square root of the heat, we can prevent some extremely network IO-intensive applications from blocking non-network IO-intensive applications, while ensuring that IO-intensive containers obtain relatively higher bandwidth. Based on this algorithm, we can significantly optimize the QoS of non-network IO-intensive containers while maintaining the QoS of network IO-intensive container applications basically unchanged.

[0051] The above formulas are obtained by collecting a large amount of data for software simulation and selecting a formula that is close to the real value. The coefficients in the formula are set by technical personnel in this field according to actual conditions. The above are only preferred specific implementations of the present invention, but the protection scope of the present invention is not limited to this. Any technical personnel familiar with the technical field within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, make equivalent replacement or change, which should be covered within the protection scope of the present invention.

Claims

1. A container network traffic control system based on XDP, characterized in that: include: Bandwidth high limit and low guarantee module, bandwidth dynamic ratio adjustment module and data packet filtering module; The bandwidth high limit and low limit module is used to distinguish the source container of the network data packet, associate the IP address with the reserved value and limit value provided by the user, and transmit the data to the bandwidth dynamic ratio adjustment module; The bandwidth dynamic ratio adjustment module is used to dynamically adjust the bandwidth allocated to each container and transmit the data to the data packet filtering module; The packet filtering module is used to perform rule matching and network packet processing at the network driver layer through the XDP program.

2. The XDP-based container network traffic control system according to claim 1, characterized in that: The specific contents of the bandwidth limit high and low guarantee module are as follows: The XDP context contains all the information needed to access the data packet in its original form. The source of the data packet is obtained by analyzing its context. The context of the XDP program is a structure named xdp_md. In this structure, data and data_end are the addresses of the packet header and tail, respectively. In the packet header, each field is of fixed length. By parsing the data field in the context, the location of the field storing the source IP address can be inferred. After obtaining the IP address, the IP address is associated with the reserved value and limit value provided by the user. The information in the BPF MAP is shared by the kernel-mode eBPF program and the user-mode program of the loader. The application and the eBPF hook program communicate through the sharing feature of the BPF MAP, so that the user-space program calls and creates the BPF MAP through the 'bpf()' system call, where the MAP uses the IP address as the key and the reserved value and limit value as the value, and stores the mapping relationship between the IP, reserved value and limit value, so that the eBPF program can obtain the relevant mapping relationship by reading the BPF MAP.

3. The XDP-based container network traffic control system according to claim 1, characterized in that: The adjustment process of the bandwidth dynamic ratio adjustment module is as follows: When it changes over time, the minimum bandwidth requirement of the container is guaranteed, and the maximum bandwidth usage is limited. The bandwidth allocated to each container is still dynamically adjusted according to the container demand. The "heat value" of each container is calculated based on the historical transmission volume and the distance from the current time by adopting the heat sorting algorithm. The container with low heat indicates that it has a small demand for network IO resources. Bandwidth is allocated to containers with small resource requirements first. When the QoS of network IO intensive containers remains basically unchanged, the QoS of non-network IO intensive containers is maximized.

4. The XDP-based container network traffic control system according to claim 1, characterized in that: The filtering process of the packet filtering module is as follows: Network data packets are processed through XDP in the network driver layer processing stage. The packet filtering solution based on XDP reduces the CPU overhead occupied when processing data packets, and uses the XDP program to implement rule matching and data packet processing. By parsing the IP address and data packet processing rules in the user input command, the IP and processing rules are stored in BPFMAPip_map. Whenever a data packet is received, the data packet IP is parsed from the XDP context, and the corresponding processing rules are found through ip_map to accept, reject or forward it.

Images