Xdp-based container network traffic control system

Abstract

The application discloses a container network traffic control system based on XDP and relates to the field of traffic control, which comprises a bandwidth limit-high and low-preservation module, a bandwidth dynamic proportion adjustment module and a data packet filtering module; the bandwidth limit-high and low-preservation module is used for distinguishing the source containers of network data packets, associating IP addresses with the reservation values and limit values provided by users, and transmitting data to the bandwidth dynamic proportion adjustment module; the bandwidth dynamic proportion adjustment module is used for dynamically adjusting the bandwidth allocated to each container and transmitting data to the data packet filtering module; and the data packet filtering module is used for performing rule matching and network data packet processing in the network driver layer through an XDP program, wherein an XDP context contains all information required for accessing data packets in the original form, the source of a data packet is acquired by analyzing the context, compared with iptables / netfilter, the application can recognize and discard messages earlier, and therefore has higher processing performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of flow control, in particular to a container network flow control system based on XDP. BACKGROUND

[0002] With the development of cloud computing and related virtualization technology, virtualization data center gradually replaces the traditional data center; virtualization technology pools the physical resources of the data center, so that users can flexibly obtain resources from the resource pool; containers and virtual machines are two different virtualization technologies, both of which are used to isolate application programs and their required running environments, and build a set of independently running units, but the two virtualization technologies are significantly different;

[0003] Virtual machine is a hardware layer abstraction technology, each virtual machine has its own virtual hardware, kernel, user space, and uses "virtual machine manager" Hypervisor to directly run on the host physical device, and uses Hypervisor to centrally and efficiently manage all virtual environments; these virtual environments are completely independent of each other and have strong isolation; however, each virtual machine will occupy a large amount of host resources, for example, a virtual machine often occupies several GB of memory space; running an application program on a virtual machine also needs to run the Guest OS of the virtual machine and all virtual hardware required to support the running of the Guest OS, which greatly increases the RAM and CPU overhead; and containers, as another virtualization technology, abstract resources on the host operating system, and all containerized applications share the host operating system;

[0004] Compared with virtual machines, containers share the host's kernel and abstract at a higher level, which is more lightweight and reduces the overhead of running GuestOS in virtual machines; however, since containers share the operating system and kernel with the host, they cannot provide strong hardware isolation and resource management; Docker, as the most popular engine supporting the creation and use of Linux containers, uses the CGroup mechanism to allocate resources such as memory and storage for containers; CGroup manages and controls the behavior of processes using system resources in groups, and the specific strategy is completed by various subsystem modules, each of which is responsible for managing and controlling a type of resource; Unfortunately, the current framework lacks management and control of network resources; therefore, it is extremely necessary to develop a network flow control system for containers.

[0005] Most of the current mainstream traffic control research is based on the original network traffic control framework in the operating system, Linux Traffic Control (TC);TC achieves traffic control by setting the queue rules of data packets when input and output;But the operation of TC is executed in the kernel protocol stack, resulting in relatively high CPU usage;In addition, the traditional TC is static in network resource configuration;

[0006] And the container has the characteristics of short life cycle, large quantity, frequent change and fierce resource competition, and the static configuration of TC cannot be well applied to the dynamically changing container environment;XDP (eXpress Data Path) is located in the network card driver layer and can process before the network data packet reaches the protocol stack;XDP and TC are located in the kernel as shown in Figure 2 Compared with TC, XDP has higher processing performance and stronger flexibility;

[0007] In view of the deficiencies in the existing network traffic control process, a solution is proposed. SUMMARY

[0008] The purpose of the present application is to overcome the above-mentioned deficiencies in the prior art, and the present application aims to explore an efficient and flexible container network traffic control mechanism based on XDP technology;Specifically, the present application dynamically adjusts the bandwidth allocated to each container based on historical data transmission characteristics, on the one hand to meet the minimum demand of each container for network resources, and on the other hand to prevent network IO intensive containers from occupying too much network resources, the present application also limits the upper limit of network resource occupation of each container;In addition, within the upper and lower limits, the present application realizes the fair allocation of bandwidth;Experimental results show that, compared with the traditional TC-based scheme, when network IO intensive container applications and non-network IO intensive container applications coexist, the present work greatly optimizes the average completion time of non-network IO intensive container applications;Finally, in the container environment, a large number of containers with different security levels or different purposes coexist on the same host;In order to avoid malicious traffic in some containers from occupying network bandwidth, this paper realizes a high-performance packet filtering mechanism based on XDP, and compares it with the mainstream iptables / netfilter mechanism to verify the efficiency of the filtering mechanism realized in this paper.

[0009] The purpose of the present application can be achieved by the following technical solutions: a container network traffic control system based on XDP, comprising a bandwidth limit high and low preservation module, a bandwidth dynamic proportional adjustment module and a data packet filtering module;

[0010] The bandwidth limit high and low module is used to distinguish the source container of network data packets, associate IP addresses with the reservation value and limit value provided by users, and transmit data to the bandwidth dynamic proportion adjustment module.

[0011] The bandwidth dynamic proportion adjustment module is used to dynamically adjust the bandwidth allocated to each container and transmit data to the data packet filtering module.

[0012] The data packet filtering module is used to perform rule matching and network data packet processing at the network driver layer through the XDP program.

[0013] Preferably, the specific content of the bandwidth limit high and low module is as follows:

[0014] The XDP context contains all the information needed to access the data packet in its original form, and the data packet source is obtained by analyzing its context, wherein the context of the XDP program is a structure named xdp_md, in which data and data_end are the addresses of the data packet header and tail, respectively, and in the data packet header, each field is of fixed length. By parsing the data field in the context, the position of the field storing the source IP address is calculated.

[0015] After obtaining the IP address, the IP address is associated with the reservation value and limit value provided by the user. The information in the BPF MAP is shared by the kernel-mode eBPF program and the user-mode program of the loader program. The communication between the application program and the eBPF hook program is through the shared feature of the BPF MAP, so that the user space program can create a BPF MAP through the 'bpf()' system call. The MAP takes the IP address as the key and the reservation value and limit value as the value, and stores the mapping relationship between the IP, the reservation value and the limit value. The eBPF program obtains the relevant mapping relationship by reading the BPF MAP. After allocating the reservation bandwidth of each container, the remaining bandwidth is allocated according to the proportion adjustment algorithm. When allocating the remaining bandwidth, the limit value input by the user is compared. If it exceeds the limit value, the excess bandwidth is recorded. The excess bandwidth is proportionally allocated to the containers that have not reached the upper limit, and the operation is repeated until all the excess bandwidth is allocated or each container reaches the upper limit value.

[0016] Preferably, the adjustment process of the bandwidth dynamic proportion adjustment module is as follows:

[0017] When the bandwidth changes over time, the minimum bandwidth requirement of the container is guaranteed and the maximum bandwidth usage is limited. The bandwidth allocated to each container is still dynamically adjusted according to the container demand. By adopting a heat sorting algorithm, the "heat value" of each container is calculated based on the historical transmission volume and the length of time from the current time. Containers with low heat indicate that they have low demand for network IO resources. Bandwidth is allocated to containers with low resource demand first to avoid such containers being blocked by network IO intensive, that is, high-heat containers, to ensure their service quality QoS. When the QoS of network IO intensive containers remains basically unchanged, the QoS of non-network IO intensive containers is maximized, and the number of reserved tokens that can be allocated to the container is calculated when the total number of tokens is N.

[0018] Preferably, the filtering process of the data packet filtering module is as follows:

[0019] Network data packets are processed at the network driver layer through XDP. The XDP-based packet filtering solution reduces the CPU overhead occupied when processing data packets, and uses the XDP program to implement rule matching and data packet processing. By parsing the IP address and data packet processing rules in the user input command, the IP and processing rules are stored in BPFMAPip_map. Whenever a data packet is received, the packet IP is parsed from the XDP context, and the corresponding processing rules are searched through i p_map to accept, reject or forward it. While maintaining the QoS of network IO-intensive container applications basically unchanged, the QoS of non-network IO-intensive containers is greatly optimized.

[0020] The beneficial effects of the present invention are as follows:

[0021] The capacity network traffic control system designed by this method significantly optimizes the QoS of non-network I / O-intensive container applications in multi-container scenarios. Furthermore, compared to traditional IP tabs / netfilter frameworks, the packet filtering mechanism designed by this method, based on XDP technology, demonstrates higher data packet processing speed and lower single-packet processing overhead.

[0022] In this experiment, two containers C1 and C2 are deployed on a virtual machine, running 500 IO-intensive connections and 500 non-IO-intensive connections respectively (both based on the TCP protocol). Figure 6 The average and tail latencies of the two connections are shown (the 90th percentile latency is used as the tail latency in this experiment). As can be seen, after implementing this flow control algorithm, the average latency / tail latency of I / O-intensive connections increased by only 18% / 10%, while the average latency / tail latency of non-I / O-intensive connections decreased by only 75% / 89%. In other words, while the user experience of I / O-intensive applications remained virtually unchanged, the user experience of non-I / O-intensive applications was significantly improved.

[0023] This solution also tested the processing performance of the IPTables / Netfilter framework and the XDP-based packet filtering module developed in this project. Specifically, a remote VM ran a container application that continuously sent packets to the local machine, testing the packet processing speed and CPU utilization of both filtering tools. Figure 7 The data packet processing rates of the two tools are shown: it can be seen that compared with the IPtables / Netfilter framework, the solution in this paper increases the processing speed by 38%, verifying the efficiency of XDP technology.

[0024] Finally, this solution also tests the CPU usage under the two mechanisms. Figure 8 The results show that this solution achieves a 6% to 30% lower CPU utilization than the iPtab es / Netfilter framework. This is because the iPtab es / Netfilter framework resides in the kernel network protocol stack. Even if a packet is discarded at the earliest hook point in the packet receiving path, the expensive Linux network protocol stack processing overhead has already occurred. XDP, on the other hand, resides in the network card driver layer. Packets can be processed by the XDP program before a socket buffer is allocated. Compared to iPtab es / Netfilter, it can identify and discard packets earlier, resulting in higher processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings;

[0026] Figure 1 It is a flow chart of the system of the present invention;

[0027] Figure 2 It is a schematic diagram of the system architecture of this method;

[0028] Figure 3 It is the XDP context content description diagram;

[0029] Figure 4 It is the BPF MAP function diagram;

[0030] Figure 5 It is a rule matching flowchart;

[0031] Figure 6 Optimize graphs for average and tail latency;

[0032] Figure 7 Rate optimization diagram for the packet filtering module;

[0033] Figure 8 Optimize the graph for CPU usage. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] Example:

[0036] See also Figures 1 to 8 As shown, the present invention is a container network traffic control system based on XDP, including a bandwidth high limit and low limit module, a bandwidth dynamic ratio adjustment module and a data packet filtering module; the bandwidth high limit and low limit module is used to distinguish the source container of the network data packet, associate the IP address with the reservation value and limit value provided by the user, and transmit the data to the bandwidth dynamic ratio adjustment module; the bandwidth dynamic ratio adjustment module is used to dynamically adjust the bandwidth allocated to each container and transmit the data to the data packet filtering module; the data packet filtering module is used to perform rule matching and network data packet processing at the network driver layer through the XDP program;

[0037] The specific contents of the bandwidth limit and minimum protection module are as follows:

[0038] The XDP context contains all the information needed to access the data packet in its original form. The source of the data packet is obtained by analyzing its context. The context of the XDP program is a structure named xdp_md. In this structure, data and data_end are the addresses of the packet header and tail, respectively. In the packet header, each field is of fixed length. By parsing the data field in the context, the location of the field storing the source IP address is inferred. To ensure that the bandwidth of each container is within the reserved and limited values ​​set by the user, this method uses a token bucket algorithm for network traffic shaping and rate limiting. At the beginning, the total number of available tokens N is set. Each packet is processed and consumes one token. The number of tokens is reset to N after each second to limit the maximum processing rate to N pps. (To effectively utilize the remaining bandwidth, the present invention adopts a "moderate over-issuance of tokens" strategy, that is, the number of tokens issued N is slightly greater than the processing capacity of the network card). First, a reserv_map is defined to store reserved value information. Specifically, this MAP uses the IP address as the key and the reserved value as the value. When the program runs, it first traverses the MAP storing the reserved values, and calculates the total of all reserved values. R, based on which the proportion of each container's retention value to the total retention value is calculated:

[0039] P i =R i / total R

[0040] Wherein, R i represents the retention value of the i th container, P i represents the proportion of the i th container retention value;

[0041] When the IP address is acquired, the IP address is associated with the retention value and the limit value provided by the user, and the information in the BPF MAP is shared by the kernel eBPF program and the user space program of the loaded program, the application program and the eBPF hook program communicate through the sharing characteristics of the BPF MAP, so that the user space program creates a BPF MAP through the 'bpf()' system call, wherein the MAP takes the IP address as the key and the retention value and the limit value as the value, the mapping relationship between the IP, the retention value and the limit value is stored, so that the eBPF program obtains the related mapping relationship by reading the BPF MAP;

[0042] Token i =N*P i

[0043] If Token i (the number of reservation tokens allocated to the i th container) is greater than the reservation value stored in reserv_map, the reservation value in reserv_map is updated to Token i, otherwise, the reservation value in reserv_map remains unchanged; after allocating the reservation bandwidth of each container, the remaining bandwidth is allocated according to the proportional adjustment algorithm; when allocating the remaining bandwidth, compare with the limit value input by the user, if it exceeds the limit value, record the excess bandwidth; the excess bandwidth is proportionally allocated to the containers that have not reached the upper limit, and the operation is repeated until all the excess bandwidth is allocated or each container reaches the upper limit value;

[0044] The adjustment process of the bandwidth dynamic proportional adjustment module is as follows: when the time changes constantly, the minimum bandwidth demand of the container is ensured, the maximum bandwidth usage is limited, and then the bandwidth allocated to each container is dynamically adjusted according to the container demand, the heat value of each container is calculated by using the heat sorting algorithm according to the historical transmission amount and the length of time from the current time, the container with low heat value represents that its demand for network IO resources is small, and the bandwidth is preferentially allocated to the container with small resource demand, when the network IO intensive container QoS is basically maintained, the QoS of the non-network IO intensive container is maximized; then, the number of reservation tokens that can be allocated to the container under the condition that the total number of tokens is N is calculated:

[0045] Secondly, the present scheme dynamically adjusts the bandwidth occupied by each container based on the Hacker News heat sorting algorithm; Hacker News is a news submission community under Y Combinator, and the news ranking algorithm used by it is called Hacker News ranking algorithm; the algorithm formula is as follows:

[0046] Score=P / (T+2)G

[0047] Score represents the calculated heat value, P represents the number of data packets sent by the container in a period, T represents the index of the period, the project stores the data packet transmission statistics of the last 10 periods, from near to far, the index is 1, 2, 3... 10, and adding 2 is to prevent the denominator from being too small to cause the calculation result to be too large, G represents the "gravity factor", that is, the speed of heat decline over time, the present application takes 10s as a period, analyzes the data transmission characteristics of the last 10 periods, and takes the sum of the heat values of the last 10 periods as the current heat value of the container;

[0048] The filtering process of the data packet filtering module is as follows: the network data packet is processed in the network driver layer processing stage through XDP, the XDP-based packet filtering scheme reduces the CPU overhead occupied when processing the data packet, and the XDP program is used to realize rule matching and data packet processing, by analyzing the IP address and data packet processing rule in the user input command, the IP and processing rule are stored in BPFMAPip_map, whenever a data packet is received, the data packet IP is parsed from the XDP context, and the corresponding processing rule is found through ip_map, and it is accepted, rejected or forwarded; based on the obtained heat value, the bandwidth of each container is dynamically adjusted; the proportional calculation formula is as follows:

[0049]

[0050] Pi represents the proportion of the ith container, Score i represents the heat of the ith container, by taking the square root of the heat, it can prevent some network IO intensive applications from blocking non-network IO intensive applications, while ensuring that the IO intensive container obtains relatively higher bandwidth; based on the algorithm, the QoS of non-network IO intensive containers can be greatly optimized while maintaining the basic QoS of network IO intensive container applications.

[0051] The above formulas are obtained by collecting a large amount of data for software simulation, and selecting a formula close to the true value, the coefficients in the formula are set by the person skilled in the art according to the actual situation, the above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can make equivalent replacement or change according to the technical scheme and inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A system for XDP-based container network traffic control, characterized in that, The application relates to a bandwidth limiting module, a bandwidth dynamic proportion adjusting module and a data packet filtering module. The bandwidth limiting module is used for distinguishing the source containers of network data packets, associating the source container IP addresses with the reservation values and the limit values provided by users and transmitting data to the bandwidth dynamic proportion adjusting module. The bandwidth dynamic proportion adjusting module is used for dynamically adjusting the bandwidths allocated to the containers and transmitting data to the data packet filtering module. The data packet filtering module is used for performing rule matching and network data packet processing in a network driver layer through an XDP program. The specific content of the bandwidth limiting module is as follows:

2. The XDP-based container network traffic control system of claim 1, wherein, The XDP context contains all information required to access the data packet in an original form, the data packet source is obtained by analyzing the context, wherein the context of the XDP program is a structure body named xdp_md, the data and data_end in the structure body are the addresses of the data packet head and tail, in the data packet head, each field is of a fixed length, the position of the field storing the source IP address is calculated by analyzing the data field in the context; After the IP address is obtained, the IP address is associated with the reservation values and the limit values provided by users, the information in the BPF MAP is shared by the kernel state eBPF program and the user state program of the loading program, the application program and the eBPF hook program communicate through the sharing characteristic of the BPF MAP, the user space program is connected with the BPF MAP through a 'bpf()' system call, the BPF MAP takes the IP address as a key and takes the reservation values and the limit values as values, the mapping relationship between the IP, the reservation values and the limit values is stored, and the eBPF program obtains the related mapping relationship by reading the BPF MAP. The adjusting process of the bandwidth dynamic proportion adjusting module is as follows:

3. The XDP-based container network traffic control system of claim 1, wherein, When the bandwidths of the containers change over time, the minimum bandwidth requirement of the containers is ensured, the maximum bandwidth usage is limited, the bandwidths allocated to the containers are dynamically adjusted according to the container requirements, the heat value of each container is calculated through a heat sorting algorithm according to the historical transmission amount and the length of time from the current time, the container with low heat value represents that the container has small network IO resource requirement, the bandwidth is preferentially allocated to the container with small resource requirement, and when the QoS of the network IO intensive container is basically maintained, the QoS of the non-network IO intensive container is maximized. The filtering process of the data packet filtering module is as follows:

4. The XDP-based container network traffic control system of claim 1, wherein, The network data packet is processed in the network driver layer processing stage through the XDP, the CPU overhead occupied when the data packet is processed is reduced based on the XDP packet filtering scheme, the rule matching and the data packet processing are realized through the XDP program, the IP and the processing rule are stored in the BPF MAP ip_map by analyzing the IP address and the data packet processing rule in the user input command, the data packet IP is analyzed from the XDP context every time a data packet is received, the corresponding processing rule is searched through the BPF MAP ip_map, and the data packet is accepted, rejected or forwarded. ​

Images